基于腾讯云CLS与Prometheus的微服务告警体系构建：从指标采集到智能降噪

业务 Pod (暴露 /metrics) 
    --> TMP (Prometheus 抓取) 
    --> 告警规则引擎 (AlertManager) 
    --> 告警去重/分组/抑制 
    --> 通知渠道 (企业微信/邮件/短信)

业务 Pod (输出 JSON 日志) 
    --> Filebeat/DaemonSet 采集 
    --> CLS (日志存储与索引) 
    --> 告警策略 (CQL 实时分析) 
    --> 通知渠道

@RestController
@RequestMapping("/api/order")
public class OrderMetricsController {
    
    private final MeterRegistry registry;
    private final Counter orderSuccessCounter;
    private final Timer orderCreateTimer;
    private final Counter orderFailureCounter;

    public OrderMetricsController(MeterRegistry registry) {
        this.registry = registry;
        this.orderSuccessCounter = Counter.builder("order.create.success")
                .description("订单创建成功总数")
                .register(registry);
        this.orderFailureCounter = Counter.builder("order.create.failure")
                .tag("error_type", "unknown")
                .description("订单创建失败总数")
                .register(registry);
        this.orderCreateTimer = Timer.builder("order.create.duration")
                .publishPercentiles(0.5, 0.95, 0.99)
                .sla(Duration.ofMillis(100), Duration.ofMillis(500), Duration.ofMillis(1000))
                .register(registry);
    }

    @PostMapping("/create")
    public Response create(@RequestBody OrderReq req) {
        long start = System.currentTimeMillis();
        try {
            // 业务逻辑...
            orderSuccessCounter.increment();
            orderCreateTimer.record(System.currentTimeMillis() - start, TimeUnit.MILLISECONDS);
            return Response.success();
        } catch (Exception e) {
            orderFailureCounter.increment();
            orderCreateTimer.record(System.currentTimeMillis() - start, TimeUnit.MILLISECONDS);
            throw e;
        }
    }
}

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: order-recording-rules
  namespace: prod
spec:
  groups:
  - name: order.rules
    rules:
    - record: order:success_rate:5m
      expr: |
        sum(rate(order_create_success_total[5m])) 
        / 
        (sum(rate(order_create_success_total[5m])) + sum(rate(order_create_failure_total[5m])))
      labels:
        service: order-service

  - name: order.alerts
    rules:
    - alert: OrderHighErrorRate
      expr: |
        (
          sum(rate(order_create_failure_total[2m])) 
          / 
          (sum(rate(order_create_success_total[2m])) + sum(rate(order_create_failure_total[2m])))
        ) > 0.05
      for: 1m
      labels:
        severity: critical
        team: order-group
      annotations:
        summary: "订单服务错误率超过 5%（当前 {{ $value | humanizePercentage }}）"
        description: "请检查 order-service 日志，关键词 'SQLException' 或 'Timeout'"

route:
  group_by: ['alertname', 'namespace']   # 按告警名称和命名空间分组
  group_wait: 10s                        # 首次等待，聚合同类告警
  group_interval: 30s                    # 同一组告警发送间隔
  repeat_interval: 4h                    # 重复告警间隔
  receiver: 'default-receiver'
  routes:
  - match:
      severity: critical
    receiver: 'urgent-webhook'           # 严重告警走企业微信
    continue: false
  - match_re:
      severity: warning|info
    receiver: 'email-receiver'           # 普通告警走邮件

receivers:
- name: 'default-receiver'
  webhook_configs:
  - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxx'
- name: 'email-receiver'
  email_configs:
  - to: 'oncall@company.com'

// 使用 Logback 的 JsonLayout
<appender name="CLS_JSON" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
        <providers>
            <timestamp>
                <fieldName>timestamp</fieldName>
                <pattern>yyyy-MM-dd'T'HH:mm:ss.SSSZ</pattern>
            </timestamp>
            <threadName>
                <fieldName>thread</fieldName>
            </threadName>
            <logLevel>
                <fieldName>level</fieldName>
            </logLevel>
            <loggerName>
                <fieldName>logger</fieldName>
            </loggerName>
            <message>
                <fieldName>message</fieldName>
            </message>
            <stackTrace>
                <fieldName>stack_trace</fieldName>
            </stackTrace>
            <mdc>
                <includeMdcKeyName>traceId</includeMdcKeyName>
                <includeMdcKeyName>userId</includeMdcKeyName>
                <includeMdcKeyName>orderId</includeMdcKeyName>
            </mdc>
        </providers>
    </encoder>
</appender>

- 查询语句： 
  message: "PaymentTimeout" AND level: "ERROR" 
  | select count(*) as cnt, date_trunc('minute', timestamp) as time 
  | where cnt > 10 
  | group by time
- 触发条件：执行结果 > 0（即每分钟超时 > 10 次）
- 告警频率：每 5 分钟告警一次，避免重复轰炸

- 查询语句： 
  message: "Slow SQL" 
  | select avg(duration) as avg_duration, max(duration) as max_duration, count(*) as cnt
  | where avg_duration > 1000
- 触发条件：avg_duration > 1000（平均耗时 > 1 秒）

- 查询语句： 
  userId: "admin" AND (action: "DELETE" OR action: "UPDATE") 
  | select count(*) as cnt 
  | where cnt > 5
- 触发条件：执行结果 > 0（管理员短时间内操作 > 5 次）

# SCF 函数伪代码：处理 CLS 告警回调
def main_handler(event, context):
    alert_data = json.loads(event['body'])
    alert_name = alert_data['AlertName']
    log_content = alert_data['LogContent']
    
    if "PaymentTimeout" in alert_name:
        # 自动调用 TSF 的熔断接口，降级支付服务
        call_tsf_api('/circuit-breaker/open', service='payment-service')
        # 发送企微通知
        send_wechat("支付服务已自动熔断，请关注根因")
    
    elif "Slow SQL" in alert_name:
        # 自动提取慢 SQL，提交到 DBA 工单系统
        submit_dba_ticket(extract_sql(log_content))

告警详情：
订单服务错误率 6.2%（超过阈值 5%）
建议排查步骤：
1. 查看 TMP 实时面板：https://console.cloud.tencent.com/monitor/prometheus/xxx?service=order-service&from=now-1h
2. 查看 CLS 关联日志：https://console.cloud.tencent.com/cls/search?query=service:order-service AND level:ERROR

告警附加信息：
TraceId: 7a8b3c9d-12e3-4f56-9876-abcd1234ef56
调用链查询：https://jaeger.xxx.com/trace/7a8b3c9d-12e3-4f56-9876-abcd1234ef56

# AlertManager 静默配置（通过 API 动态创建）
{
  "matchers": [
    {"name": "severity", "value": "warning"}
  ],
  "startsAt": "2026-08-13T10:00:00Z",
  "endsAt": "2026-08-13T12:00:00Z",
  "createdBy": "ops-team",
  "comment": "压测期间屏蔽 warning 级别告警"
}