java 结合OpenTelemetry可观测性实践

OpenTelemetry Java Agent

项目架构总览：

graph TD
 OpenTelemetry中的java-agent  -->  OpenTelemetry中的collector -->es

1、在java opts中新增如下参数

-javaagent:/opt/opentelemetry-javaagent-v2.3.0.jar  
-Dotel.javaagent.extensions=/opt/opentelemetry-java-extension-env.jar   -Dotel.metrics.exporter=none  
-Dotel.logs.exporter=none  
-Dotel.exporter.otlp.traces.endpoint=http://OpenTelemetry-agent-collector.xxxxx:4318/v1/traces  
-Dotel.service.name=svw-integration-service

2、部署OpenTelemetry-agent-collector服务

3、修改OpenTelemetry-agent-collector服务配置文件

extensions:
  health_check:
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
exporters:
  file:
    path: /agent.json
  debug:
  elasticsearch/trace:
    endpoints: [http://es-xxxx:9200]
    user: esxxx
    password: esxxxxx
    traces_index: trace_index_prd
processors:
  batch/trace:
  memory_limiter/trace:
    limit_mib: 3800
    spike_limit_mib: 760
    check_interval: 5s
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: slow_calls
        type: latency
        latency:
          threshold_ms: 5000
      - name: error_otel_status
        type: status_code
        status_code:
          status_codes:
            - ERROR
      - name: error_http_status
        type: numeric_attribute
        numeric_attribute:
            key: http.status_code
            min_value: 500
      - name: keep_sampled_success
        type: and
        and:
          and_sub_policy:
            - name: drop_noisy_traces_url
              type: string_attribute
              string_attribute:
                key: http.url
                values:
                  - /metrics
                  - opentelemetry.proto
                  - favicon.ico
                  - /health
                enabled_regex_matching: true
                invert_match: true
            - name: keep_percentage
              type: probabilistic
              probabilistic:
                sampling_percentage: 10
service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter/trace,tail_sampling,batch/trace] #这里需要上面的exporters 值保持一致
      exporters: [debug,elasticsearch/trace]

4、扩展span中的自定义参数

基于官方资料github.com/open-teleme… -Dotel.javaagent.extensions 可在span中新增属性参数，比如Attribute.environment的值为svw

@Override
public void onStart(Context parentContext, ReadWriteSpan span) {
// 通过环境变量APP_ENV获取值
  String app_env = System.getenv("APP_ENV");
  if (app_env == null || app_env.isEmpty()) {
    span.setAttribute("environment", "xxxx");
  }
  span.setAttribute("environment", app_env);
}