View a markdown version of this page

摄取应用程序遥测数据 - 亚马逊 OpenSearch 服务

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

摄取应用程序遥测数据

要在 Amazon Serv OpenSearch ice 中使用可观察性功能,您需要获取应用程序跟踪、日志和指标。本页介绍如何配置 OpenTelemetry 收集器和 OpenSearch 摄取管道,以处理遥测数据并将其路由到适用于 Prometheus 的 OpenSearch Amazon 托管服务。

配置 OpenTelemetry 收集器

OpenTelemetry (OTel) 收集器是所有应用程序遥测的入口点。它通过 OTLP 接收数据,并将跟踪和日志路由到 OpenSearch Ingestion,同时向 Prometheus 发送指标。

您可以使用以下方法之一配置 OTel 收集器:

OTel 收集器使用 Sigv4 身份验证将跟踪和日志导出到 OpenSearch 摄取终端节点,并使用 Prometheus 远程写入导出器将指标导出到适用于 Prometheus 的亚马逊托管服务。 OpenSearch Ingestion 负责处理、扩充和路由到。 OpenSearch

OTel 使用 OpenSearch Ingesttion 配置收集器

以下示例配置使用 Sigv4 身份验证将跟踪和日志导出到 OpenSearch 摄取端点,将指标导出到 Prometheus:

extensions: sigv4auth: region: us-west-2 service: osis exporters: otlphttp/osis-traces: traces_endpoint: ${OSIS_ENDPOINT}/v1/traces auth: { authenticator: sigv4auth } compression: none otlphttp/osis-logs: logs_endpoint: ${OSIS_ENDPOINT}/v1/logs auth: { authenticator: sigv4auth } compression: none # Amazon Managed Service for Prometheus via Prometheus Remote Write with SigV4 auth prometheusremotewrite/amp: endpoint: "https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/remote_write" auth: authenticator: sigv4auth service: extensions: [sigv4auth] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp/osis-traces] logs: receivers: [otlp] processors: [batch] exporters: [otlphttp/osis-logs] metrics: receivers: [otlp] processors: [batch] exporters: [prometheusremotewrite/amp]
注意

在管道上发送数据的 IAM 委托人需要osis:Ingestaps:RemoteWrite许可 ARN。

配置 OpenSearch 摄取管道

OpenSearch 摄取(或自我管理的 Data Prepper)从 OTel收集器接收遥测数据,并对其进行处理以进行应用程序性能监控 (APM)。

管道架构

该管道分以下几个阶段处理遥测数据:

  1. 入口管道接收所有遥测数据,并将日志和跟踪路由到不同的子管道。

  2. 日志管道 OpenSearch 使用log-analytics-plain索引类型将日志数据写入。

  3. 追踪管道将跨度分配到原始存储管道和服务映射管道。

  4. 原始跟踪管道进程跨越otel_traces处理器,并以trace-analytics-plain-raw索引类型存储它们。

  5. 服务映射管道使用otel_apm_service_map处理器生成拓扑和 RED(速率、错误、持续时间)指标。它通过远程写入向 Prometheus 写入 OpenSearch 和写入。

管道配置

以下示例显示了 OpenSearch Ingestion 的完整管道配置,该配置涵盖了所有可观察性信号类型(日志、跟踪和指标)。您可以包含所有管道,也可以仅包含与您的用例相关的管道。用您自己的信息替换这些placeholder值。

version: '2' # Main OTLP pipeline - receives all telemetry and routes by signal type otlp-pipeline: source: otlp: logs_path: '/pipeline-name/v1/logs' traces_path: '/pipeline-name/v1/traces' metrics_path: '/pipeline-name/v1/metrics' route: - logs: 'getEventType() == "LOG"' - traces: 'getEventType() == "TRACE"' - metrics: 'getEventType() == "METRIC"' processor: [] sink: - pipeline: name: otel-logs-pipeline routes: - logs - pipeline: name: otel-traces-pipeline routes: - traces - pipeline: name: otel-metrics-pipeline routes: - metrics # Log processing pipeline otel-logs-pipeline: source: pipeline: name: otlp-pipeline processor: - copy_values: entries: - from_key: "time" to_key: "@timestamp" sink: - opensearch: hosts: - 'https://opensearch-endpoint' index_type: log-analytics-plain aws: serverless: false region: 'region' sts_role_arn: "arn:aws:iam::account-id:role/pipeline-role" # Trace fan-out pipeline otel-traces-pipeline: source: pipeline: name: otlp-pipeline processor: [] sink: - pipeline: name: traces-raw-pipeline routes: [] - pipeline: name: service-map-pipeline routes: [] # Raw trace storage pipeline traces-raw-pipeline: source: pipeline: name: otel-traces-pipeline processor: - otel_traces: sink: - opensearch: hosts: - 'https://opensearch-endpoint' index_type: trace-analytics-plain-raw aws: serverless: false region: 'region' sts_role_arn: "arn:aws:iam::account-id:role/pipeline-role" # Service map generation pipeline (APM) service-map-pipeline: source: pipeline: name: otel-traces-pipeline processor: - otel_apm_service_map: group_by_attributes: - telemetry.sdk.language # Add any resource attribute to group by window_duration: 30s route: - otel_apm_service_map_route: 'getEventType() == "SERVICE_MAP"' - service_processed_metrics: 'getEventType() == "METRIC"' sink: - opensearch: hosts: - 'https://opensearch-endpoint' aws: serverless: false region: 'region' sts_role_arn: "arn:aws:iam::account-id:role/pipeline-role" routes: - otel_apm_service_map_route index_type: otel-v2-apm-service-map - prometheus: url: 'https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/remote_write' aws: region: 'region' routes: - service_processed_metrics # Metrics processing pipeline otel-metrics-pipeline: source: pipeline: name: otlp-pipeline processor: - otel_metrics: sink: - prometheus: url: 'https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/remote_write' aws: region: 'region'

验证摄取

配置 OTel 收集器和管道后,请验证遥测数据是否正常流动。

  • 验证 OpenSearch 索引-确认您的域中存在以下索引:otel-v1-apm-span-*otel-v2-apm-service-map、和logs-otel-v1-*

  • 验证 Prometheus 目标 — 确认 Prometheu s 远程写入目标正在接收来自服务映射管道的指标。

  • 在 OpenSearch 用户界面中验证-导航到可观察性,然后导航到应用程序监控,以确认您的服务已出现。

后续步骤

确认已摄取遥测数据后,请浏览以下主题: