# CloudWatch metric stream output in OpenTelemetry 1.0.0 format **Note** With the OpenTelemetry 1.0.0 format, metric attributes are encoded as a list of `KeyValue` objects instead of the `StringKeyValue` type used in the 0.7.0 format. As a consumer, this is the only major change between the 0.7.0 and 1.0.0 formats. A parser generated from the 0.7.0 proto files won't parse metric attributes encoded in the 1.0.0 format. The same is true in reverse, a parser generated from the 1.0.0 proto files will not parse metric attributes encoded in the 0.7.0 format. OpenTelemetry is a collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis. OpenTelemetry is part of the Cloud Native Computing Foundation. For more information, see [OpenTelemetry](https://opentelemetry.io/). For information about the full OpenTelemetry 1.0.0 specification, see [Release version 1.0.0](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v1.0.0). A Kinesis record can contain one or more `ExportMetricsServiceRequest` OpenTelemetry data structures. Each data structure starts with a header with an `UnsignedVarInt32` indicating the record length in bytes. Each `ExportMetricsServiceRequest` may contain data from multiple metrics at once. The following is a string representation of the message of the `ExportMetricsServiceRequest` OpenTelemetry data structure. OpenTelemetry serializes the Google Protocol Buffers binary protocol, and this is not human-readable. ``` resource_metrics { resource { attributes { key: "cloud.provider" value { string_value: "aws" } } attributes { key: "cloud.account.id" value { string_value: "123456789012" } } attributes { key: "cloud.region" value { string_value: "us-east-1" } } attributes { key: "aws.exporter.arn" value { string_value: "arn:aws:cloudwatch:us-east-1:123456789012:metric-stream/MyMetricStream" } } } scope_metrics { metrics { name: "amazonaws.com/AWS/DynamoDB/ConsumedReadCapacityUnits" unit: "NoneTranslated" summary { data_points { start_time_unix_nano: 60000000000 time_unix_nano: 120000000000 count: 1 sum: 1.0 quantile_values { value: 1.0 } quantile_values { quantile: 0.95 value: 1.0 } quantile_values { quantile: 0.99 value: 1.0 } quantile_values { quantile: 1.0 value: 1.0 } attributes { key: "Namespace" value { string_value: "AWS/DynamoDB" } } attributes { key: "MetricName" value { string_value: "ConsumedReadCapacityUnits" } } attributes { key: "Dimensions" value { kvlist_value { values { key: "TableName" value { string_value: "MyTable" } } } } } } data_points { start_time_unix_nano: 70000000000 time_unix_nano: 130000000000 count: 2 sum: 5.0 quantile_values { value: 2.0 } quantile_values { quantile: 1.0 value: 3.0 } attributes { key: "Namespace" value { string_value: "AWS/DynamoDB" } } attributes { key: "MetricName" value { string_value: "ConsumedReadCapacityUnits" } } attributes { key: "Dimensions" value { kvlist_value { values { key: "TableName" value { string_value: "MyTable" } } } } } } } } } } ``` **Top-level object to serialize OpenTelemetry metric data** `ExportMetricsServiceRequest` is the top-level wrapper to serialize an OpenTelemetry exporter payload. It contains one or more `ResourceMetrics`. ``` message ExportMetricsServiceRequest { // An array of ResourceMetrics. // For data coming from a single resource this array will typically contain one // element. Intermediary nodes (such as OpenTelemetry Collector) that receive // data from multiple origins typically batch the data before forwarding further and // in that case this array will contain multiple elements. repeated opentelemetry.proto.metrics.v1.ResourceMetrics resource_metrics = 1; } ``` `ResourceMetrics` is the top-level object to represent MetricData objects. ``` // A collection of ScopeMetrics from a Resource. message ResourceMetrics { reserved 1000; // The resource for the metrics in this message. // If this field is not set then no resource info is known. opentelemetry.proto.resource.v1.Resource resource = 1; // A list of metrics that originate from a resource. repeated ScopeMetrics scope_metrics = 2; // This schema_url applies to the data in the "resource" field. It does not apply // to the data in the "scope_metrics" field which have their own schema_url field. string schema_url = 3; } ``` **The Resource object** A `Resource` object is a value-pair object that contains some information about the resource that generated the metrics. For metrics created by AWS, the data structure contains the Amazon Resource Name (ARN) of the resource related to the metric, such as an EC2 instance or an S3 bucket. The `Resource` object contains an attribute called `attributes`, which store a list of key-value pairs. + `cloud.account.id` contains the account ID + `cloud.region` contains the Region + `aws.exporter.arn` contains the metric stream ARN + `cloud.provider` is always `aws`. ``` // Resource information. message Resource { // Set of attributes that describe the resource. // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). repeated opentelemetry.proto.common.v1.KeyValue attributes = 1; // dropped_attributes_count is the number of dropped attributes. If the value is 0, then // no attributes were dropped. uint32 dropped_attributes_count = 2; } ``` **The ScopeMetrics object** The `scope` field will not be filled. We fill only the metrics field that we are exporting. ``` // A collection of Metrics produced by an Scope. message ScopeMetrics { // The instrumentation scope information for the metrics in this message. // Semantically when InstrumentationScope isn't set, it is equivalent with // an empty instrumentation scope name (unknown). opentelemetry.proto.common.v1.InstrumentationScope scope = 1; // A list of metrics that originate from an instrumentation library. repeated Metric metrics = 2; // This schema_url applies to all metrics in the "metrics" field. string schema_url = 3; } ``` **The Metric object** The metric object contains some metadata and a `Summary` data field that contains a list of `SummaryDataPoint`. For metric streams, the metadata is as follows: + `name` will be `amazonaws.com/metric_namespace/metric_name` + `description` will be blank + `unit` will be filled by mapping the metric datum's unit to the case-sensitive variant of the Unified code for Units of Measure. For more information, see [Translations with OpenTelemetry 1.0.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation-100.md) and [The Unified Code For Units of Measure](https://ucum.org/ucum.html). + `type` will be `SUMMARY` ``` message Metric { reserved 4, 6, 8; // name of the metric, including its DNS name prefix. It must be unique. string name = 1; // description of the metric, which can be used in documentation. string description = 2; // unit in which the metric value is reported. Follows the format // described by http://unitsofmeasure.org/ucum.html. string unit = 3; // Data determines the aggregation type (if any) of the metric, what is the // reported value type for the data points, as well as the relatationship to // the time interval over which they are reported. oneof data { Gauge gauge = 5; Sum sum = 7; Histogram histogram = 9; ExponentialHistogram exponential_histogram = 10; Summary summary = 11; } } message Summary { repeated SummaryDataPoint data_points = 1; } ``` **The SummaryDataPoint object** The SummaryDataPoint object contains the value of a single data point in a time series in a DoubleSummary metric. ``` // SummaryDataPoint is a single data point in a timeseries that describes the // time-varying values of a Summary metric. message SummaryDataPoint { reserved 1; // The set of key/value pairs that uniquely identify the timeseries from // where this point belongs. The list may be empty (may contain 0 elements). // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). repeated opentelemetry.proto.common.v1.KeyValue attributes = 7; // StartTimeUnixNano is optional but strongly encouraged, see the // the detailed comments above Metric. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. fixed64 start_time_unix_nano = 2; // TimeUnixNano is required, see the detailed comments above Metric. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. fixed64 count = 4; // sum of the values in the population. If count is zero then this field // must be zero. // // Note: Sum should only be filled out when measuring non-negative discrete // events, and is assumed to be monotonic over the values of these events. // Negative events *can* be recorded, but sum should not be filled out when // doing so. This is specifically to enforce compatibility w/ OpenMetrics, // see: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#summary double sum = 5; // Represents the value at a given quantile of a distribution. // // To record Min and Max values following conventions are used: // - The 1.0 quantile is equivalent to the maximum value observed. // - The 0.0 quantile is equivalent to the minimum value observed. // // See the following issue for more context: // https://github.com/open-telemetry/opentelemetry-proto/issues/125 message ValueAtQuantile { // The quantile of a distribution. Must be in the interval // [0.0, 1.0]. double quantile = 1; // The value at the given quantile of a distribution. // // Quantile values must NOT be negative. double value = 2; } // (Optional) list of values at different quantiles of the distribution calculated // from the current snapshot. The quantiles must be strictly increasing. repeated ValueAtQuantile quantile_values = 6; // Flags that apply to this specific data point. See DataPointFlags // for the available flags and their meaning. uint32 flags = 8; } ``` For more information, see [Translations with OpenTelemetry 1.0.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation-100.md). # Translations with OpenTelemetry 1.0.0 format in CloudWatch CloudWatch performs some transformations to put CloudWatch data into OpenTelemetry format. **Translating namespace, metric name, and dimensions** These attributes are key-value pairs encoded in the mapping. + One attribute has the key `Namespace` and its value is the namespace of the metric + One attribute has the key `MetricName` and its value is the name of the metric + One pair has the key `Dimensions` and its value is a nested list of key-value pairs. Each pair in this list maps to a CloudWatch metric dimension, where the pair's key is the name of the dimension and its value is the value of the dimension. **Translating Average, Sum, SampleCount, Min and Max** The Summary datapoint enables CloudWatch to export all of these statistics using one datapoint. + `startTimeUnixNano` contains the CloudWatch `startTime` + `timeUnixNano` contains the CloudWatch `endTime` + `sum` contains the Sum statistic. + `count` contains the SampleCount statistic. + `quantile_values` contains two `valueAtQuantile.value` objects: + `valueAtQuantile.quantile = 0.0` with `valueAtQuantile.value = Min value` + `valueAtQuantile.quantile = 0.99` with `valueAtQuantile.value = p99 value` + `valueAtQuantile.quantile = 0.999` with `valueAtQuantile.value = p99.9 value` + `valueAtQuantile.quantile = 1.0` with `valueAtQuantile.value = Max value` Resources that consume the metric stream can calculate the Average statistic as **Sum/SampleCount**. **Translating units** CloudWatch units are mapped to the case-sensitive variant of the Unified code for Units of Measure, as shown in the following table. For more information, see [The Unified Code For Units of Measure](https://ucum.org/ucum.html). | CloudWatch | OpenTelemetry | | --- | --- | | Second | s | | Second or Seconds | s | | Microseconds | us | | Milliseconds | ms | | Bytes | By | | Kilobytes | kBy | | Megabytes | MBy | | Gigabytes | GBy | | Terabytes | TBy | | Bits | bit | | Kilobits | kbit | | Megabits | MBit | | Gigabits | GBit | | Terabits | Tbit | | Percent | % | | Count | \$1Count\$1 | | None | 1 | Units that are combined with a slash are mapped by applying the OpenTelemetry conversion of both the units. For example, Bytes/Second is mapped to By/s. # How to parse OpenTelemetry 1.0.0 messages This section provides information to help you get started with parsing OpenTelemetry 1.0.0. First, you should get language-specific bindings, which enable you to parse OpenTelemetry 1.0.0 messages in your preferred language. **To get language-specific bindings** + The steps depend on your preferred language. + To use Java, add the following Maven dependency to your Java project: [OpenTelemetry Java >> 0.14.1](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-proto/0.14.1). + To use any other language, follow these steps: 1. Make sure that your language is supported by checking the list at [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). 1. Install the Protobuf compiler by following the steps at [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). 1. Download the OpenTelemetry 1.0.0 ProtoBuf definitions at [Release version 1.0.0](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v1.0.0). 1. Confirm that you are in the root folder of the downloaded OpenTelemetry 1.0.0 ProtoBuf definitions. Then create a `src` folder and then run the command to generate language-specific bindings. For more information, see [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). The following is an example for how to generate Javascript bindings. ``` protoc --proto_path=./ --js_out=import_style=commonjs,binary:src \ opentelemetry/proto/common/v1/common.proto \ opentelemetry/proto/resource/v1/resource.proto \ opentelemetry/proto/metrics/v1/metrics.proto \ opentelemetry/proto/collector/metrics/v1/metrics_service.proto ``` The following section includes examples of using the language-specific bindings that you can build using the previous instructions. **Java** ``` package com.example; import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.List; public class MyOpenTelemetryParser { public List parse(InputStream inputStream) throws IOException { List result = new ArrayList<>(); ExportMetricsServiceRequest request; /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|ExportMetricsService... ------ --------------------------- ------ ----------------------- */ while ((request = ExportMetricsServiceRequest.parseDelimitedFrom(inputStream)) != null) { // Do whatever we want with the parsed message result.add(request); } return result; } } ``` **Javascript** This example assumes that the root folder with the bindings generated is `./` The data argument of the function `parseRecord` can be one of the following types: + `Uint8Array` this is optimal + `Buffer` optimal under node + `Array.number` 8-bit integers ``` const pb = require('google-protobuf') const pbMetrics = require('./opentelemetry/proto/collector/metrics/v1/metrics_service_pb') function parseRecord(data) { const result = [] // Loop until we've read all the data from the buffer while (data.length) { /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|ExportMetricsService... ------ --------------------------- ------ ----------------------- */ const reader = new pb.BinaryReader(data) const messageLength = reader.decoder_.readUnsignedVarint32() const messageFrom = reader.decoder_.cursor_ const messageTo = messageFrom + messageLength // Extract the current `ExportMetricsServiceRequest` message to parse const message = data.subarray(messageFrom, messageTo) // Parse the current message using the ProtoBuf library const parsed = pbMetrics.ExportMetricsServiceRequest.deserializeBinary(message) // Do whatever we want with the parsed message result.push(parsed.toObject()) // Shrink the remaining buffer, removing the already parsed data data = data.subarray(messageTo) } return result } ``` **Python** You must read the `var-int` delimiters yourself or use the internal methods `_VarintBytes(size)` and `_DecodeVarint32(buffer, position)`. These return the position in the buffer just after the size bytes. The read-side constructs a new buffer that is limited to reading only the bytes of the message. ``` size = my_metric.ByteSize() f.write(_VarintBytes(size)) f.write(my_metric.SerializeToString()) msg_len, new_pos = _DecodeVarint32(buf, 0) msg_buf = buf[new_pos:new_pos+msg_len] request = metrics_service_pb.ExportMetricsServiceRequest() request.ParseFromString(msg_buf) ``` **Go** Use `Buffer.DecodeMessage()`. **C\$1** Use `CodedInputStream`. This class can read size-delimited messages. **C\$1\$1** The functions described in `google/protobuf/util/delimited_message_util.h` can read size-delimited messages. **Other languages** For other languages, see [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). When implementing the parser, consider that a Kinesis record can contain multiple `ExportMetricsServiceRequest` Protocol Buffers messages, each of them starting with a header with an `UnsignedVarInt32` that indicates the record length in bytes.