# CloudWatch metric stream output in OpenTelemetry 0.7.0 format OpenTelemetry is a collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis. OpenTelemetry is part of the Cloud Native Computing Foundation. For more information, see [OpenTelemetry](https://opentelemetry.io/). For information about the full OpenTelemetry 0.7.0 specification, see [v0.7.0 release](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v0.7.0). A Kinesis record can contain one or more `ExportMetricsServiceRequest` OpenTelemetry data structures. Each data structure starts with a header with an `UnsignedVarInt32` indicating the record length in bytes. Each `ExportMetricsServiceRequest` may contain data from multiple metrics at once. The following is a string representation of the message of the `ExportMetricsServiceRequest` OpenTelemetry data structure. OpenTelemetry serializes the Google Protocol Buffers binary protocol, and this is not human-readable. ``` resource_metrics { resource { attributes { key: "cloud.provider" value { string_value: "aws" } } attributes { key: "cloud.account.id" value { string_value: "2345678901" } } attributes { key: "cloud.region" value { string_value: "us-east-1" } } attributes { key: "aws.exporter.arn" value { string_value: "arn:aws:cloudwatch:us-east-1:123456789012:metric-stream/MyMetricStream" } } } instrumentation_library_metrics { metrics { name: "amazonaws.com/AWS/DynamoDB/ConsumedReadCapacityUnits" unit: "1" double_summary { data_points { labels { key: "Namespace" value: "AWS/DynamoDB" } labels { key: "MetricName" value: "ConsumedReadCapacityUnits" } labels { key: "TableName" value: "MyTable" } start_time_unix_nano: 1604948400000000000 time_unix_nano: 1604948460000000000 count: 1 sum: 1.0 quantile_values { quantile: 0.0 value: 1.0 } quantile_values { quantile: 0.95 value: 1.0 } quantile_values { quantile: 0.99 value: 1.0 } quantile_values { quantile: 1.0 value: 1.0 } } data_points { labels { key: "Namespace" value: "AWS/DynamoDB" } labels { key: "MetricName" value: "ConsumedReadCapacityUnits" } labels { key: "TableName" value: "MyTable" } start_time_unix_nano: 1604948460000000000 time_unix_nano: 1604948520000000000 count: 2 sum: 5.0 quantile_values { quantile: 0.0 value: 2.0 } quantile_values { quantile: 1.0 value: 3.0 } } } } } } ``` **Top-level object to serialize OpenTelemetry metric data** `ExportMetricsServiceRequest` is the top-level wrapper to serialize an OpenTelemetry exporter payload. It contains one or more `ResourceMetrics`. ``` message ExportMetricsServiceRequest { // An array of ResourceMetrics. // For data coming from a single resource this array will typically contain one // element. Intermediary nodes (such as OpenTelemetry Collector) that receive // data from multiple origins typically batch the data before forwarding further and // in that case this array will contain multiple elements. repeated opentelemetry.proto.metrics.v1.ResourceMetrics resource_metrics = 1; } ``` `ResourceMetrics` is the top-level object to represent MetricData objects. ``` // A collection of InstrumentationLibraryMetrics from a Resource. message ResourceMetrics { // The resource for the metrics in this message. // If this field is not set then no resource info is known. opentelemetry.proto.resource.v1.Resource resource = 1; // A list of metrics that originate from a resource. repeated InstrumentationLibraryMetrics instrumentation_library_metrics = 2; } ``` **The Resource object** A `Resource` object is a value-pair object that contains some information about the resource that generated the metrics. For metrics created by AWS, the data structure contains the Amazon Resource Name (ARN) of the resource related to the metric, such as an EC2 instance or an S3 bucket. The `Resource` object contains an attribute called `attributes`, which store a list of key-value pairs. + `cloud.account.id` contains the account ID + `cloud.region` contains the Region + `aws.exporter.arn` contains the metric stream ARN + `cloud.provider` is always `aws`. ``` // Resource information. message Resource { // Set of labels that describe the resource. repeated opentelemetry.proto.common.v1.KeyValue attributes = 1; // dropped_attributes_count is the number of dropped attributes. If the value is 0, // no attributes were dropped. uint32 dropped_attributes_count = 2; } ``` **The InstrumentationLibraryMetrics object** The instrumentation\$1library field will not be filled. We will fill only the metrics field that we are exporting. ``` // A collection of Metrics produced by an InstrumentationLibrary. message InstrumentationLibraryMetrics { // The instrumentation library information for the metrics in this message. // If this field is not set then no library info is known. opentelemetry.proto.common.v1.InstrumentationLibrary instrumentation_library = 1; // A list of metrics that originate from an instrumentation library. repeated Metric metrics = 2; } ``` **The Metric object** The metric object contains a `DoubleSummary` data field that contains a list of `DoubleSummaryDataPoint`. ``` message Metric { // name of the metric, including its DNS name prefix. It must be unique. string name = 1; // description of the metric, which can be used in documentation. string description = 2; // unit in which the metric value is reported. Follows the format // described by http://unitsofmeasure.org/ucum.html. string unit = 3; oneof data { IntGauge int_gauge = 4; DoubleGauge double_gauge = 5; IntSum int_sum = 6; DoubleSum double_sum = 7; IntHistogram int_histogram = 8; DoubleHistogram double_histogram = 9; DoubleSummary double_summary = 11; } } message DoubleSummary { repeated DoubleSummaryDataPoint data_points = 1; } ``` **The MetricDescriptor object** The MetricDescriptor object contains metadata. For more information, see [metrics.proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto#L110) on GitHub. For metric streams, the MetricDescriptor has the following contents: + `name` will be `amazonaws.com/metric_namespace/metric_name` + `description` will be blank. + `unit` will be filled by mapping the metric datum's unit to the case-sensitive variant of the Unified code for Units of Measure. For more information, see [Translations with OpenTelemetry 0.7.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation.md) and [The Unified Code For Units of Measure](https://ucum.org/ucum.html). + `type` will be `SUMMARY`. **The DoubleSummaryDataPoint object** The DoubleSummaryDataPoint object contains the value of a single data point in a time series in a DoubleSummary metric. ``` // DoubleSummaryDataPoint is a single data point in a timeseries that describes the // time-varying values of a Summary metric. message DoubleSummaryDataPoint { // The set of labels that uniquely identify this timeseries. repeated opentelemetry.proto.common.v1.StringKeyValue labels = 1; // start_time_unix_nano is the last time when the aggregation value was reset // to "zero". For some metric types this is ignored, see data types for more // details. // // The aggregation value is over the time interval (start_time_unix_nano, // time_unix_nano]. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. // // Value of 0 indicates that the timestamp is unspecified. In that case the // timestamp may be decided by the backend. fixed64 start_time_unix_nano = 2; // time_unix_nano is the moment when this aggregation value was reported. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. fixed64 count = 4; // sum of the values in the population. If count is zero then this field // must be zero. double sum = 5; // Represents the value at a given quantile of a distribution. // // To record Min and Max values following conventions are used: // - The 1.0 quantile is equivalent to the maximum value observed. // - The 0.0 quantile is equivalent to the minimum value observed. message ValueAtQuantile { // The quantile of a distribution. Must be in the interval // [0.0, 1.0]. double quantile = 1; // The value at the given quantile of a distribution. double value = 2; } // (Optional) list of values at different quantiles of the distribution calculated // from the current snapshot. The quantiles must be strictly increasing. repeated ValueAtQuantile quantile_values = 6; } ``` For more information, see [Translations with OpenTelemetry 0.7.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation.md). # Translations with OpenTelemetry 0.7.0 format in CloudWatch CloudWatch performs some transformations to put CloudWatch data into OpenTelemetry format. **Translating namespace, metric name, and dimensions** These attributes are key-value pairs encoded in the mapping. + One pair contains the namespace of the metric + One pair contains the name of the metric + For each dimension, CloudWatch stores the following pair: `metricDatum.Dimensions[i].Name, metricDatum.Dimensions[i].Value` **Translating Average, Sum, SampleCount, Min and Max** The Summary datapoint enables CloudWatch to export all of these statistics using one datapoint. + `startTimeUnixNano` contains the CloudWatch `startTime` + `timeUnixNano` contains the CloudWatch `endTime` + `sum` contains the Sum statistic. + `count` contains the SampleCount statistic. + `quantile_values` contains two `valueAtQuantile.value` objects: + `valueAtQuantile.quantile = 0.0` with `valueAtQuantile.value = Min value` + `valueAtQuantile.quantile = 0.99` with `valueAtQuantile.value = p99 value` + `valueAtQuantile.quantile = 0.999` with `valueAtQuantile.value = p99.9 value` + `valueAtQuantile.quantile = 1.0` with `valueAtQuantile.value = Max value` Resources that consume the metric stream can calculate the Average statistic as **Sum/SampleCount**. **Translating units** CloudWatch units are mapped to the case-sensitive variant of the Unified code for Units of Measure, as shown in the following table. For more information, see [The Unified Code For Units of Measure](https://ucum.org/ucum.html). | CloudWatch | OpenTelemetry | | --- | --- | | Second | s | | Second or Seconds | s | | Microsecond | us | | Milliseconds | ms | | Bytes | By | | Kilobytes | kBy | | Megabytes | MBy | | Gigabytes | GBy | | Terabytes | TBy | | Bits | bit | | Kilobits | kbit | | Megabits | MBit | | Gigabits | GBit | | Terabits | Tbit | | Percent | % | | Count | \$1Count\$1 | | None | 1 | Units that are combined with a slash are mapped by applying the OpenTelemetry conversion of both the units. For example, Bytes/Second is mapped to By/s. # How to parse OpenTelemetry 0.7.0 messages This section provides information to help you get started with parsing OpenTelemetry 0.7.0. First, you should get language-specific bindings, which enable you to parse OpenTelemetry 0.7.0 messages in your preferred language. **To get language-specific bindings** + The steps depend on your preferred language. + To use Java, add the following Maven dependency to your Java project: [OpenTelemetry Java >> 0.14.1](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-proto/0.14.1). + To use any other language, follow these steps: 1. Make sure that your language is supported by checking the list at [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). 1. Install the Protobuf compiler by following the steps at [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). 1. Download the OpenTelemetry 0.7.0 ProtoBuf definitions at [v0.7.0 release](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v0.7.0). 1. Confirm that you are in the root folder of the downloaded OpenTelemetry 0.7.0 ProtoBuf definitions. Then create a `src` folder and then run the command to generate language-specific bindings. For more information, see [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). The following is an example for how to generate Javascript bindings. ``` protoc --proto_path=./ --js_out=import_style=commonjs,binary:src \ opentelemetry/proto/common/v1/common.proto \ opentelemetry/proto/resource/v1/resource.proto \ opentelemetry/proto/metrics/v1/metrics.proto \ opentelemetry/proto/collector/metrics/v1/metrics_service.proto ``` The following section includes examples of using the language-specific bindings that you can build using the previous instructions. **Java** ``` package com.example; import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.List; public class MyOpenTelemetryParser { public List parse(InputStream inputStream) throws IOException { List result = new ArrayList<>(); ExportMetricsServiceRequest request; /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|pExportMetricsService... ------ --------------------------- ------ ----------------------- */ while ((request = ExportMetricsServiceRequest.parseDelimitedFrom(inputStream)) != null) { // Do whatever we want with the parsed message result.add(request); } return result; } } ``` **Javascript** This example assumes that the root folder with the bindings generated is `./` The data argument of the function `parseRecord` can be one of the following types: + `Uint8Array` this is optimal + `Buffer` optimal under node + `Array.number` 8-bit integers ``` const pb = require('google-protobuf') const pbMetrics = require('./opentelemetry/proto/collector/metrics/v1/metrics_service_pb') function parseRecord(data) { const result = [] // Loop until we've read all the data from the buffer while (data.length) { /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|ExportMetricsService... ------ --------------------------- ------ ----------------------- */ const reader = new pb.BinaryReader(data) const messageLength = reader.decoder_.readUnsignedVarint32() const messageFrom = reader.decoder_.cursor_ const messageTo = messageFrom + messageLength // Extract the current `ExportMetricsServiceRequest` message to parse const message = data.subarray(messageFrom, messageTo) // Parse the current message using the ProtoBuf library const parsed = pbMetrics.ExportMetricsServiceRequest.deserializeBinary(message) // Do whatever we want with the parsed message result.push(parsed.toObject()) // Shrink the remaining buffer, removing the already parsed data data = data.subarray(messageTo) } return result } ``` **Python** You must read the `var-int` delimiters yourself or use the internal methods `_VarintBytes(size)` and `_DecodeVarint32(buffer, position)`. These return the position in the buffer just after the size bytes. The read-side constructs a new buffer that is limited to reading only the bytes of the message. ``` size = my_metric.ByteSize() f.write(_VarintBytes(size)) f.write(my_metric.SerializeToString()) msg_len, new_pos = _DecodeVarint32(buf, 0) msg_buf = buf[new_pos:new_pos+msg_len] request = metrics_service_pb.ExportMetricsServiceRequest() request.ParseFromString(msg_buf) ``` **Go** Use `Buffer.DecodeMessage()`. **C\$1** Use `CodedInputStream`. This class can read size-delimited messages. **C\$1\$1** The functions described in `google/protobuf/util/delimited_message_util.h` can read size-delimited messages. **Other languages** For other languages, see [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). When implementing the parser, consider that a Kinesis record can contain multiple `ExportMetricsServiceRequest` Protocol Buffers messages, each of them starting with a header with an `UnsignedVarInt32` that indicates the record length in bytes.