full.pdf

Docs Ecosystem Status Community Blog Documentation OpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs. As an industry-standard, OpenTelemetry is supported...

Docs Ecosystem Status Community Blog Documentation OpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs. As an industry-standard, OpenTelemetry is supported by more than 40 observability vendors, integrated by many libraries, services, and apps, and adopted by numerous end users. What is OpenTelemetry? A short explanation of what OpenTelemetry is and isn’t. Getting Started Get started with OpenTelemetry based on your role. OpenTelemetry Concepts Key concepts in OpenTelemetry OpenTelemetry Demo Documentation Language APIs & SDKs OpenTelemetry code instrumentation is supported for many popular programming languages Collector Vendor-agnostic way to receive, process and export telemetry data. OpenTelemetry with Kubernetes Using OpenTelemetry with Kubernetes Functions as a Service OpenTelemetry supports various methods of monitoring Function-as-a-Service provided by different cloud vendors Migration How to migrate to OpenTelemetry Specifications Security Contributing Learn how to contribute to OpenTelemetry documentation. Last modified January 30, 2024: Edit main landing pages for style and clarity (#3848) (6970b1a4) Docs Ecosystem Status Community Blog What is OpenTelemetry? A short explanation of what OpenTelemetry is and isn’t. OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. Crucially, OpenTelemetry is vendor- and toolagnostic, meaning that it can be used with a broad variety of Observability backends, including open source tools like Jaeger  and Prometheus , as well as commercial offerings. OpenTelemetry is not an observability backend like Jaeger, Prometheus, or other commercial vendors. OpenTelemetry is focused on the generation, collection, management, and export of telemetry. A major goal of OpenTelemetry is that you can easily instrument your applications or systems, no matter their language, infrastructure, or runtime environment. Crucially, the storage and visualization of telemetry is intentionally left to other tools. What is observability? # Observability is the ability to understand the internal state of a system by examining its outputs. In the context of software, this means being able to understand the internal state of a system by examining its telemetry data, which includes traces, metrics, and logs. To make a system observable, it must be instrumented. That is, the code must emit traces, metrics, or logs. The instrumented data must then be sent to an observability backend. Why OpenTelemetry? # With the rise of cloud computing, microservices architectures, and increasingly complex business requirements, the need for software and infrastructure observability is greater than ever. OpenTelemetry satisfies the need for observability while following two key principles: 1. You own the data that you generate. There’s no vendor lockin. 2. You only have to learn a single set of APIs and conventions. Both principles combined grant teams and organizations the flexibility they need in today’s modern computing world. If you want to learn more, take a look at OpenTelemetry’s mission, vision, and values. Main OpenTelemetry components # OpenTelemetry consists of the following major components: A specification for all components A standard protocol that defines the shape of telemetry data Semantic conventions that define a standard naming scheme for common telemetry data types APIs that define how to generate telemetry data Language SDKs that implement the specification, APIs, and export of telemetry data A library ecosystem that implements instrumentation for common libraries and frameworks Automatic instrumentation components that generate telemetry data without requiring code changes The OpenTelemetry Collector, a proxy that receives, processes, and exports telemetry data Various other tools, such as the OpenTelemetry Operator for Kubernetes, OpenTelemetry Helm Charts, and community assets for FaaS OpenTelemetry is used by a wide variety of libraries, services and apps that have OpenTelemetry integrated to provide observability by default. OpenTelemetry is supported by numerous vendors, many of whom provide commercial support for OpenTelemetry and contribute to the project directly. Extensibility # OpenTelemetry is designed to be extensible. Some examples of how it can be extended include: Adding a receiver to the OpenTelemetry Collector to support telemetry data from a custom source Loading custom instrumentation libraries into an SDK Creating a distribution of an SDK or the Collector tailored to a specific use case Creating a new exporter for a custom backend that doesn’t yet support the OpenTelemetry protocol (OTLP) Creating a custom propagator for a nonstandard context propagation format Although most users might not need to extend OpenTelemetry, the project is designed to make it possible at nearly every level. History # OpenTelemetry is a Cloud Native Computing Foundation (CNCF)  project that is the result of a merger between two prior projects, OpenTracing  and OpenCensus . Both of these projects were created to solve the same problem: the lack of a standard for how to instrument code and send telemetry data to an Observability backend. As neither project was fully able to solve the problem independently, they merged to form OpenTelemetry and combine their strengths while offering a single solution. If you are currently using OpenTracing or OpenCensus, you can learn how to migrate to OpenTelemetry in the Migration guide. What next? # Getting started — jump right in! Learn about OpenTelemetry concepts. Last modified January 30, 2024: Edit main landing pages for style and clarity (#3848) (6970b1a4) Docs Ecosystem Status Community Blog Getting started for Developers This is the getting-started page for you if: You develop software. Your goal is to get observability by writing code. You want to have your dependencies emit telemetry for you automatically. OpenTelemetry can help you! To accomplish your goals of having your dependencies instrumented automatically and instrumenting your own code with our API manually, we recommend that you learn the following concepts first: What is OpenTelemetry? How can I add instrumentation to my code base? If you develop libraries, frameworks or middleware that is used as a dependency within other software, we recommend that you learn how you can provide telemetry natively: How can I add native instrumentation to my library? Next, you can deep dive into the documentations for the language you are using: C++.NET Erlang / Elixir Go Java JavaScript / TypeScript PHP Python Ruby Rust Swift Other Last modified February 27, 2024: Update instrumentation concept page (#3996) (cb429263) Docs Ecosystem Status Community Blog Getting started for Ops This is the getting-started page for you if: You run a set of applications in production. Your goal is to get telemetry out of them without touching their code. You want to collect traces, metrics, and logs from several services and send them off to your observability backend. OpenTelemetry can help you! To accomplish your goal of getting telemetry out of applications without touching their code, we recommend that you learn the following: What is OpenTelemetry? How can I instrument applications without touching their code? How can I set up a collector? How can I get automation for Kubernetes with the OpenTelemetry Operator? If you are looking for a set of applications to try things out, you will find our official OpenTelemetry demo useful! Last modified February 27, 2024: Update instrumentation concept page (#3996) (cb429263) Docs Ecosystem Status Community Blog Observability Primer Core observability concepts. What is Observability? # Observability lets us understand a system from the outside, by letting us ask questions about that system without knowing its inner workings. Furthermore, it allows us to easily troubleshoot and handle novel problems (i.e. “unknown unknowns”), and helps us answer the question, “Why is this happening?” In order to be able to ask those questions of a system, the application must be properly instrumented. That is, the application code must emit signals such as traces, metrics, and logs. An application is properly instrumented when developers don’t need to add more instrumentation to troubleshoot an issue, because they have all of the information they need. OpenTelemetry is the mechanism by which application code is instrumented, to help make a system observable. Reliability & Metrics # Telemetry refers to data emitted from a system, about its behavior. The data can come in the form of traces, metrics, and logs. Reliability answers the question: “Is the service doing what users expect it to be doing?” A system could be up 100% of the time, but if, when a user clicks “Add to Cart” to add a black pair of shoes to their shopping cart, and instead, the system doesn’t always add black shoes, then the system would be said to be unreliable. Metrics are aggregations over a period of time of numeric data about your infrastructure or application. Examples include: system error rate, CPU utilization, request rate for a given service. For more on metrics and how they pertain to OpenTelemetry, see Metrics. SLI, or Service Level Indicator, represents a measurement of a service’s behavior. A good SLI measures your service from the perspective of your users. An example SLI can be the speed at which a web page loads. SLO, or Service Level Objective, is the means by which reliability is communicated to an organization/other teams. This is accomplished by attaching one or more SLIs to business value. Understanding Distributed Tracing # To understand Distributed Tracing, let’s start with some basics. Logs # A log is a timestamped message emitted by services or other components. Unlike traces, however, they are not necessarily associated with any particular user request or transaction. They are found almost everywhere in software, and have been heavily relied on in the past by both developers and operators alike to help them understand system behavior. Sample log: I, [2021-02-23T13:26:23.505892 #22473]  INFO -- : Unfortunately, logs aren’t extremely useful for tracking code execution, as they typically lack contextual information, such as where they were called from. They become far more useful when they are included as part of a span, or when they are correlated with a trace and a span. For more on logs and how they pertain to OTel, see Logs. Spans # A span represents a unit of work or operation. It tracks specific operations that a request makes, painting a picture of what happened during the time in which that operation was executed. A span contains name, time-related data, structured log messages, and other metadata (that is, Attributes) to provide information about the operation it tracks. Span attributes # The following table contains examples of span attributes: Key Value http.request.method "GET" network.protocol.version "1.1" url.path "/webshop/articles/4" url.query "?s=1" server.address "example.com" server.port 8080 url.scheme "https" http.route "/webshop/articles/:article http.response.status_code 200 client.address "192.0.2.4" Key client.socket.address Value "192.0.2.5" (the client goes through a proxy) user_agent.original "Mozilla/5.0 (Windows NT 10 Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0 For more on spans and how they pertain to OpenTelemetry, see Spans. Distributed Traces # A distributed trace, more commonly known as a trace, records the paths taken by requests (made by an application or enduser) as they propagate through multi-service architectures, like microservice and serverless applications. Without tracing, it is challenging to pinpoint the cause of performance problems in a distributed system. It improves the visibility of our application or system’s health and lets us debug behavior that is difficult to reproduce locally. Tracing is essential for distributed systems, which commonly have nondeterministic problems or are too complicated to reproduce locally. Tracing makes debugging and understanding distributed systems less daunting by breaking down what happens within a request as it flows through a distributed system. A trace is made of one or more spans. The first span represents the root span. Each root span represents a request from start to finish. The spans underneath the parent provide a more in-depth context of what occurs during a request (or what steps make up a request). Many Observability backends visualize traces as waterfall diagrams that may look something like this: Waterfall diagrams show the parent-child relationship between a root span and its child spans. When a span encapsulates another span, this also represents a nested relationship. For more on traces and how they pertain to OpenTelemetry, see Traces. Last modified March 14, 2024: Change wording to suit global audience (#4163) (9790b1e4) Docs Ecosystem Status Community Blog Context Propagation Learn about the concept that enables Distributed Tracing. With Context Propagation, Signals can be correlated with each other, regardless of where they are generated. Although not limited to tracing, it is what allows traces to build causal information about a system across services that are arbitrarily distributed across process and network boundaries. We define Context Propagation by two sub-concepts: Context and Propagation. Context # A Context is an object that contains the information for the sending and receiving service (or execution unit) to correlate one signal with another. For example, if Service A calls Service B, then a span from Service A whose ID is in context will be used as the parent span for the next span created in Service B. The trace ID that is in context will be used for the next span created in Service B as well, which signifies that the span is part of the same trace as the span from Service A. Propagation # Propagation is the mechanism that moves context between services and processes. It serializes or deserializes the context object and provides the relevant information to be propagated from one service to another. Propagation is usually handled by instrumentation libraries and is transparent to the user, but in the event that you need to manually propagate context, you can use Propagation APIs. OpenTelemetry maintains several official propagators. The default propagator is using the headers specified by the W3C TraceContext  specification. Specification # To learn more about Context Propagation, see the Context specification. Last modified December 16, 2023: Update contextpropagation.md (#3693) (d2126314) Docs Ecosystem Status Community Blog Traces Understand the full path through your distributed application. Traces give us the big picture of what happens when a request is made to an application. Whether your application is a monolith with a single database or a sophisticated mesh of services, traces are essential to understanding the full “path” a request takes in your application. Let’s explore this with three units of work, represented as Spans: hello span:  { "name": "hello", "context": { "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df "span_id": "0x051581bf3cb55c13" }, "parent_id": null, "start_time": "2022-04-29T18:52:58.114201Z", "end_time": "2022-04-29T18:52:58.114687Z", "attributes": { "http.route": "some_route1" }, "events": [ { "name": "Guten Tag!", "timestamp": "2022-04-29T18:52:58.114561Z", "attributes": { "event_attributes": 1 } } ] } This is the root span, denoting the beginning and end of the entire operation. Note that it has a trace_id field indicating the trace, but has no parent_id. That’s how you know it’s the root span. hello-greetings span:  { "name": "hello-greetings", "context": { "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df "span_id": "0x5fb397be34d26b51" }, "parent_id": "0x051581bf3cb55c13", "start_time": "2022-04-29T18:52:58.114304Z", "end_time": "2022-04-29T22:52:58.114561Z", "attributes": { "http.route": "some_route2" }, "events": [ { "name": "hey there!", "timestamp": "2022-04-29T18:52:58.114561Z", "attributes": { "event_attributes": 1 } }, { "name": "bye now!", "timestamp": "2022-04-29T18:52:58.114585Z", "attributes": { "event_attributes": 1 } } ] } This span encapsulates specific tasks, like saying greetings, and its parent is the hello span. Note that it shares the same trace_id as the root span, indicating it’s a part of the same trace. Additionally, it has a parent_id that matches the span_id of the hello span. hello-salutations span: {  "name": "hello-salutations", "context": { "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df "span_id": "0x93564f51e1abe1c2" }, "parent_id": "0x051581bf3cb55c13", "start_time": "2022-04-29T18:52:58.114492Z", "end_time": "2022-04-29T18:52:58.114631Z", "attributes": { "http.route": "some_route3" }, "events": [ { "name": "hey there!", "timestamp": "2022-04-29T18:52:58.114561Z", "attributes": { "event_attributes": 1 } } ] } This span represents the third operation in this trace and, like the previous one, it’s a child of the ‘hello’ Span. That also makes it a sibling of the hello-greetings span. These three blocks of JSON all share the same trace_id , and the parent_id field represents a hierarchy. That makes it a Trace! Another thing you’ll note is that each Span looks like a structured log. That’s because it kind of is! One way to think of Traces is that they’re a collection of structured logs with context, correlation, hierarchy, and more baked in. However, these “structured logs” can come from different processes, services, VMs, data centers, and so on. This is what allows tracing to represent an end-to-end view of any system. To understand how tracing in OpenTelemetry works, let’s look at a list of components that will play a part in instrumenting our code. Tracer Provider # A Tracer Provider (sometimes called TracerProvider ) is a factory for Tracer s. In most applications, a Tracer Provider is initialized once and its lifecycle matches the application’s lifecycle. Tracer Provider initialization also includes Resource and Exporter initialization. It is typically the first step in tracing with OpenTelemetry. In some language SDKs, a global Tracer Provider is already initialized for you. Tracer # A Tracer creates spans containing more information about what is happening for a given operation, such as a request in a service. Tracers are created from Tracer Providers. Trace Exporters # Trace Exporters send traces to a consumer. This consumer can be standard output for debugging and development-time, the OpenTelemetry Collector, or any open source or vendor backend of your choice. Context Propagation # Context Propagation is the core concept that enables Distributed Tracing. With Context Propagation, Spans can be correlated with each other and assembled into a trace, regardless of where Spans are generated. To learn more about this topic, see the concept page on Context Propagation. Spans # A span represents a unit of work or operation. Spans are the building blocks of Traces. In OpenTelemetry, they include the following information: Name Parent span ID (empty for root spans) Start and End Timestamps Span Context Attributes Span Events Span Links Span Status Sample span:  { "name": "/v1/sys/health", "context": { "trace_id": "7bba9f33312b3dbb8b2c2c62bb7abe2d" "span_id": "086e83747d0e381e" }, "parent_id": "", "start_time": "2021-10-22 16:04:01.209458162 +00 "end_time": "2021-10-22 16:04:01.209514132 +0000 "status_code": "STATUS_CODE_OK", "status_message": "", "attributes": { "net.transport": "IP.TCP", "net.peer.ip": "172.17.0.1", "net.peer.port": "51820", "net.host.ip": "10.177.2.152", "net.host.port": "26040", "http.method": "GET", "http.target": "/v1/sys/health", "http.server_name": "mortar-gateway", "http.route": "/v1/sys/health", "http.user_agent": "Consul Health Check", "http.scheme": "http", "http.host": "10.177.2.152:26040", "http.flavor": "1.1" }, "events": [ { "name": "", "message": "OK", "timestamp": "2021-10-22 16:04:01.209512872 } ] } Spans can be nested, as is implied by the presence of a parent span ID: child spans represent sub-operations. This allows spans to more accurately capture the work done in an application. Span Context # Span context is an immutable object on every span that contains the following: The Trace ID representing the trace that the span is a part of The span’s Span ID Trace Flags, a binary encoding containing information about the trace Trace State, a list of key-value pairs that can carry vendorspecific trace information Span context is the part of a span that is serialized and propagated alongside Distributed Context and Baggage. Because Span Context contains the Trace ID, it is used when creating Span Links. Attributes # Attributes are key-value pairs that contain metadata that you can use to annotate a Span to carry information about the operation it is tracking. For example, if a span tracks an operation that adds an item to a user’s shopping cart in an eCommerce system, you can capture the user’s ID, the ID of the item to add to the cart, and the cart ID. You can add attributes to spans during or after span creation. Prefer adding attributes at span creation to make the attributes available to SDK sampling. If you have to add a value after span creation, update the span with the value. Attributes have the following rules that each language SDK implements: Keys must be non-null string values Values must be a non-null string, boolean, floating point value, integer, or an array of these values Additionally, there are Semantic Attributes, which are known naming conventions for metadata that is typically present in common operations. It’s helpful to use semantic attribute naming wherever possible so that common kinds of metadata are standardized across systems. Span Events # A Span Event can be thought of as a structured log message (or annotation) on a Span, typically used to denote a meaningful, singular point in time during the Span’s duration. For example, consider two scenarios in a web browser: 1. Tracking a page load 2. Denoting when a page becomes interactive A Span is best used to the first scenario because it’s an operation with a start and an end. A Span Event is best used to track the second scenario because it represents a meaningful, singular point in time. When to use span events versus span attributes # Since span events also contain attributes, the question of when to use events instead of attributes might not always have an obvious answer. To inform your decision, consider whether a specific timestamp is meaningful. For example, when you’re tracking an operation with a span and the operation completes, you might want to add data from the operation to your telemetry. If the timestamp in which the operation completes is meaningful or relevant, attach the data to a span event. If the timestamp isn’t meaningful, attach the data as span attributes. Span Links # Links exist so that you can associate one span with one or more spans, implying a causal relationship. For example, let’s say we have a distributed system where some operations are tracked by a trace. In response to some of these operations, an additional operation is queued to be executed, but its execution is asynchronous. We can track this subsequent operation with a trace as well. We would like to associate the trace for the subsequent operations with the first trace, but we cannot predict when the subsequent operations will start. We need to associate these two traces, so we will use a span link. You can link the last span from the first trace to the first span in the second trace. Now, they are causally associated with one another. Links are optional but serve as a good way to associate trace spans with one another. Span Status # Each span has a status. The three possible values are: Unset Error Ok The default value is Unset. A span status that is Unset means that the operation it tracked successfully completed without an error. When a span status is Error , then that means some error occurred in the operation it tracks. For example, this could be due to an HTTP 500 error on a server handling a request. When a span status is Ok , then that means the span was explicitly marked as error-free by the developer of an application. Although this is unintuitive, it’s not required to set a span status as Ok when a span is known to have completed without error, as this is covered by Unset. What Ok does is represent an unambiguous “final call” on the status of a span that has been explicitly set by a user. This is helpful in any situation where a developer wishes for there to be no other interpretation of a span other than “successful”. To reiterate: Unset represents a span that completed without an error. Ok represents when a developer explicitly marks a span as successful. In most cases, it is not necessary to explicitly mark a span as Ok. Span Kind # When a span is created, it is one of Client , Server , Internal , Producer , or Consumer. This span kind provides a hint to the tracing backend as to how the trace should be assembled. According to the OpenTelemetry specification, the parent of a server span is often a remote client span, and the child of a client span is usually a server span. Similarly, the parent of a consumer span is always a producer and the child of a producer span is always a consumer. If not provided, the span kind is assumed to be internal. For more information regarding SpanKind, see SpanKind. Client # A client span represents a synchronous outgoing remote call such as an outgoing HTTP request or database call. Note that in this context, “synchronous” does not refer to async/await , but to the fact that it is not queued for later processing. Server # A server span represents a synchronous incoming remote call such as an incoming HTTP request or remote procedure call. Internal # Internal spans represent operations which do not cross a process boundary. Things like instrumenting a function call or an Express middleware may use internal spans. Producer # Producer spans represent the creation of a job which may be asynchronously processed later. It may be a remote job such as one inserted into a job queue or a local job handled by an event listener. Consumer # Consumer spans represent the processing of a job created by a producer and may start long after the producer span has already ended. Specification # For more information, see the traces specification. Last modified March 21, 2024: Update span events/attributes guidance based on missed feedback (#4190) (926a8d97) Docs Ecosystem Status Community Blog Metrics A measurement captured at runtime A metric is a measurement of a service captured at runtime. The moment of capturing a measurements is known as a metric event, which consists not only of the measurement itself, but also the time at which it was captured and associated metadata. Application and request metrics are important indicators of availability and performance. Custom metrics can provide insights into how availability indicators impact user experience or the business. Collected data can be used to alert of an outage or trigger scheduling decisions to scale up a deployment automatically upon high demand. To understand how metrics in OpenTelemetry works, let’s look at a list of components that will play a part in instrumenting our code. Meter Provider # A Meter Provider (sometimes called MeterProvider ) is a factory for Meter s. In most applications, a Meter Provider is initialized once and its lifecycle matches the application’s lifecycle. Meter Provider initialization also includes Resource and Exporter initialization. It is typically the first step in metering with OpenTelemetry. In some language SDKs, a global Meter Provider is already initialized for you. Meter # A Meter creates metric instruments, capturing measurements about a service at runtime. Meters are created from Meter Providers. Metric Exporter # Metric Exporters send metric data to a consumer. This consumer can be standard output for debugging during development, the OpenTelemetry Collector, or any open source or vendor backend of your choice. Metric Instruments # In OpenTelemetry measurements are captured by metric instruments. A metric instrument is defined by: Name Kind Unit (optional) Description (optional) The name, unit, and description are chosen by the developer or defined via semantic conventions for common ones like request and process metrics. The instrument kind is one of the following: Counter: A value that accumulates over time – you can think of this like an odometer on a car; it only ever goes up. Asynchronous Counter: Same as the Counter, but is collected once for each export. Could be used if you don’t have access to the continuous increments, but only to the aggregated value. UpDownCounter: A value that accumulates over time, but can also go down again. An example could be a queue length, it will increase and decrease with the number of work items in the queue. Asynchronous UpDownCounter: Same as the UpDownCounter, but is collected once for each export. Could be used if you don’t have access to the continuous changes, but only to the aggregated value (e.g., current queue size). Gauge: Measures a current value at the time it is read. An example would be the fuel gauge in a vehicle. Gauges are asynchronous. Histogram: A client-side aggregation of values, such as request latencies. A histogram is a good choice if you are interested in value statistics. For example: How many requests take fewer than 1s? For more on synchronous and asynchronous instruments, and which kind is best suited for your use case, see Supplementary Guidelines. Aggregation # In addition to the metric instruments, the concept of aggregations is an important one to understand. An aggregation is a technique whereby a large number of measurements are combined into either exact or estimated statistics about metric events that took place during a time window. The OTLP protocol transports such aggregated metrics. The OpenTelemetry API provides a default aggregation for each instrument which can be overridden using the Views. The OpenTelemetry project aims to provide default aggregations that are supported by visualizers and telemetry backends. Unlike request tracing, which is intended to capture request lifecycles and provide context to the individual pieces of a request, metrics are intended to provide statistical information in aggregate. Some examples of use cases for metrics include: Reporting the total number of bytes read by a service, per protocol type. Reporting the total number of bytes read and the bytes per request. Reporting the duration of a system call. Reporting request sizes in order to determine a trend. Reporting CPU or memory usage of a process. Reporting average balance values from an account. Reporting current active requests being handled. Views # A view provides SDK users with the flexibility to customize the metrics output by the SDK. You can customize which metric instruments are to be processed or ignored. You can also customize aggregation and what attributes you want to report on metrics. Language Support # Metrics are a stable signal in the OpenTelemetry specification. For the individual language specific implementations of the Metrics API & SDK, the status is as follows: Language Metrics C++ Stable C#/.NET Stable Language Metrics Erlang/Elixir Experimental Go Stable Java Stable JavaScript Stable PHP Stable Python Stable Ruby In development Rust Alpha Swift Experimental Specification # To learn more about metrics in OpenTelemetry, see the metrics specification. Last modified February 15, 2024: Update metrics docs for JS (#3853) (e73b510b) Docs Ecosystem Status Community Blog Logs A log is a timestamped text record, either structured (recommended) or unstructured, with metadata. Of all telemetry signals, logs have the biggest legacy. Most programming languages have built-in logging capabilities or well-known, widely used logging libraries. Although logs are an independent data source, they may also be attached to spans. In OpenTelemetry, any data that is not part of a distributed trace or a metric is a log. For example, events are a specific type of log. Logs often contain detailed debugging/diagnostic info, such as inputs to an operation, the result of the operation, and any supporting metadata for that operation. For traces and metrics, OpenTelemetry takes the approach of a clean-sheet design, specifies a new API, and provides full implementations of this API in multiple language SDKs. OpenTelemetry’s approach with logs is different. Because existing logging solutions are widespread in language and operational ecosystems, OpenTelemetry acts as a “bridge” between those logs, the tracing and metrics signals, and other OpenTelemetry components. In fact, the API for logs is called the “Logs Bridge API” for this reason. The logs specification contains more details on this philosophy. To understand how logging in OpenTelemetry works, let’s look at a list of components that will play a part in instrumenting our code. Log Appender / Bridge # As an application developer, the Logs Bridge API should not be called by you directly, as it is provided for logging library authors to build log appenders / bridges. Instead, you just use your preferred logging library and configure it to use a log appender (or log bridge) that is able to emit logs into an OpenTelemetry LogRecordExporter. OpenTelemetry language SDKs offer this functionality. Logger Provider # Part of the Logs Bridge API and should only be used if you are the author of a logging library. A Logger Provider (sometimes called LoggerProvider ) is a factory for Logger s. In most cases, the Logger Provider is initialized once and its lifecycle matches the application’s lifecycle. Logger Provider initialization also includes Resource and Exporter initialization. Logger # Part of the Logs Bridge API and should only be used if you are the author of a logging library. A Logger creates log records. Loggers are created from Log Providers. Log Record Exporter # Log Record Exporters send log records to a consumer. This consumer can be standard output for debugging and development-time, the OpenTelemetry Collector, or any open source or vendor backend of your choice. Log Record # A log record represents the recording of an event. In OpenTelemetry a log record contains two kinds of fields: Named top-level fields of specific type and meaning Resource and attributes fields of arbitrary value and type The top-level fields are: Field Name Description Timestamp Time when the event occurred. Field Name Description ObservedTimestamp Time when the event was observed. TraceId Request trace ID. SpanId Request span ID. TraceFlags W3C trace flag. SeverityText The severity text (also known as log level). SeverityNumber Numerical value of the severity. Body The body of the log record. Resource Describes the source of the log. InstrumentationScope Describes the scope that emitted the log. Attributes Additional information about the event. For more details on log records and log fields, see Logs Data Model. Language Support # Logs are a stable signal in the OpenTelemetry specification. For the individual language specific implementations of the Logs API & SDK, the status is as follows: Language Logs C++ Stable C#/.NET Stable Erlang/Elixir Experimental Go In development Java Stable JavaScript Experimental PHP Stable Python Experimental Ruby In development Rust Alpha Swift In development Specification # To learn more about logs in OpenTelemetry, see the logs specification. Last modified January 20, 2024: Fix typos in "Signals" documentation section (#3825) (6e3bf583) Docs Ecosystem Status Community Blog Baggage Contextual information that is passed between signals In OpenTelemetry, Baggage is contextual information that’s passed between spans. It’s a key-value store that resides alongside span context in a trace, making values available to any span created within that trace. For example, imagine you want to have a CustomerId attribute on every span in your trace, which involves multiple services; however, CustomerId is only available in one specific service. To accomplish your goal, you can use OpenTelemetry Baggage to propagate this value across your system. OpenTelemetry uses Context Propagation to pass Baggage around, and each of the different library implementations has propagators that parse and make that Baggage available without you needing to explicitly implement it. Why does OTel Baggage exist? # Baggage provides a uniform way to store and propagate information across a trace and other signals. For example, you may want to attach information from your application to a span and retrieve that information much later and use it later on with another span. However, spans in OpenTelemetry are immutable once created, and can be exported before you need information on them later on. Baggage allows you to work around this problem by providing a place to store and retrieve information. What should OTel Baggage be used for? # Common use cases include information that’s only accessible further up a stack. This can include things like Account Identification, User IDs, Product IDs, and origin IPs, for example. Passing these down your stack allows you to then add them to your Spans in downstream services to make it easier to filter when you’re searching in your Observability backend. Baggage security considerations # Sensitive Baggage items could be shared with unintended resources, like third-party APIs. This is because automatic instrumentation includes Baggage in most of your service’s network requests. Specifically, Baggage and other parts of trace context are sent in HTTP headers, making it visible to anyone inspecting your network traffic. If traffic is restricted within your network, then this risk may not apply, but keep in mind that downstream services could propagate Baggage outside your network. Also, there are no built-in integrity checks to ensure that Baggage items are yours, so exercise caution when retrieving them. Baggage is not the same as Span attributes # One important thing to note about Baggage is that it is not a subset of the Span Attributes. When you add something as Baggage, it does not automatically end up on the Attributes of the child system’s spans. You must explicitly take something out of Baggage and append it as Attributes. For example, in.NET you might do this:  var accountId = Baggage.GetBaggage("AccountId"); Activity.Current?.SetTag("AccountId", accountId); For more information, see the baggage specification. Last modified January 30, 2024: [CI/textlint] Enforce "backend" rather than "back-end" (#3892) (81610838) Docs Ecosystem Status Community Blog Instrumentation How OpenTelemetry facilitates instrumentation In order to make a system observable, it must be instrumented: That is, code from the system’s components must emit traces, metrics, and logs. Using OpenTelemetry, you can instrument your code in two primary ways: 1. Code-based solutions via official APIs and SDKs for most languages 2. Zero-code solutions Code-based solutions allow you to get deeper insight and rich telemetry from your application itself. They let you use the OpenTelemetry API to generate telemetry from your application, which acts as an essential complement to the telemetry generated by zero-code solutions. Zero-code solutions are great for getting started, or when you can’t modify the application you need to get telemetry out of. They provide rich telemetry from libraries you use and/or the environment your application runs in. Another way to think of it is that they provide information about what’s happening at the edges of your application. You can use both solutions simultaneously. Additional OpenTelemetry Benefits # OpenTelemetry provides more than just zero-code and codebased telemetry solutions. The following things are also a part of OpenTelemetry: Libraries can leverage the OpenTelemetry API as a dependency, which will have no impact on applications using that library, unless the OpenTelemetry SDK is imported. For each signal (traces, metrics, logs) you have several methods at your disposals to create, process, and export them. With context propagation built into the implementations, you can correlate signals regardless of where they are generated. Resources and Instrumentation Scopes allow grouping of signals, by different entities, like, the host, operating system or K8s cluster Each language-specific implementation of the API and SDK follows the requirements and expectations of the OpenTelemetry specification. Semantic Conventions provide a common naming schema that can be used for standardization across code bases and platforms. Zero-code Learn how to add observability to an application without the need to write code Code-based Learn the essential steps in setting up code-based instrumentation Libraries Learn how to add native instrumentation to your library. Last modified March 4, 2024: Concepts pages: copyedits for instrumentation pages (#4066) (8d7aa298) Docs Ecosystem Status Community Blog Zero-code Learn how to add observability to an application without the need to write code As ops you might want to add observability to one or more applications without having to edit the source. OpenTelemetry lets you quickly gain some observability for a service without having to use the OpenTelemetry API & SDK for code-based instrumentation. Zero-code instrumentation adds the OpenTelemetry API and SDK capabilities to your application typically as an agent or agent-like installation. The specific mechanisms involved may differ by language, ranging from bytecode manipulation, monkey patching, or eBPF to inject calls to the OpenTelemetry API and SDK into your application. Typically, zero-code instrumentation adds instrumentation for the libraries you’re using. This means that requests and responses, database calls, message queue calls, and so forth are what are instrumented. Your application’s code, however, is not typically instrumented. To instrument your code, you’ll need to use codebased instrumentation. Additionally, zero-code instrumentation lets you configure the Instrumentation Libraries and exporters loaded. You can configure zero-code instrumentation through environment variables and other language-specific mechanisms, such as system properties or arguments passed to initialization methods. To get started, you only need a service name configured so that you can identify the service in the observability backend of your choice. Other configuration options are available, including: Data source specific configuration Exporter configuration Propagator configuration Resource configuration Automatic instrumentation is available for the following languages:.NET Java JavaScript PHP Python Last modified March 4, 2024: Concepts pages: copyedits for instrumentation pages (#4066) (8d7aa298) Docs Ecosystem Status Community Blog Code-based Learn the essential steps in setting up code-based instrumentation Import the OpenTelemetry API and SDK # You’ll first need to import OpenTelemetry to your service code. If you’re developing a library or some other component that is intended to be consumed by a runnable binary, then you would only take a dependency on the API. If your artifact is a standalone process or service, then you would take a dependency on the API and the SDK. For more information about the OpenTelemetry API and SDK, see the specification. Configure the OpenTelemetry API # In order to create traces or metrics, you’ll need to first create a tracer and/or meter provider. In general, we recommend that the SDK should provide a single default provider for these objects. You’ll then get a tracer or meter instance from that provider, and give it a name and version. The name you choose here should identify what exactly is being instrumented – if you’re writing a library, for example, then you should name it after your library (for example com.example.myLibrary ) as this name will namespace all spans or metric events produced. It is also recommended that you supply a version string (i.e., semver:1.0.0 ) that corresponds to the current version of your library or service. Configure the OpenTelemetry SDK # If you’re building a service process, you’ll also need to configure the SDK with appropriate options for exporting your telemetry data to some analysis backend. We recommend that this configuration be handled programmatically through a configuration file or some other mechanism. There are also perlanguage tuning options you may wish to take advantage of. Create Telemetry Data # Once you’ve configured the API and SDK, you’ll then be free to create traces and metric events through the tracer and meter objects you obtained from the provider. Make use of Instrumentation Libraries for your dependencies – check out the registry or your language’s repository for more information on these. Export Data # Once you’ve created telemetry data, you’ll want to send it somewhere. OpenTelemetry supports two primary methods of exporting data from your process to an analysis backend, either directly from a process or by proxying it through the OpenTelemetry Collector. In-process export requires you to import and take a dependency on one or more exporters, libraries that translate OpenTelemetry’s in-memory span and metric objects into the appropriate format for telemetry analysis tools like Jaeger or Prometheus. In addition, OpenTelemetry supports a wire protocol known as OTLP , which is supported by all OpenTelemetry SDKs. This protocol can be used to send data to the OpenTelemetry Collector, a standalone binary process that can be run as a proxy or sidecar to your service instances or run on a separate host. The Collector can then be configured to forward and export this data to your choice of analysis tools. In addition to open source tools such as Jaeger or Prometheus, a growing list of companies support ingesting telemetry data from OpenTelemetry. For details, see Vendors. Last modified March 4, 2024: Concepts pages: copyedits for instrumentation pages (#4066) (8d7aa298) Docs Ecosystem Status Community Blog Libraries Learn how to add native instrumentation to your library. OpenTelemetry provides instrumentation libraries for many libraries, which is typically done through library hooks or monkey-patching library code. Native library instrumentation with OpenTelemetry provides better observability and developer experience for users, removing the need for libraries to expose and document hooks: custom logging hooks can be replaced by common and easy to use OpenTelemetry APIs, users will only interact with OpenTelemetry traces, logs, metrics from library and application code are correlated and coherent common conventions allow users to get similar and consistent telemetry within same technology and across libraries and languages telemetry signals can be fine tuned (filtered, processed, aggregated) for various consumption scenarios using a wide variety of well-documented OpenTelemetry extensibility points. Semantic Conventions # Check out available semantic conventions that cover webframeworks, RPC clients, databases, messaging clients, infra pieces and more! If your library is one of those things - follow the conventions, they are the main source of truth and tell which information should be included on spans. Conventions make instrumentation consistent: users who work with telemetry don’t have to learn library specifics and observability vendors can build experiences for a wide variety of technologies (e.g. databases or messaging systems). When libraries follow conventions, many scenarios may be enabled out of the box without the user’s input or configuration. Semantic conventions are always evolving and new ones are constantly added. If some don’t exist for your library, then please consider adding them . Pay special attention to span names; strive to use meaningful names and consider cardinality when defining them. There is a schema_url attribute that can be used to record what version of the semantic conventions are being used. Please set this attribute, when possible. If you have any feedback or want to add a new convention please come and contribute! Instrumentation Slack  or Specification repository  are a good places to start! Defining spans # Think of your library from the perspective of a library user and what the user might be interested in knowing about the behavior and activity of the library. As the library maintainer, you know the internals but the user will most likely be less interested in the inner-workings of the library and more interested in the functionality of their application. Think about what information can be helpful in analyzing the usage of your library, then think about an appropriate way to model that data. Some things to consider are: Spans and span hierarchies Numerical attributes on spans (as an alternative to aggregated metrics) Span events Aggregated Metrics For example, if your library is making requests to a database, create spans only for the logical request to the database. The physical requests over the network should be instrumented within the libraries implementing that functionality. You should also favor capturing other activities, like object/data serialization as span events, rather than as additional spans. Follow the semantic conventions when setting span attributes. When not to instrument # Some libraries are thin clients wrapping network calls. Chances are that OpenTelemetry has an instrumentation library for the underlying RPC client (check out the registry). In this case, instrumenting the wrapper library may not be necessary. As a general guideline, only instrument your library at its own level. Don’t instrument if: your library is a thin proxy on top of documented or selfexplanatory APIs and OpenTelemetry has instrumentation for underlying network calls and there are no conventions your library should follow to enrich telemetry If you’re in doubt - don’t instrument - you can always do it later when you see a need. If you choose not to instrument, it may still be useful to provide a way to configure OpenTelemetry handlers for your internal RPC client instance. It’s essential in languages that don’t support fully automatic instrumentation and still useful in others. The rest of this document gives guidance on what and how to instrument if you decide to do it. OpenTelemetry API # The first step is to take dependency on the OpenTelemetry API package. OpenTelemetry has two main modules - API and SDK. OpenTelemetry API is a set of abstractions and not-operational implementations. Unless your application imports the OpenTelemetry SDK, your instrumentation does nothing and does not impact application performance. Libraries should only use the OpenTelemetry API. You may be rightfully concerned about adding new dependencies, here are some considerations to help you decide how to minimize dependency hell: OpenTelemetry Trace API reached stability in early 2021, it follows Semantic Versioning 2.0 and we take API stability seriously. When taking dependency, use the earliest stable OpenTelemetry API (1.0.*) and avoid updating it unless you have to use new features. While your instrumentation stabilizes, consider shipping it as a separate package, so that will never cause issues for users who don’t use it. You can keep it in your repository, or add it to OpenTelemetry , so it will ship with other instrumentation packages. Semantic Conventions are stable, but subject to evolution: while this does not cause any functional issues, you may need to update your instrumentation every once in a while. Having it in a preview plugin or in OpenTelemetry contrib repository may help keeping conventions up-to-date without breaking changes for your users. Getting a tracer # All application configuration is hidden from your library through the Tracer API. Libraries may allow applications to pass instances of TracerProvider to facilitate dependency injection and ease of testing, or obtain it from global TracerProvider. OpenTelemetry language implementations may have different preferences for passing instances or accessing the global based on what’s idiomatic. When obtaining the tracer, provide your library (or tracing plugin) name and version - they show up on the telemetry and help users process and filter telemetry, understand where it came from, and debug/report any instrumentation issues. What to instrument # Public APIs # Public APIs are a good candidates for tracing: spans created for public API calls allow users to map telemetry to application code, understand the duration and outcome of library calls. Which calls to trace: public methods that make network calls internally or local operations that take significant time and may fail (e.g. IO) handlers that process requests or messages Instrumentation example: private static Tracer tracer =  getTracer(TracerPr public static void setTracerProvider(TracerProvide tracer = getTracer(tracerProvider); } private static Tracer getTracer(TracerProvider tra return tracerProvider.getTracer("demo-db-clien } private Response selectWithTracing(Query query) { // check out conventions for guidance on span Span span = tracer.spanBuilder(String.format(".setSpanKind(SpanKind.CLIENT).setAttribute("db.name", dbName)....startSpan(); // makes span active and allows correlating lo try (Scope unused = span.makeCurrent()) { Response response = query.runWithRetries() if (response.isSuccessful()) { span.setStatus(StatusCode.OK); } if (span.isRecording()) { // populate response attributes for res } } catch (Exception e) { span.recordException(e); span.setStatus(StatusCode.ERROR, e.getClas throw e; } finally { span.end(); } } Follow conventions to populate attributes! If there is no applicable one, check out general conventions. Nested network and other spans # Network calls are usually traced with OpenTelemetry autoinstrumentations through corresponding client implementation. If OpenTelemetry does not support tracing your network client, use your best judgement, here are some considerations to help: Would tracing network calls improve observability for users or your ability to support them? Is your library a wrapper on top of public, documented RPC API? Would users need to get support from the underlying service in case of issues? instrument the library and make sure to trace individual network tries Would tracing those calls with spans be very verbose? or would it noticeably impact performance? use logs with verbosity or span events: logs can be correlated to parent (public API calls), while span events should be set on public API span. if they have to be spans (to carry and propagate unique trace context), put them behind a configuration option and disable them by default. If OpenTelemetry already supports tracing your network calls, you probably don’t want to duplicate it. There may be some exceptions: to support users without auto-instrumentation (which may not work in certain environments or users may have concerns with monkey-patching) to enable custom (legacy) correlation and context propagation protocols with underlying service enrich RPC spans with absolutely essential library/servicespecific information not covered by auto-instrumentation WARNING: Generic solution to avoid duplication is under construction 🚧. Events # Traces are one kind of signal that your apps can emit. Events (or logs) and traces complement, not duplicate, each other. Whenever you have something that should have a verbosity, logs are a better choice than traces. Chances are that your app uses logging or some similar module already. Your module might already have OpenTelemetry integration – to find out, see the registry. Integrations usually stamp active trace context on all logs, so users can correlate them. If your language and ecosystem don’t have common logging support, use span events to share additional app details. Events maybe more convenient if you want to add attributes as well. As a rule of thumb, use events or logs for verbose data instead of spans. Always attach events to the span instance that your instrumentation created. Avoid using the active span if you can, since you don’t control what it refers to. Context propagation # Extracting context # If you work on a library or a service that receives upstream calls, e.g. a web framework or a messaging consumer, you should extract context from the incoming request/message. OpenTelemetry provides the Propagator API, which hides specific propagation standards and reads the trace Context from the wire. In case of a single response, there is just one context on the wire, which becomes the parent of the new span the library creates. After you create a span, you should pass new trace context to the application code (callback or handler), by making the span active; if possible, you should do this explicitly.  // extract the context Context extractedContext = propagator.extract(Cont Span span = tracer.spanBuilder("receive").setSpanKind(SpanKind.SERVER).setParent(extractedContext).startSpan(); // make span active so any nested telemetry is cor try (Scope unused = span.makeCurrent()) { userCode(); } catch (Exception e) { span.recordException(e); span.setStatus(StatusCode.ERROR); throw e; } finally { span.end(); } Here’re the full examples of context extraction in Java, check out OpenTelemetry documentation in your language. In the case of a messaging system, you may receive more than one message at once. Received messages become links on the span you create. Refer to messaging conventions for details (WARNING: messaging conventions are under constructions  🚧). Injecting context # When you make an outbound call, you will usually want to propagate context to the downstream service. In this case, you should create a new span to trace the outgoing call and use Propagator API to inject context into the message. There may be other cases where you might want to inject context, e.g. when creating messages for async processing. Span span = tracer.spanBuilder("send").setSpanKind(SpanKind.CLIENT).startSpan();  // make span active so any nested telemetry is cor // even network calls might have nested layers of try (Scope unused = span.makeCurrent()) { // inject the context propagator.inject(Context.current(), transportLa send(); } catch (Exception e) { span.recordException(e); span.setStatus(StatusCode.ERROR); throw e; } finally { span.end(); } Here’s the full example of context injection in Java. There might be some exceptions: downstream service does not support metadata or prohibits unknown fields downstream service does not define correlation protocols. Is it possible that some future service version will support compatible context propagation? Inject it! downstream service supports custom correlation protocol. best effort with custom propagator: use OpenTelemetry trace context if compatible. or generate and stamp custom correlation IDs on the span. In-process # Make your spans active (aka current): it enables correlating spans with logs and any nested autoinstrumentations. If the library has a notion of context, support optional explicit trace context propagation in addition to active spans put spans (trace context) created by library in the context explicitly, document how to access it allow users to pass trace context in your context Within the library, propagate trace context explicitly - active spans may change during callbacks! capture active context from users on the public API surface as soon as you can, use it as a parent context for your spans pass context around and stamp attributes, exceptions, events on explicitly propagated instances this is essential if you start threads explicitly, do background processing or other things that can break due to async context flow limitations in your language Misc # Instrumentation registry # Please add your instrumentation library to the OpenTelemetry registry, so users can find it. Performance # OpenTelemetry API is no-op and very performant when there is no SDK in the application. When OpenTelemetry SDK is configured, it consumes bound resources. Real-life applications, especially on the high scale, would frequently have head-based sampling configured. Sampled-out spans are cheap and you can check if the span is recording, to avoid extra allocations and potentially expensive calculations, while populating attributes.  // some attributes are important for sampling, the Span span = tracer.spanBuilder(String.format("SELE.setSpanKind(SpanKind.CLIENT).setAttribute("db.name", dbName)....startSpan(); // other attributes, especially those that are exp // should be added if span is recording if (span.isRecording()) { span.setAttribute("db.statement", sanitize(que } Error handling # OpenTelemetry API is forgiving at runtime - does not fail on invalid arguments, never throws, and swallows exceptions. This way instrumentation issues do not affect application logic. Test the instrumentation to notice issues OpenTelemetry hides at runtime. Testing # Since OpenTelemetry has variety of auto-instrumentations, it’s useful to try how your instrumentation interacts with other telemetry: incoming requests, outgoing requests, logs, etc. Use a typical application, with popular frameworks and libraries and all tracing enabled when trying out your instrumentation. Check out how libraries similar to yours show up. For unit testing, you can usually mock or fake SpanProcessor and SpanExporter. @Test public void checkInstrumentation() { SpanExporter exporter = new TestExporter();  Tracer tracer = OpenTelemetrySdk.builder().setTracerProvider(SdkTracerProvider.bu.addSpanProcessor(SimpleSpanProcesso.getTracer("test"); // run test... validateSpans(exporter.exportedSpans); } class TestExporter implements SpanExporter { public final List exportedSpans = Coll @Override public CompletableResultCode export(CollectionSetStatus(status) span->SetStatus(StatusCode::kOk); Tracing context propagation #  In C++ propagation is not automatically handled. You need to extract it from the caller and inject the propagation context into subsequent spans. The GrpcServerCarrier class defines a method to extract context from inbound gRPC requests which is leveraged in the service call implementations. The GrpcServerCarrier class is defined in tracer_common.h as follows:  class GrpcServerCarrier : public opentelemetry::co { public: GrpcServerCarrier(ServerContext *context) : cont GrpcServerCarrier() = default; virtual opentelemetry::nostd::string_view Get( opentelemetry::nostd::string_view key) const { auto it = context_->client_metadata().find(key if (it != context_->client_metadata().end()) { return it->second.data(); } return ""; } virtual void Set(opentelemetry::nostd::string_vi opentelemetry::nostd::string_vi { // Not required for server } ServerContext *context_; }; This class is leveraged in the Convert method to extract context and create a StartSpanOptions object to contain the right context which is used when creating new spans. StartSpanOptions options; options.kind = SpanKind::kServer; GrpcServerCarrier carrier(context); auto prop auto current_ctx auto new_context options.parent = = = =  context::propagation::GlobalTex context::RuntimeContext::GetCur prop->Extract(carrier, current_ GetSpan(new_context)->GetContex Metrics # Initializing Metrics # The OpenTelemetry MeterProvider is initialized from main() using the initMeter() function defined in meter_common.h. void initMeter() { // Build MetricExporter  otlp_exporter::OtlpGrpcMetricExporterOptions otl auto exporter = otlp_exporter::OtlpGrpcMetricExp // Build MeterProvider and Reader metric_sdk::PeriodicExportingMetricReaderOptions std::unique_ptr reader new metric_sdk::PeriodicExportingMetricReade auto provider = std::shared_ptrCreateUInt64Counter(co return int_counter; } Counting currency conversion requests # The method CurrencyCounter() is implemented as follows:  void CurrencyCounter(const std::string& currency_c { std::map labels = { auto labelkv = common::KeyValueIterableViewAdd(1, labelkv); } Every time the function Convert() is called, the currency code received as to_code is used to count the conversions. CurrencyCounter(to_code);  Logs # The OpenTelemetry LoggerProvider is initialized from main() using the initLogger() function defined in logger_common.h.  void initLogger() { otlp::OtlpGrpcLogRecordExporterOptions loggerOpt auto exporter = otlp::OtlpGrpcLogRecordExporter auto processor = logs_sdk::SimpleLogRecordProces std::vectorgetTracer('manual-instrumentation') ->spanBuilder('calculate-quote') ->setSpanKind(SpanKind::KIND_INTERNAL) ->startSpan(); $span->end(); Add span attributes #  You can obtain the current span using OpenTelemetry\API\Trace\Span. $span = Span::getCurrent();  Adding attributes to a span is accomplished using setAttribute on the span object. In the calculateQuote function 2 attributes are added to the childSpan.  $childSpan->setAttribute('app.quote.items.count', $childSpan->setAttribute('app.quote.cost.total', $ Add span events # Adding span events is accomplished using addEvent on the span object. In the getquote route span events are added. Some events have additional attributes, others do not. Adding a span event without attributes:  $span->addEvent('Received get quote request, proce Adding a span event with additional attributes:  $span->addEvent('Quote processed, response sent ba 'app.quote.cost.total' => $payload ]); Metrics # TBD Logs # TBD Last modified October 19, 2023: Support page-relative aliases (#3417) (7f46ec2d) Docs Ecosystem Status Community Blog Recommendation Service This service is responsible to get a list of recommended products for the user based on existing product IDs the user is browsing. Recommendation service source  Auto-instrumentation # This Python based service, makes use of the OpenTelemetry auto-instrumentor for Python, accomplished by leveraging the opentelemetry-instrument Python wrapper to run the scripts. This can be done in the ENTRYPOINT command for the service’s Dockerfile.  ENTRYPOINT [ "opentelemetry-instrument", "python", Traces # Initializing Tracing # The OpenTelemetry SDK is initialized in the __main__ code block. This code will create a tracer provider, and establish a Span Processor to use. Export endpoints, resource attributes, and service name are automatically set by the OpenTelemetry auto instrumentor based on environment variables.  tracer = trace.get_tracer_provider().get_tracer("r Add attributes to auto-instrumented spans # Within the execution of auto-instrumented code you can get current span from context. span = trace.get_current_span()  Adding attributes to a span is accomplished using set_attribute on the span object. In the ListRecommendations function an attribute is added to the span.  span.set_attribute("app.products_recommended.count Create new spans # New spans can be created and placed into active context using start_as_current_span from an OpenTelemetry Tracer object. When used in conjunction with a with block, the span will automatically be ended when the block ends execution. This is done in the get_product_list function.  with tracer.start_as_current_span("get_product_lis Metrics # Initializing Metrics # The OpenTelemetry SDK is initialized in the __main__ code block. This code will create a meter provider. Export endpoints, resource attributes, and service name are automatically set by the OpenTelemetry auto instrumentor based on environment variables.  meter = metrics.get_meter_provider().get_meter("re Custom metrics # The following custom metrics are currently available: app_recommendations_counter : Cumulative count of # recommended products per service call Auto-instrumented metrics # The following metrics are available through auto-instrumentation, courtesy of the opentelemetry-instrumentation-systemmetrics , which is installed as part of opentelemetrybootstrap on building the recommendationservice Docker image: runtime.cpython.cpu_time runtime.cpython.memory runtime.cpython.gc_count Logs # Initializing logs # The OpenTelemetry SDK is initialized in the __main__ code block. The following code creates a logger provider with a batch processor, an OTLP log exporter, and a logging handler. Finally, it creates a logger for use throughout the application. logger_provider = LoggerProvider( resource=Resource.create(  { 'service.name': service_name, } ), ) set_logger_provider(logger_provider) log_exporter = OTLPLogExporter(insecure=True) logger_provider.add_log_record_processor(BatchLogR handler = LoggingHandler(level=logging.NOTSET, log logger = logging.getLogger('main') logger.addHandler(handler) Create log records # Create logs using the logger. Examples can be found in ListRecommendations and get_product_list functions.  logger.info(f"Receive ListRecommendations for prod As you can see, after the initialization, log records can be created in the same way as in standard Python. OpenTelemetry libraries automatically add a trace ID and span ID for each log record and, in this way, enable correlating logs and traces. Notes # Logs for Python are still experimental, and some changes can be expected. The implementation in this service follows the Python log example . Last modified October 19, 2023: Support page-relative aliases (#3417) (7f46ec2d) Docs Ecosystem Status Community Blog Shipping Service This service is responsible for providing shipping information including pricing and tracking information, when requested from Checkout Service. Shipping service is built primarily with Tonic, Reqwest, and OpenTelemetry Libraries/Components. Other sub-dependencies are included in Cargo.toml. Depending on your framework and runtime, you may consider consulting Rust docs to supplement. You’ll find examples of async and sync spans in quote requests and tracking IDs respectively. The build.rs supports development outside docker, given a Rust installation. Otherwise, consider building with docker compose to edit / assess changes as needed. Shipping service source  Traces # Initializing Tracing # The OpenTelemetry SDK is initialized from main.  fn init_tracer() -> Result