Panel | ||||||
---|---|---|---|---|---|---|
|
...
OpenTelemetry is an open-source project that standardizes the collection of telemetry data from software systems, making it easier for organizations to gain holistic visibility into their environments. By seamlessly integrating with various programming languages, frameworks, and cloud platforms, OpenTelemetry simplifies the instrumentation of applications, allowing developers and operators to collect rich, actionable data about their systems' behavior. The adoption of OpenTelemetry by software vendors and Application Performance Monitoring (APM) tools represents a significant shift in the observability landscape. OpenTelemetry has gained substantial traction across the industry due to its open-source, vendor-neutral approach and its ability to standardize telemetry data collection.
Many software vendors have started incorporating OpenTelemetry into their frameworks and libraries. Major cloud service providers like AWS, Azure, and Google Cloud have also embraced OpenTelemetry. In addition, many APM tools have integrated OpenTelemetry into their offerings. This integration allows users of these APM solutions to easily collect and visualize telemetry data from their applications instrumented with OpenTelemetry. It enhances the compatibility and flexibility of APM tools, making them more versatile in heterogeneous technology stacks.
Solution Architecture (Component Description)
...
- Linux Server
- Memory: 16GB RAM
- Storage: 70GB Net storage
- CPU: 4 CPU
- Distribution: Any major Linux distribution
- For the installation and configurations of the required Observability tools Administrative privileges are required
- Ports
The Following default ports will be used.
Application | Port |
---|---|
Prometheus | http: 9090 |
Grafana: | http:3000 |
Jaeger | http:16686 |
Elastic | http:9200 |
OTEL Collector | 4317 (grpc), 4318 (http) |
...
Note: This Startup Guide has been tested with the provide Software Version in the table below.
Software | Version | Linux Archive |
---|---|---|
elasticsearch | 7.17.12 | https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.12-linux-x86_64.tar.gz |
otelcol-contrib | 0.86.0 | https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.86.0/otelcol-contrib_0.86.0_linux_amd64.tar.gz |
jaeger all in one | 1.49.0 | https://github.com/jaegertracing/jaeger/releases/download/v1.49.0/jaeger-1.49.0-linux-amd64.tar.gz |
prometheus | 2.47.1 | https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz |
grafana-enterprise | 10.1.4 | https://dl.grafana.com/enterprise/release/grafana-enterprise-10.1.4.linux-amd64.tar.gz |
...
Elasticsearch is a distributed, RESTful search and analytics engine designed for real-time search and data storage. It is used for log and event data analysis, full-text search, and more.
In this set-up Elasticsearch is used as the storage backend for Jaeger.
Installation Steps:
Follow the official documentation to install Elasticsearch on your Linux Server.
...
Configuration File:
- elasticsearch.yml: Main configuration file for Elasticsearch, containing cluster, node, network, memory, and other settings.
No adjustments to the default elasticsearch.yml file are required.
Code Block | ||||
---|---|---|---|---|
| ||||
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # #cluster.name: my-application # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # #node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # By default Elasticsearch is only accessible on localhost. Set a different # address here to expose this node on the network: # #network.host: 192.168.0.1 # # By default Elasticsearch listens for HTTP traffic on the first free port it # finds starting at 9200. Set a specific HTTP port here: # #http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1", "host2"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # #cluster.initial_master_nodes: ["node-1", "node-2"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true # # ---------------------------------- Security ---------------------------------- # # *** WARNING *** # # Elasticsearch security features are not enabled by default. # These features are free, but require configuration changes to enable them. # This means that users don’t have to provide credentials and can get full access # to the cluster. Network connections are also not encrypted. # # To protect your data, we strongly encourage you to enable the Elasticsearch security features. # Refer to the following documentation for instructions. # # https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html |
...
Setup up Jaeger
Description:
Jaeger is an open-source distributed tracing system used for monitoring and troubleshooting microservices-based applications.
In this set-up Universal Controller will manually instrument Open Telemetry trace on Universal Controller (UC), OMS, Universal Agent (UA), and Universal Task Extension interactions associated with task instance executions, agent registration, and Universal Task of type Extension deployment.
...
Install the Version listed in chapter Required Software for the Observability.
For a quick local installation the Jaeger all-in-one executable can be used. It includes the Jaeger UI, jaeger-collector, jaeger-query, and jaeger-agent, with an in memory storage component.
Configuration:
When starting the jaeger-all-in-one application the following command-line argument need to be set:
--collector.otlp.grpc.host-port :14317
: This is a command-line argument passed to the Jaeger all-in-one binary to configure the host and port for the gRPC OTLP ( OpenTelemetry Protocol) endpoint. It specifies that the gRPC OTLP endpoint should listen on port 14317.--collector.otlp.http.host-port :14318
: This is another command-line argument that configures the host and port for the HTTP OTLP endpoint, specifying port 14318.
...
Setup OTEL Collector
Description:
OpenTelemetry Collector is a vendor-agnostic observability data collector that gathers traces, metrics, and other telemetry data from various sources and sends it to different backends for analysis.
In this set-up OpenTelemetry collects Metrics data from Universal Controller, Universal Agent, OMS and Universal Tasks of type Extension.
Installation Steps:
Follow the official documentation to install OpenTelemetry on your Linux Server.
Official Documentation: OpenTelemetry Collector Installation
...
- otel-collector-config.yaml, Primary configuration file, which specifies how the collector should receive, process, and export telemetry data.
Let's break down the key sections and settings in this configuration
- Receiver: For UAC the HTTP, GRPC receiver for the OpenTelemetry Collector needs to be configured. The HTTP port (4318) or GRPC port (4317) should match the ports configured in the 'omss.conf' and 'uags.conf' files.
- Exporters: The following Exporters are configured to send telemetry data to: logging, Prometheus, and Jaeger
- Pipelines: Two pipelines are configured:
- traces: otlp → batch → jaeger
- metrics: otlp → batch → prometheus
The following provides a sample otel-collector-config.yaml file.
Note: In the omss.conf and uags.conf file the port needs to be set to 4318. refer to Configuring OTLP Endpoint to configure the omss.conf and uags.conf port for OpenTelemetry.
Code Block | ||||
---|---|---|---|---|
| ||||
# otel-collector-config.yaml # the http port 4318 (default) or grpc port 4317 should be the same as in the omss.conf and uags.conf receivers: otlp: protocols: http: #tls: #cert_file: ue-cert.crt #key_file: ue-cert.key #endpoint: 0.0.0.0:4318 grpc: #endpoint: 0.0.0.0:4317 exporters: logging: verbosity: detailed prometheus: endpoint: 0.0.0.0:8881 #tls: #ca_file: "/path/to/ca.pem" #cert_file: "/path/to/cert.pem" #key_file: "/path/to/key.pem" #namespace: UAgent #const_labels: #label1: value1 #"another label": spaced value #send_timestamps: true #metric_expiration: 180m #enable_open_metrics: true #add_metric_suffixes: false resource_to_telemetry_conversion: enabled: true jaeger: endpoint: localhost:14250 tls: insecure: true processors: batch: service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [jaeger] metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] |
...
Set up Prometheus
Description:
Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It collects metrics from monitored targets, stores them, and provides powerful querying and alerting capabilities.
In this set-up Prometheus is used to store the Metrics data retrieved via Opentelemetry and the Universal Controller Metrics REST API.
Installation Steps:
Follow the official documentation to install Prometheus on your Linux Server.
Official Documentation: Prometheus Installation Guide
...
Configuration Files:
- prometheus.yml: Main configuration file for Prometheus, defining scrape targets (what to monitor), alerting rules, and other settings.
In the prometheus.yaml configuration file for UAC the following scrape jobs are defined what metrics to collect and from where.
- OTelCollector' job, Prometheus collects metrics from the target
127.0.0.1:8881
, which corresponds to the OpenTelemetry Collector's OTLP endpoint. - prometheus job, with the the
/metrics
endpoint provides internal metrics related to the Prometheus server's performance and resource utilization. - controller job, Prometheus collects data via the Universal Controller Webservice Metrics API. Replace
'ops.admin'
and'xxx'
with the actual username and password required to access the metrics.
Code Block | ||||
---|---|---|---|---|
| ||||
# my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. # metrics_path defaults to '/metrics' # scheme defaults to 'http'. - job_name: 'OTelCollector' static_configs: - targets: ["127.0.0.1:8881"] #- job_name: "OTelCollector" #static_configs: #- targets: ["127.0.0.1:8888"] - job_name: 'prometheus' metrics_path: /metrics static_configs: - targets: - localhost:9090 - job_name: 'controller' basic_auth: username: 'ops.admin' password: 'canton123' metrics_path: '/uc/resources/metrics' # The correct path where the metrics are exposed by the web service. Note for cloud controller: '/resources/metrics' static_configs: - targets: - 'localhost:8080' # Use the correct hostname or IP address and port of your web service. |
...
Code Block | ||
---|---|---|
| ||
# Check default Port: ss -tuln | grep 9090 # Result: tcp LISTEN 0 128 *:9090 *:* |
Test that the prometheus GUI is accessibly: http://<hostname>:9090
...
Set up Grafana
Description:
Grafana is an open-source platform for monitoring and observability that allows you to create, explore, and share dynamic dashboards and visualizations for various data sources, including time series databases.
In the this set-up Grafana is used to Visualize and Analyze the Metrics data store in Prometheus data source (time series database).
Installation Steps:
Follow the official documentation to install Grafana on your Linux Server.
Install the Version listed in chapter Required Software for the Observability.
...
- Linux Server
- Memory: 16GB RAM
- Storage: 70GB Net storage
- CPU: 4 CPU
- Distribution: Any major Linux distribution
- For the installation and configurations of the required Observability tools, Administrative privileges are required
- Ports
The Following default ports will be used.
Application | Port |
---|---|
Prometheus | http:9090 |
Grafana | http:3000 |
OTEL Collector | 4317 (grpc), 4318 (http) |
...
This Tutorial has been tested with the provided Software Versions in the table below.
Software | Version | Linux Archive |
---|---|---|
otelcol-contrib | 0.82.0 | https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.82.0/otelcol-contrib_0.82.0_linux_amd64.tar.gz |
prometheus | 2.47.1 | https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz |
grafana-enterprise | 10.1.4 | https://dl.grafana.com/enterprise/release/grafana-enterprise-10.1.4.linux-amd64.tar.gz |
...
The following additional ports need to be opened:
Application | Port |
---|---|
Elastic Search | http:9200 |
Jaeger | http:16686 |
...
This tutorial requires the following software, installed during the first Tutorial.
Software | Version | Linux Archive |
---|---|---|
Universal Controller | 7.5.0.0 or higher | Download via Stonebranch Support Portal |
Universal Agent | 7.5.0.0 or higher | Download via Stonebranch Support Portal |
otelcol-contrib | 0.82.0 | https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.82.0/otelcol-contrib_0.82.0_linux_amd64.tar.gz |
prometheus | 2.47.1 | https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz |
grafana-enterprise | 10.1.4 | https://dl.grafana.com/enterprise/release/grafana-enterprise-10.1.4.linux-amd64.tar.gz |
...
To add support for traces, the following Opensource Software needs to be installed.
Software | Version | Linux Archive |
---|---|---|
elasticsearch | 7.17.12 | https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.12-linux-x86_64.tar.gz |
jaeger | 1.47.0 | https://github.com/jaegertracing/jaeger/releases/download/v1.47.0/jaeger-1.47.0-linux-amd64.tar.gz |
...