Universal Controller - Provided Metrics

System metrics are scraped through our Prometheus Metrics Web Service endpoint.

To enable Open Telemetry metrics, you must configure an Open Telemetry metrics exporter in uc.properties , otherwise, a noop meter provider is used.

To configure UC with optional metric labels, you can add the following to the uc.properties.

uc.prometheus.metrics.<metric-name>.optional_labels=<comma-delimited-optional-labels>

For example,

uc.prometheus.metrics.uc_history_total.optional_labels=agent_id,task_instance_exit_code
uc.prometheus.metrics.uc_task_instance_duration_seconds.optional_labels=agent_id

Metric Name	Description	Metric Type	Default Labels	Optional Labels
uc_history_total	Total number of task instances that completed with an end status of In Doubt, Start Failure, Confirmation Required, Cancelled, Failed, Skipped, Finished, or Success.	Counter	cluster_node_id task_type task_type_detailed task_instance_status	agent_id task_name security_business_services task_instance_exit_code
uc_task_instance_launch_total	Total number of task instances launched.	Counter	cluster_node_id task_type task_type_detailed	agent_id task_name, security_business_services
uc_task_instance_late_start_total	Total number of task instances that started late.	Counter	cluster_node_id task_type task_type_detailed	agent_id task_name, security_business_services
uc_task_instance_late_finish_total	Total number of task instances that finished late.	Counter	cluster_node_id task_type task_type_detailed	agent_id task_name, security_business_services
uc_task_instance_early_finish_total	Total number of task instances that finished early.	Counter	cluster_node_id task_type task_type_detailed	agent_id task_name security_business_services
uc_task_instance_duration_seconds	Task instance duration in seconds. 1, 2.5, 5, 10, 15, 30, 45, 60, 150, 300, 600, 900, 1800, 2700, 3600 The values are in seconds and translates to the following: 1s, 2.5s, 5s, 10s, 15s, 30s, 45s, 60s, 2.5m, 5m, 10m, 15m, 30m, 45m, 1hr You can customize the buckets by adding the following property to `uc.properties`. `uc.prometheus.metrics.uc_task_instance_duration_seconds.buckets`=1,2.5,5,10,15,30,45,60,150,300,600,900,1800,2700,3600.	Histogram	cluster_node_id task_type task_type_detailed	agent_id task_name security_business_services
uc_universal_event_total	Total number of Universal Events published by Universal Event Templates and Universal Template Event Templates.	Counter	cluster_node_id universal_event_name universal_template_name universal_extension_name The universal_template_name and universal_extension_name label values are non-empty for published local Universal Events (defined by a Universal Template Event Template) and empty for published global Universal Events (defined by a Universal Event Template).	agent_id task_name security_business_services

Standard Exports CPU Time Spent and Memory Usage

Metric Name	Description	Metric Type
process_cpu_seconds_total	Total user and system CPU time spent in seconds.	Counter
process_start_time_seconds	Start time of the process since unix epoch in seconds.	Gauge
process_open_fds	Number of open file descriptors.	Gauge
process_max_fds	Maximum number of open file descriptors.	Gauge
process_virtual_memory_bytes	Virtual memory size in bytes.	Gauge
process_resident_memory_bytes	Resident memory size in bytes.	Gauge

JVM Class Loading

Metric Name	Description	Metric Type
jvm_memory_objects_pending_finalization	The number of objects waiting in the finalizer queue.	Gauge
jvm_memory_bytes_used	Used bytes of a given JVM memory area.	Gauge
jvm_memory_bytes_committed	Committed (bytes) of a given JVM memory area.	Gauge
jvm_memory_bytes_max	Max (bytes) of a given JVM memory area.	Gauge
jvm_memory_bytes_init	Initial bytes of a given JVM memory area.	Gauge
jvm_memory_pool_bytes_used	Used bytes of a given JVM memory pool.	Gauge
jvm_memory_pool_bytes_committed	Committed bytes of a given JVM memory pool.	Gauge
jvm_memory_pool_bytes_max	Max bytes of a given JVM memory pool.	Gauge
jvm_memory_pool_bytes_init	Initial bytes of a given JVM memory pool.	Gauge
jvm_memory_pool_collection_used_bytes	Used bytes after last collection of a given JVM memory pool.	Gauge
jvm_memory_pool_collection_committed_bytes	Committed after last collection bytes of a given JVM memory pool.	Gauge
jvm_memory_pool_collection_max_bytes	Max bytes after last collection of a given JVM memory pool.	Gauge
jvm_memory_pool_collection_init_bytes	Initial bytes after last collection of a given JVM memory pool.	Gauge
jvm_gc_collection_seconds	Time spent in a given JVM garbage collector in seconds.	Summary
jvm_memory_pool_allocated_bytes_total	Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.	Counter
jvm_classes_currently_loaded	The number of classes that are currently loaded in the JVM.	Gauge
jvm_classes_loaded_total	The total number of classes that have been loaded since the JVM has started execution.	Counter
jvm_classes_unloaded_total	The total number of classes that have been unloaded since the JVM has started execution	Counter
jvm_buffer_pool_used_bytes	Used bytes of a given JVM buffer pool.	Gauge
jvm_buffer_pool_capacity_bytes	Bytes capacity of a given JVM buffer pool.	Gauge
jvm_buffer_pool_used_buffers	Used buffers of a given JVM buffer pool.	Gauge
jvm_info	VM version info.	Gauge
jvm_threads_current	Current thread count of a JVM.	Gauge
jvm_threads_daemon	Daemon thread count of a JVM.	Gauge
jvm_threads_peak	Peak thread count of a JVM.	Gauge
jvm_threads_started_total	Started thread count of a JVM.	Counter
jvm_threads_deadlocked	Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers.	Gauge
jvm_threads_deadlocked_monitor	Cycles of JVM-threads that are in deadlock waiting to acquire object monitors.	Gauge
jvm_threads_state	Current count of threads by state.	Gauge
jvm_memory_pool_allocated_bytes_created	Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.	Gauge