Observability

FourEyes is observable two ways out of the box: a Prometheus pull endpoint and an OpenTelemetry (OTLP) push exporter — “one source of truth, two transports.”

Prometheus /metrics

A third flush-drain tap (alongside the alert engine and the live SSE stream) subscribes to the result pipeline and folds each flushed batch into in-memory state. At scrape time a custom collector snapshots that state and emits const metrics with dynamic labels:

• foureyes_probe_results_total{org_id,test_id,node_id,test_type,success} (counter)
• foureyes_probe_success / ..._latency_ms / ..._loss_pct (latest gauges)
• foureyes_probe_path_rtt_ms{stat}, ..._http_timing_ms{phase}, ..._bgp_up, ..._bgp_routes{afi}
• Operational gauges: ingestion_buffer_depth, ingestion_flush_lag_ms, alerts_firing, nodes_total{status}, tests_total{type}

Plus the standard Go-runtime and process collectors. Mounted at /metrics on the root mux (next to /healthz).

OpenTelemetry export

The same Prometheus registry is bridged into an OTel metric producer and pushed on a timer to an OTLP/gRPC receiver — so pull and push see identical state, with zero duplicated metric logic.

Traces ship too: one probe.round span per flushed result, plus a probe.step child span per transaction step. The node stays trace-unaware; spans use the result’s own timestamps.

Configuration

OTLP push is disabled by default (FOUREYES_OTEL_ENDPOINT empty → no exporter, no goroutine). Enable it with:

Variable	Default	Description
FOUREYES_OTEL_ENDPOINT	(empty)	OTLP/gRPC target, e.g. localhost:4317. Empty = disabled.
FOUREYES_OTEL_EXPORT_INTERVAL	30s	Periodic push cadence.
FOUREYES_OTEL_INSECURE	true	Plaintext gRPC (dev collector); set false for TLS.
FOUREYES_OTEL_HEADERS	(empty)	k=v,k=v auth/tenant headers for hosted backends.

Both Prometheus and OTLP are pure-Go dependencies — the CGO-free build thesis is intact. See the full variable table on the [Configuration](/reference/configuration/) page.