Observability🔗

RSigma is built on tracing plus the prometheus crate. Every meaningful event in the daemon and CLI lands on one of:

• A tracing event (info, warning, error, debug, or trace) on stderr.
• A Prometheus counter, gauge, or histogram exposed on /metrics (daemon only).

This page covers the four observability surfaces you actually operate on: the log subscriber and its format, the RUST_LOG filter targets that surface specific concerns (NATS lifecycle, dynamic sources, hot-reload, HTTP requests, correlation memory pressure), what to scrape with Prometheus, and how to recognize the most useful tracing spans.

Log format and verbosity🔗

The daemon always emits structured JSON to stderr. The other commands (engine eval, rule lint, rule validate, backend convert, pipeline resolve) default to human-readable stdout/stderr output with no structured logs. Opt into a tracing subscriber for them with --log-format:

rsigma --log-format json rule validate rules/ -p pipelines/ecs.yml
rsigma --log-format text engine eval -r rules/ -e @events.ndjson

--log-format adds the diagnostic-log stream alongside the existing stdout/stderr output; it never replaces them. So rsigma --log-format json engine eval ... still prints the MatchResult lines to stdout exactly as before; the JSON log lines arrive on stderr.

Verbosity is controlled by the standard RUST_LOG environment variable (tracing_subscriber::EnvFilter). The default is info. The flag has no effect on engine daemon, which is always JSON.

RUST_LOG filter targets🔗

Every emitted event carries a target field naming the module that produced it. Use that to narrow RUST_LOG to the area you care about:

A few combinations that come up often in practice:

# Quiet production daemon: only warnings and above, but keep INFO for the
# daemon's own lifecycle messages so the boot sequence stays readable.
RUST_LOG="warn,rsigma::daemon=info" \
    rsigma engine daemon -r rules/

# Trace a hot-reload that is not picking up a rule change.
RUST_LOG="info,rsigma::daemon::reload=debug,rsigma_runtime::engine=debug" \
    rsigma engine daemon -r rules/

# Investigate a dynamic source that is timing out.
RUST_LOG="info,rsigma_runtime::sources=debug" \
    rsigma engine daemon -r rules/ -p pipelines/dynamic.yml

# HTTP access log on the daemon API.
RUST_LOG="info,tower_http=debug" \
    rsigma engine daemon -r rules/ --input http

Spans🔗

A tracing span is a structured scope around a unit of work. When the daemon resolves dynamic sources during a rule load, the span tree looks like this in the JSON output:

{
  "timestamp": "...",
  "level": "DEBUG",
  "fields": {"message": "Source fetched successfully", "source_id": "ips"},
  "target": "rsigma_runtime::sources",
  "span": {"rules_path": "/tmp/obs-test/rules", "name": "load_rules"},
  "spans": [{"rules_path": "/tmp/obs-test/rules", "name": "load_rules"}]
}

The span and spans fields tell you the call stack that produced the event without needing distributed tracing infrastructure. The named spans currently emitted:

Spans are emitted alongside events. To capture them in a structured aggregator (Loki, Datadog Logs, ClickHouse, etc.), index on the span.name field as well as target and level.

Prometheus metrics🔗

The daemon binds /metrics on the same --api-addr as the REST API. It exposes 27 metric definitions across three concerns:

Some metrics only appear after their first relevant event (per-rule labels, OTLP counters). A startup /metrics scrape shows about 20 distinct metric names; the full 27 emerge as the daemon does real work.

Scrape /metrics at 15-30 s intervals. The histograms (event_processing_seconds, pipeline_latency_seconds, batch_size) use the default Prometheus bucket boundaries; alert on the _bucket{le="..."} quantiles you care about rather than on the raw average.

A minimal scrape config:

scrape_configs:
  - job_name: rsigma
    scrape_interval: 15s
    static_configs:
      - targets: ['rsigma.internal:9090']

The full table with every label and source-of-truth pointer lives in the Prometheus metrics reference.

Useful alerting recipes🔗

These four alerts catch most operational regressions for free.

groups:
  - name: rsigma
    rules:
      # Engine is unable to keep up with the source.
      - alert: RsigmaBackPressure
        expr: rate(rsigma_back_pressure_events_total[5m]) > 0
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: rsigma is back-pressuring the input

      # Correlation state heading toward the hard cap (default 100k).
      - alert: RsigmaCorrelationStatePressure
        expr: rsigma_correlation_state_entries > 80000
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: rsigma correlation state above 80% of the hard cap

      # DLQ getting events means upstream is sending unparseable data.
      - alert: RsigmaDlqVolume
        expr: rate(rsigma_dlq_events_total[5m]) > 1
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: rsigma is routing events to the dead-letter queue

      # Reload-failure rate. Rules path issues, pipeline parse errors.
      - alert: RsigmaReloadsFailing
        expr: rate(rsigma_reloads_failed_total[5m]) > 0
        for: 10m
        labels: {severity: critical}
        annotations:
          summary: rsigma rule reload is failing

Health probes🔗

For Kubernetes-style orchestrators:

/healthz is intentionally cheap and side-effect-free; do not rely on it to detect "the engine is silently dropping events". Use rsigma_back_pressure_events_total, rsigma_dlq_events_total, and rsigma_reloads_failed_total for that.

OpenTelemetry receivers🔗

OTLP is one of the supported input formats for the daemon (with the daemon-otlp feature). RSigma does NOT export traces of its own internal work over OTLP; the OTLP wiring is one-way and is for receiving log records from agents.

If you want to ship the daemon's tracing events into a tracing backend, the standard tracing-opentelemetry Rust bridge would be the path, but no first-party integration ships today. The structured JSON log stream is the canonical observability surface; pipe it into Loki, Vector → ClickHouse, Datadog Logs, or any equivalent.

Quick verification🔗

# Confirm the metrics endpoint is alive.
curl -s http://127.0.0.1:9090/metrics | head -20

# Confirm structured-log emission with the daemon target.
rsigma engine daemon -r rules/ 2>&1 | head -3

The first line of /metrics should be a # HELP rsigma_back_pressure_events_total ... block. The first daemon log line should be a Rules loaded event with target=rsigma::daemon::server. If either is missing, the build is probably without the daemon feature or with a broken --api-addr.

See also🔗

• Streaming Detection for the daemon's HTTP API surface that complements the metrics endpoint.
• Performance Tuning for which metric to watch when sizing --buffer-size, --batch-size, or correlation max_state_entries.
• NATS Streaming for the NATS-specific log targets (async_nats::connector).
• Prometheus metrics reference for the full 27-metric catalog.
• HTTP API reference for every endpoint exposed alongside /metrics.
• tracing filter syntax for the exact RUST_LOG directive grammar.