Observability

When something goes wrong — a handler is slow, a topic is backing up, a retry loop is burning through the budget — you need to know where to look. shove integrates with the standard Rust tracing ecosystem. Every interesting event — handler invocation outcomes, retry routing, DLQ routing, group scaling, autoscaler decisions, connection errors — is emitted as a structured tracing event with named fields. Add a subscriber and you have a full observability trail without any instrumentation code in your handlers.

For dashboards and alerting, shove also emits operational metrics through the metrics facade. See Metrics below.

Metrics

shove is a library, not a service, so it does not expose its own scrape endpoint. Instead it emits operational metrics through the metrics facade crate. The consuming service installs a recorder of its choice — metrics-exporter-prometheus, metrics-exporter-statsd, OpenTelemetry, etc. — and exposes the endpoint itself.

Enabling

Add the metrics feature in your Cargo.toml:

[dependencies]
shove = { version = "0.x", features = ["rabbitmq", "metrics"] }
metrics = "0.24"
metrics-exporter-prometheus = "0.16"

Recorder setup

A minimal Prometheus exporter that exposes /metrics on :9100:

use metrics_exporter_prometheus::PrometheusBuilder;
use std::net::Ipv4Addr;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    PrometheusBuilder::new()
        .with_http_listener((Ipv4Addr::UNSPECIFIED, 9100))
        // Histogram bucket recommendations for shove's two duration histograms.
        .set_buckets_for_metric(
            metrics_exporter_prometheus::Matcher::Suffix(
                "_duration_seconds".to_string(),
            ),
            &[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
        )?
        .install()?;

    // ... your shove broker / publisher / supervisor setup ...
    Ok(())
}

Custom prefix

By default every metric name starts with shove_. To override (for example to namespace under your service name) call shove::metrics::set_prefix once before installing the recorder:

shove::metrics::set_prefix("billing_shove");
// ... then install your recorder ...

Important: set_prefix must be called before any metric emission, not just before installing the recorder. The metric-name cache is materialised on first use, so any backend or publisher activity locks in the default shove prefix; calling set_prefix after that point panics rather than silently produce mis-named metrics. The prefix string itself must match Prometheus' name grammar ([a-zA-Z_][a-zA-Z0-9_]*); hyphens or other special characters produce invalid metric names that the exporter will reject.

Metric reference

• outcome on messages_consumed_total and message_processing_duration_seconds: ack, retry, reject, defer. These mirror the Outcome enum returned by handlers.

• outcome on messages_published_total and message_publish_duration_seconds: success, error.

• reason on messages_failed_total:

  reason	When	Emitted by
  oversize	Payload exceeds max_message_size	all
  deserialize	Payload present, codec rejected it	all
  malformed	The delivery is not a well-formed shove message at all — a Redis stream entry with no payload field. Points at a foreign writer or a non-shove publisher, which is a different fix from deserialize	Redis
  pending_full	The in-order routing buffer is full	RabbitMQ
  schema_frame / schema_validation	Schema-registry frame decode / subject-resolve or validation failure	Kafka (kafka-schema-registry)
  timeout	A single handler exceeded handler_timeout	all
  max_retries_exceeded	Retries exhausted, message dead-lettered	all
  rejected	Handler returned Reject	all
  sequence_timeout	A sequence key sat in AwaitingRetry past hold_queue_timeout, so the messages buffered behind it were dead-lettered to unblock the key. Deliberately separate from timeout, which is a handler-latency problem, not a stuck-ordering-key one	RabbitMQ

  Everything above timeout happens before the handler runs; timeout, max_retries_exceeded and rejected are emitted after it runs. sequence_timeout is the odd one out: those messages never reach the handler either, but they are retired by an ordering decision rather than by anything about the message itself.

  Two discard classes are deliberately not counted as failures. On a sequenced topic using SequenceFailure::FailAll, only the message that actually failed is counted — the messages dead-lettered behind its poisoned key are collateral of that one already-counted failure, and counting them would scale the counter by queue depth. And the in-process backend's shutdown drops are log-only, because it has no durability to begin with and counting only the drops visible to consumer code would under-report by an unbounded amount. Both classes warn with the queue (or shard), the message id, and the sequence key where there is one.

  The cascade exclusion applies to messages_failed_total only. A cascaded message dropped with no DLQ is just as gone as any other, so it still increments messages_discarded_total — see below.

  malformed is Redis-only in practice, which is why SQS is not listed against it. The SQS sequenced consumer carries the same guard for a message with no MessageGroupId, but declare creates sequenced shard queues as FIFO and SQS rejects a send with no group id outright, so the guard cannot fire on the supported topology. It is kept deliberately, against a future non-FIFO sequenced transport. Do not build an SQS alert on malformed.

• reason on messages_discarded_total: the same values, minus the ones that cannot be terminal on the backend in question. This counter increments in addition to messages_failed_total, only when the message was actually retired with nowhere to go — the topic declares no DLQ, or a DLQ was declared and the publish to it failed. Any non-zero rate is data loss. It is not a strict subset of messages_failed_total: a FailAll cascade adds a discard without adding a failure. See Alerting on silent discards.

• direction on autoscaler_decisions_total: up, down, hold.

• shove_autoscaler_messages_ready, shove_autoscaler_messages_in_flight, and shove_autoscaler_active_consumers are emitted once per group on every autoscaler poll, carrying only the consumer_group label. They expose the raw signal the scaling decision is made from: backlog waiting, messages in flight, and the live consumer count. Watching messages_ready stay high while active_consumers is flat at max is the canonical "saturated / scaling commands not keeping up" signal. If you do not autoscale, these go silent — use shove_queue_backlog instead, see Queue depth without the autoscaler.

• backend on backend_errors_total: inmemory, rabbitmq, kafka, nats, sns_sqs, redis.

• kind on backend_errors_total: connection, publish, consume, topology, ack.

• topic: the queue name from the topology (QueueTopology::queue()) — including on the dedicated DLQ-drain consumers (run_dlq), which label their samples with the source topic rather than the DLQ name, so a topic's main-path and DLQ-path series stay summable under one topic. To separate them, filter on consumer_group instead.

• consumer_group: the group name set by a coordinated group registry, set explicitly with ConsumerOptions::with_consumer_group (or BatchConsumerOptions::with_consumer_group on any Broker::batch_consumer() — the label is the same logical group name on every entry point, generic or backend-specific), or "default" when the consumer was registered through ConsumerSupervisor directly. It is always that logical name, never a backend's own group identifier — on Kafka in particular it is not the group.id, which stays in the consumer's startup log line and the broker's own tooling. Every consume entrypoint reports it the same way, so sum by (consumer_group) over a topic consumed with run, run_fifo and run_batch returns one series, not three. A dedicated DLQ drain follows the same rule: it reports "default" unless it was given a group explicitly, never the internal {dlq}-consumer id it joins under on Kafka. A broadcast subscription reports whatever its ConsumerOptions carried, and "default" otherwise, so name its group explicitly if you want its series separable from the topic's other readers.

A pre-handler rejection is counted once, under its precise reason. A payload rejected by the oversize gate is counted as oversize and not additionally as rejected when it is retired; the same holds for deserialize, malformed, and the schema-registry reasons. That is what makes the reason label trustworthy: oversize means the gate fired, not "something failed somewhere upstream of the DLQ".

shove_message_size_bytes is recorded on receipt, before the max_message_size gate, on every consumer backend. A payload rejected as oversize therefore still contributes a sample — deliberately, because this histogram is what you size max_message_size against, and one showing only the messages already under the limit could never tell you the limit was too low. The rejection itself is counted separately, as shove_messages_failed_total{reason="oversize"}. Like that counter, this histogram measures deliveries, not distinct messages: a message that is retried contributes one sample per delivery.

shove_messages_failed_total counts failed attempts, not retired messages — do not read its rate as a count of messages lost or dead-lettered. Two cases make one message contribute more than one increment:

• A handler timeout is counted every time it happens. timeout is recorded when the deadline fires, whatever the timeout resolves to. With the default Retry (or Defer) the message is redelivered and survives, so a handler that times out on five attempts contributes five increments and may still eventually Ack.
• A timeout that resolves to a terminal outcome is counted twice. with_handler_timeout_outcome(Outcome::Reject) records timeout for the deadline and then rejected for the retirement — and on run_batch the second is recorded once per message in the batch.

If you want retired-without-ack, the counter you want is shove_messages_discarded_total (messages that were dropped with nowhere to go) plus your DLQ's own depth or ingest rate — not a rate over messages_failed_total.

shove_backend_errors_total covers the most operationally-meaningful error sites (connection drops, topology conflicts, broker NACKs, consume-stream closures, ack failures); not every possible internal Err propagation produces a counter increment. Treat the counter as a "something is wrong with backend X" signal rather than an exhaustive error count.

shove_messages_published_total increments once per message in a batch publish, matching what consumers see downstream. shove_message_publish_duration_seconds records one sample per batch (the user-observable call latency). Empty batches are no-ops and emit no events.

Queue depth without the autoscaler

Backlog used to be a side effect of autoscaling: shove_autoscaler_messages_ready is set once per group on every scaling poll, so a service that runs a fixed consumer pool — or scales on something other than shove — had no backlog series at all.

Broker::queue_depth_sampler() publishes it directly. Name the queues to watch and drive it alongside your consumers:

let sampler = broker
    .queue_depth_sampler()
    .watch_topic::<OrdersTopic>()
    .watch_topic::<InvoicesTopic>()
    .with_poll_interval(Duration::from_secs(15));
tokio::spawn(sampler.run(shutdown.clone()));

That emits, once per queue per poll:

Five things worth knowing before you build a dashboard on these:

• shove_queue_inflight is not shove_messages_inflight. The queue gauge is what the broker has handed out and is waiting to have acked, across every consumer of that queue. shove_messages_inflight is what this process has inside a handler right now. On a healthy single-consumer service they track each other; a large gap means deliveries are sitting unacked somewhere else.
• shove_queue_inflight is absent on Kafka. Committed-offset lag cannot separate "fetched and being processed" from "not fetched", and Kafka exposes no group-wide in-flight count, so shove publishes no series rather than a hard zero you might alert on. Backlog is unaffected.
• A failed poll publishes nothing. The gauge keeps its last value rather than dropping to zero, so a broker outage does not render as "the backlog drained". Pair any backlog alert with shove_backend_errors_total and the sampler's WARN line so a flat gauge is distinguishable from a quiet queue.
• On Kafka the sampler reads the default group. Backlog is committed-offset lag, and lag is a property of a consumer group, so the sampler has to name one. It always names {queue}-consumer with auto.offset.reset=earliest. A group reached any other way reports its own lag through shove_autoscaler_messages_ready but not through shove_queue_backlog: a with_group_id override, a for_consumer_group fan-out group ({queue}-{group}-consumer), and a sequenced topic's FIFO group ({queue}-fifo) all fall outside it. Where the two disagree, the autoscaler gauge is the one describing the group you are actually running.
• Every poll is a real round trip (RabbitMQ Management HTTP, SQS GetQueueAttributes, Kafka watermark fetches). The default 5 s interval matches the autoscaler's; raise it if you watch many queues. The first poll fires after one interval rather than immediately, so a sampler spawned alongside declare does not race it — call sample_once() yourself first if you want a reading at t=0.

Running both the sampler and the autoscaler is fine: they read the same per-backend snapshot, and on every backend but Kafka they report the same number for the same queue. The shove_autoscaler_* gauges stay the right ones for why a scaling decision was made, keyed by consumer_group and including active_consumers; shove_queue_backlog is the right one for how deep the queue is, keyed by topic so it joins the rest of the consumer metrics.

Alerting on silent discards

A topic declared with a bare TopologyBuilder::new("...").build() has no DLQ. When such a message exhausts its retry budget — or a handler returns Outcome::Reject — there is nowhere to route it, so shove drops it. This is the intended behaviour for a topology that opted out of a DLQ, but it is easy to arrive at by accident, and until shove_messages_discarded_total existed the only evidence was a WARN log line.

Alert on any discard at all:

sum(rate(shove_messages_discarded_total[5m])) by (topic, reason) > 0

Every increment is a message that no longer exists anywhere. The fix is almost always to give the topic a DLQ:

TopologyBuilder::new("prices").dlq_named("prices-dlq").build()

Four limits worth knowing:

• Coverage is per backend. Kafka, RabbitMQ and the in-memory backend count every drop on the consume path — retry-budget exhaustion, explicit rejects, poisoned sequence keys, and pre-handler oversize/deserialize rejections (single-message and batch alike; on Kafka a pre-handler discard settles against the commit that actually retires the offset, exactly like its terminal-outcome path). NATS and Redis Streams count the terminal-outcome path only; their pre-handler rejections route to the DLQ separately and are not counted yet — on those two backends, treat the counter as a lower bound on dropped messages.
• The DLQ drain loop is not covered, on any backend. A dead message that run_dlq cannot decode — or that fails its size gate — is acknowledged and dropped with no discard recorded, and a drained message never re-enters a DLQ, so that drop is final. The only evidence is a WARN log line. If you run typed DLQ drains against topics whose dead messages can be undecodable (a poison payload usually is, by definition), do not read a flat shove_messages_discarded_total as "nothing is being lost in the drain".
• SNS/SQS never increments it, by design. SQS's terminal path deletes nothing: it sets the message's visibility timeout to zero, which hands it back to the queue. Whether it then reaches a DLQ is decided by the queue's AWS-side redrive policy, which shove does not own — and with no redrive policy the message cycles until the retention period expires rather than being discarded. Counting that as a discard would be wrong twice: the message still exists, and its receive count stays above the budget, so every later receive would count the same message again. On SQS, alert on the WARN line and on shove_messages_failed_total{reason="max_retries_exceeded"} instead, and configure a redrive policy on the queue.
• A slow handler that keeps timing out will burn retry budget and eventually land here. If that is the failure mode you are seeing, ConsumerOptions::with_handler_timeout_outcome lets a timeout resolve to something other than Retry so a slow consumer stops consuming budget — see Handlers.

Histogram buckets

shove does not configure histogram buckets. Set them at the recorder, as shown above. Reasonable starting points:

• shove_message_processing_duration_seconds: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
• shove_message_publish_duration_seconds: same as processing — most publishes are sub-100 ms but tail latency under broker stress is the interesting region.
• shove_message_size_bytes: [256, 1024, 4096, 16384, 65536, 262144, 1048576, 10485760] — covers the path from tiny envelopes up to the default 10 MiB ceiling.

tracing integration

The recommended subscriber setup for a consumer binary:

use tracing_subscriber::{EnvFilter, fmt};

tracing_subscriber::registry()
    .with(EnvFilter::from_default_env())
    .with(fmt::layer())
    .init();

Run with RUST_LOG=info,shove=debug to get per-message delivery traces. For production, use RUST_LOG=info,shove=info to get group scaling events and errors without per-message noise.

Available log levels used by shove:

Key events emitted

shove does not use structured span names in the tracing::span!() form. All observability is through event macros (debug!, info!, warn!, error!) with named fields. The following are the most useful events to watch for, with their actual field names from the source:

• "handler task panicked — retrying message" (warn) — a handler future panicked. Fields: error, ticket. The message is retried.
• "handler timed out — retrying" (warn) — handler exceeded handler_timeout. Fields: timeout. The message is retried.
• "rejecting oversized message" (warn) — payload exceeds max_message_size. Fields: error. Routed to DLQ.
• "failed to deserialize message — rejecting" (warn) — JSON deserialization failed. Fields: error. Routed to DLQ.
• "message handled (concurrent)" (debug) — a concurrent consumer processed a message. Fields: queue, outcome.
• "message handled (concurrent-sequenced)" (debug) — a concurrent-sequenced consumer processed a message. Fields: queue, sequence_key, outcome.
• "DLQ declared but not found in broker" (error) — topic has a DLQ in the topology but the broker queue was not found. Fields: queue. Indicates a topology declaration failure.

• "starting consumer group" (info) — group starts its initial consumers. Fields: group, queue, initial_consumers.
• "scaled up: spawned new consumer" (info) — autoscaler or manual call added a consumer. Fields: group, consumers.
• "scaled down: cancelled an idle consumer" (info) — autoscaler or manual call removed a consumer. Fields: group, consumers.
• "scale_up rejected: at max capacity" (debug) — scale-up attempted but already at max_consumers. Fields: group, max.
• "scale_down rejected: at min capacity" (debug) — scale-down attempted but already at min_consumers. Fields: group, min.
• "scale_down rejected: all consumers are busy" (warn) — tried to scale down but every consumer is processing a message. Fields: group.
• "shutting down consumer group" (info) — drain started. Fields: group, consumers.

• "autoscaler started" (info) — autoscaler loop entered.
• "autoscaler shutting down" (info) — shutdown token was cancelled.
• "failed to list groups: {e}" (error) — autoscaler could not enumerate consumer groups (backend error).
• "failed to fetch metrics for {group}: {e}" (error) — metrics fetch failed for one group; others continue.
• "failed to scale {group}: {e}" (error) — scaling command failed.

• "audit handler failed, retrying message" (error) — AuditHandler::audit() returned Err. Fields: error, delivery_id. The message will be retried.
• "audit handler timed out, returning original outcome" (error) — audit exceeded audit_timeout. Fields: delivery_id, timeout_ms. Original outcome is preserved.

• "drain timeout elapsed; aborting surviving tasks" (warn) — run_until_timeout deadline elapsed. Fields: timeout_ms.

Header propagation

MessageMetadata::headers is a HashMap<String, String> carrying broker-level headers that survive hold-queue hops. Notable headers:

• x-trace-id — used by Audited as the trace_id on audit records. Publishers do not set it automatically; if absent on delivery, Audited generates a fresh UUID per message. Set it explicitly via publish_with_headers (shown below) when you want a trace ID that connects to upstream/downstream systems. Once set, it is preserved across retries — hold-queue hops forward headers.
• x-shove-retry-count — the internal retry counter maintained by shove's consumer routing layer.
• Backend-specific identifiers (see the table below).

To set x-trace-id at publish time:

use std::collections::HashMap;

let mut headers = HashMap::new();
headers.insert("x-trace-id".to_string(), your_trace_id.to_string());

publisher.publish_with_headers::<Orders>(&msg, headers).await?;

The consumer surfaces this via metadata.headers.get("x-trace-id") in the handler. The audit wrapper reads it automatically.

Connecting to OpenTelemetry

Use the tracing-opentelemetry bridge to route shove's structured events into your OTel pipeline:

[dependencies]
tracing-opentelemetry = "0.x"
opentelemetry = "0.x"

use tracing_subscriber::layer::SubscriberExt;
use tracing_opentelemetry::OpenTelemetryLayer;

let tracer = init_otel_tracer(); // your OpenTelemetry tracer setup
let otel_layer = OpenTelemetryLayer::new(tracer);

tracing_subscriber::registry()
    .with(EnvFilter::from_default_env())
    .with(fmt::layer())      // keep local logging too
    .with(otel_layer)        // forward to OTel
    .init();

All named fields on shove events (group, queue, outcome, error, timeout_ms, etc.) become OTel span attributes automatically via the bridge. Scale events, panics, and audit failures all become searchable attributes in your OTel backend (Jaeger, Tempo, Honeycomb, etc.).

Per-backend identifier headers

Each backend stamps deliveries with a stable per-message identifier. Access these through MessageMetadata::headers:

For RabbitMQ specifically: x-message-id is stamped by RabbitMqPublisher on every outgoing message. Handlers can read it for deduplication:

if let Some(mid) = metadata.headers.get("x-message-id") {
    if store.already_processed(mid).await? {
        return Outcome::Ack;
    }
    store.mark_processed(mid).await?;
}
// ... business logic ...

Audit as observability

Audit records are a high-resolution observability stream complementary to tracing events. Every delivery produces one record with the full payload, outcome, duration in milliseconds, and trace ID. Where tracing events are best for operational monitoring (scale events, error rates), audit records are best for business-level visibility: which messages were processed, what the outcome was, how long each one took.

Wire ShoveAuditHandler (see Audit Logging) and consume the shove-audit-log topic to build:

• Per-topic outcome histograms (ratio of Ack / Retry / Reject).
• Handler latency percentiles (p50 / p99 from duration_ms).
• Per-entity message history (all records for a given trace_id or entity ID in the payload).
• Alerting on sustained rejection spikes.

What's next

• Audit Logging
• Handlers & Context
• Shutdown & Exit Codes