Autonomous alert triage pipeline
CData · Platform Engineering In productionAn LLM sitting inside the on-call path, fed by three heterogeneous alert producers. Flux, Datadog and Prometheus Alertmanager each emit a different payload shape; the pipeline normalises them into one envelope, enriches it, has Claude classify and summarise, then routes deterministically so only what matters reaches the engineer — with the reasoning attached.
- Fan-in across three alert sources — FluxCD reconciliation events, Datadog monitors and Prometheus Alertmanager — normalised into a single alert envelope so downstream stages never branch on producer
- Deterministic routing around a non-deterministic core: the model classifies and summarises; severity and destination are decided by code, so an LLM error cannot misroute or silently drop an alert
- Measured, not asserted: 333 executions over 11 days — 251 Datadog, 51 Alertmanager, 30 FluxCD — at 49s median enrichment latency and under $0.20 per alert, with an 88% prompt-cache hit rate
- Built for failure: bounded retries, structured output parsing, and graceful degradation to raw alerts when the model call fails