Observability for a Delivery Pipeline, Not Just the App
This is the final part of Ship the Proof. The series built a pipeline that produces verifiable artifacts, defends them, and measures their delivery honestly. This part makes the pipeline itself observable, so the invariants the earlier parts established produce a signal when they break, instead of failing silently.
You can see every span your application emits in production and still be blind to your delivery pipeline. When a deploy stalls, when an attestation fails to verify, when a restore drill quietly stops running, there is often no signal at all, because nobody instrumented the pipeline the way they instrumented the app. The pipeline is software too. It deserves a signal catalog.
A catalog answers four questions for every signal
A signal catalog answers four questions for every signal worth emitting: what is it, which stage produces it, who consumes it, and what, if anything, pages someone. Writing that down forces a useful discipline. You discover the signals you assumed existed but do not, and you stop alerting on things no human acts on. The catalog is the method. The tools in the last column are just what each signal is grounded in.
Lean on a standard where one exists, and be honest where it does not
Most pipeline signals are application-defined: you emit them, and no specification standardizes them. That is fine, as long as you do not dress them up as standards. There is exactly one standardized slice here, and it is worth using. The OpenTelemetry CI/CD semantic conventions define a cicd.* namespace, including cicd.pipeline.run.duration and result attributes on pipeline spans (OpenTelemetry CI/CD metrics). Two honesty caveats: those conventions are at development status, not yet stable, and there is no deployment metric, so your DORA (DevOps Research and Assessment) keys derive from the pipeline signals plus deployment metadata you attach yourself.
Everything else in the catalog is application-defined but grounded in the standard or tool that motivates it. Where no industry threshold exists, the catalog says so rather than inventing one. “Set per service” is a real answer. A fabricated number is not.
The catalog
Each row maps a signal to the stage that emits it, the consumer who reads it, and the alert or SLO (service-level objective), if any. The last column is what the signal is grounded in, not a tool recommendation; substitute the equivalent in your own stack.
| Signal | Stage | Consumer | Alert / SLO | Grounded in |
|---|---|---|---|---|
cicd.pipeline.run.duration | build | dashboard | alert on sustained regression vs baseline; no standard SLO | OTel CI/CD |
| image content digest | build | audit | none; it is identity, not a measurement | OCI dist-spec |
| bill-of-materials generated (attached as referrer) | bill of materials | audit | alert if missing for a built digest | Syft |
| vulnerability scan result (+ attestation) | scan | dashboard / gate | block on findings above the severity gate | Grype |
| provenance attested | signing | admission input / audit | alert if missing for a promotable digest | SLSA v1.2 |
| test result attested | signing | admission input / audit | alert if absent for a promotable digest | in-toto |
| signing event (transparency-log entry) | signing | admission input / audit | alert on signing failure | cosign signing |
| attestation verification result (fail-closed) | promotion | gate / audit | block on verify failure or orphaned referrers | cosign verify |
| sync / health status | rollout | dashboard / DORA | alert on out-of-sync prod or degraded health | Argo CD |
| admission verification result (deny on missing attestations) | admission | gate / audit | block on admission deny | Kyverno |
| rollout abort (runtime fallback to prior stable) | rollout | alert / human | alert on abort; it is an incident signal | Argo Rollouts |
| revert to prior verified digest | recovery | DORA (recovery time) / audit | none; time tracked in DORA | Argo CD |
| restore-drill execution | maintenance | audit / dashboard | alert if drill overdue or last drill failed | SOC 2 criteria |
| re-scan result (as vuln DB updates) | maintenance | dashboard / alert | alert when re-scan surfaces findings above the gate | Grype |
That is not exhaustive, but it covers the shape. Notice how few rows carry a numeric SLO. Most are pass/fail gates or hygiene signals where “passing” is the only target, and the few rate metrics genuinely have no industry-standard threshold.
Three signals worth singling out
Most of the catalog is mechanical. Three signals repay extra attention because they catch failures the rest of your observability will not.
Gate duration. End-to-end pipeline timing lumps together build, review, and approval. Add a per-gate duration metric of your own, tagged with the gate name and the digest, and you can decompose lead time across the gates and point at the slow one. The validation that it is wired correctly: the per-gate durations plus the pipeline run duration should reconcile against the commit-to-healthy-deploy lead time for the same digest.
Attestation-verification result. This is the signal most teams skip and most regret. When you promote an image between registries, the attestations attached to it are separate referrer manifests, and a naive copy by digest leaves them behind. If you do not emit and alert on the verification result, the first you hear of an orphaned-referrer problem is a production admission denial during a deploy. Emit it at every promotion boundary, and alert on both verify failure and orphaned referrers, so you catch the gap at promotion rather than at the production gate.
Restore-drill execution. Recovery capability decays silently. A backup job that has been failing for three weeks looks identical to one that never runs, until the day you need it. Emit a signal each time the restore drill executes and records a result, and alert when the drill is overdue or the last run failed. The cadence is a local policy choice; the alert on “overdue or failed” is what turns an untested recovery plan into a tested one. It is also the signal an auditor asks for, because it is evidence that recovery was exercised rather than merely documented.
What to alert on, and what not to
The catalog’s most useful column is the alert column, because it forces the question: would a human act on this page? Build duration regressing against baseline, an attestation missing for a built digest, a signing failure, an admission deny, a rollout abort, a restore drill gone overdue: all yes, because each is a broken invariant someone has to fix.
The DORA rate metrics, by contrast, are trend signals, not pages. So is branch age, and the presence or absence of an AI-provenance trailer. Paging on a slow-moving ratio trains people to ignore the pager. Put those on a dashboard, review them on a cadence, and reserve the alert for the gates and invariants that have actually broken. An observability layer for your pipeline is only worth building if the alerts it produces are ones people act on. Everything else is a dashboard, and that is fine. The point is to know which is which before you wire the page.
That closes the series. Identify the release by its digest, let the branching model follow, make the evidence travel and get checked at admission, declare your build level and your bill of materials, defend against the mutable references that bit everyone in 2026, measure delivery honestly, record AI authorship as durable fact, and finally watch the pipeline that does all of it. None of these is exotic on its own. Together they are the difference between shipping code and shipping proof.