CI said the deploy succeeded

There is a moment in every incident where someone asks “wait, is the fix even deployed?” and the room goes quiet. Somebody pulls up the CI dashboard. The pipeline is green. The deploy step says success. Everyone relaxes. And then, sometimes, the incident continues — because the green checkmark was never a statement about production. It was a statement about a pipeline.

Zero — b0gy’s platform for engineering truth — did not start as a resource inventory or a cost tool. It started with that quiet moment. The first thing we built was a tool called Landed, and it answered exactly one question: which commit is live, where — and is it really.

This is part 2 of a three-part series on building Zero. Part 1 covered the keyless connector model — why the platform holds no cloud credentials. This part is about the feature that turned out to be the spine of everything else.

“CI succeeded” is a claim about the wrong system

When a pipeline reports a successful deploy, it is telling you that a series of steps ran without a non-zero exit code. The image built. The push returned 200. The rollout command didn’t error. That is genuinely useful information. It is also not the same as “the commit you pushed is the commit serving traffic.”

The gap between those two things is where a lot of confused hours go. A rollout that silently kept the old revision because the new one failed its health check. A deploy that went to the wrong environment. A pipeline that reported success on a service that then crash-looped and got rolled back by the orchestrator — while your dashboard still shows green. An image tag that resolved to a stale digest. In each case the pipeline did its job and reported honestly on what it did. It just wasn’t watching production. Nothing was.

Most teams answer “what’s live?” by reading their CI history. That is reading the intent, not the outcome. Landed exists because the outcome deserves its own source of truth.

How Landed actually works

The design principle was that this had to work with zero application-code changes. Nobody edits a Dockerfile or adds a middleware to get deploy truth. So there are three independent ways the platform learns what shipped, and it reconciles them.

Ingest from any CI. There’s a single idempotent, fast-ack endpoint — POST /v1/deployments, authenticated with an API key. Your pipeline fires a one-line curl at it after a deploy: service, environment, commit SHA. It acks immediately so it never slows your pipeline down, and because it’s idempotent, a retried step doesn’t double-count. That’s the reported deploy — what CI believes it shipped.

GitHub webhooks. The GitHub App receives deployment_status, push, and PR events, all HMAC-verified so a forged event gets rejected. This captures deploy and process context automatically, without anyone wiring up the ingest call at all.

A liveness probe. This is the one that closes the loop. The platform runs its own read-only probe — starting with Render — that asks the running service what SHA is actually live. Not what CI said. Not what a webhook reported. What’s answering requests right now.

Then it reconciles all three into a single trust state per service × environment. Usually they agree, and the cell says the commit is live and verified. Sometimes they don’t — and that’s the state that matters. When the reported deploy disagrees with what the probe finds actually live, the cell goes into drift. Drift is the whole point. It’s the machine version of “wait, is the fix even deployed?” — asked continuously, answered before anyone has to ask it in a war room.

Notice that the trust state is per service × environment, not per service. That’s because where is half the question. “Live” isn’t a property of a commit — it’s a property of a commit in a place. A service doesn’t have one deployed state; it has one per environment, and they disagree all the time: staging three commits ahead of prod, a hotfix verified in staging that never made it through the prod pipeline, a deploy that landed in the wrong environment and looks perfectly healthy there. A single “deployed ✓” flag on a service papers over exactly the cases that hurt. The incident-room question was never just “is the fix deployed?” — it’s “is the fix deployed here?”

The board is a services × environments grid because the question has two axes. You look at it and see, at a glance, what’s live where — and your own latest commit is highlighted, so “is my change in staging yet?” is a glance, not a Slack thread. And we write the answer back to where developers already look: a commit status that reads “✅ live in staging.” You don’t have to come to Zero to get the truth. The truth comes to your PR.

Why this became the spine

We thought we were building a deploy dashboard. What we’d actually built was the one thing no visibility tool has: a record of which commit shipped where, and when.

That record turns out to be the difference between describing your infrastructure and explaining it. A resource inventory tells you what exists. Deploy truth tells you what changed. And change is what breaks things and moves bills. Every incident and every cost surprise has a “what changed right before this” — and for most teams that question is answered by grepping Slack and squinting at graphs. Zero knows the answer as data.

So the deploy timeline became the attribution layer for the rest of the platform.

Cost. When a service’s daily spend steps up — we gate it at ≥30% over baseline and ≥$25/mo so it’s a real signal, not noise — Zero doesn’t just flag the anomaly. It names the deploy that landed just before it. “Spend rose ~40% ≈ +$180/mo — deploy abc123 by @ben shipped Jun 14, just before; roll it back?” That’s not visibility. That’s root cause. A cost tool that doesn’t know what a deploy is can show you the step-up and stop there; it has no way to point at the commit that caused it.

Observability. Same mechanism, different signal. An error-rate spike against the service’s own 7-day baseline, a p95 latency regression, sustained CPU or memory saturation, a flood of ERROR logs — each finding names the deploy that shipped just before the change. The regression and its likely cause arrive together. And it’s freshness-guarded: stale or missing metric data renders no verdict, because a dead metrics pipeline should read as a broken connector, never as a phantom regression.

A monitoring tool tells you the graph went red at 14:12. Zero tells you the graph went red at 14:12 and abc123 shipped at 14:09. One of those is a starting point for an investigation. The other is most of the way to the fix.

What this costs us

Deploy truth is only as honest as the signals feeding it, and there are real gaps we haven’t closed.

Probe staleness. This is the sharpest one. Reconcile has no freshness window today. An old probe observation can “verify” a cell indefinitely — if the liveness check ran an hour ago and hasn’t run since, the board still reads that hour-old SHA as current truth. It isn’t lying, exactly, but it’s stating an old fact with present-tense confidence. The fix is a max-age on observations so a stale cell degrades to “unknown” instead of quietly holding. It’s spotted, not built.

Ingest depends on CI actually reporting. The cleanest signal — the POST /v1/deployments call — only exists if your pipeline makes it. A team that never wires it up loses the reported half of the reconciliation. Webhooks and the liveness probe hedge this: even with no ingest call, GitHub events and the probe can establish what’s live. But full confidence — reported and verified agreeing — needs the one line of curl.

Drift detection is only as good as your probes. The drift state is a comparison, and a comparison needs both sides. On a service with no liveness probe, Zero can tell you what was reported deployed but can’t confirm it’s actually live — so it can’t catch the silent-rollback case that motivated the whole thing. We’re expanding probe coverage beyond Render precisely because drift is worthless on a service we can’t independently observe.

We shipped it anyway, with the gaps visible, because a deploy board that’s honest about what it doesn’t know is worth far more than a green checkmark that’s confident about the wrong system.

The heuristic

A green pipeline tells you a deploy ran; it does not tell you a commit is serving traffic, or in which environment. Those are different systems, and only one of them is production — so verify what’s live, per environment, by asking the running service, not by trusting the thing that pushed to it.

tl;dr

The pattern. Teams answer “what’s live?” by reading CI history, which reports on the pipeline that pushed the deploy — not on the commit actually serving requests — so silent rollbacks, failed health checks, and wrong-environment deploys pass unnoticed behind a green checkmark. The fix. Reconcile three independent signals — an idempotent ingest call from CI, HMAC-verified GitHub webhooks, and a read-only liveness probe that asks the running service what SHA it’s on — into one trust state per service × environment (because “live” is always live somewhere), and flag drift when what was reported disagrees with what’s live. The outcome. You get a continuously-verified answer to “which commit is live, where,” written back to the PR where developers already look — and because the platform uniquely knows what shipped when, it can name the deploy behind a cost step-up or an error spike, not just show you the graph.

This is part 2 of a three-part series on building Zero. In part 3, we’ll follow the thread further: once you know what exists, what it costs, and what changed, the natural next question is what to do about it — how deploy truth and inventory grew into a findings-and-remediation loop that routes the fix to the person who owns it.