Surviving the Shai-Hulud era: runtime scanning and an SBOM that actually blocks something

A worm that harvests your tokens and republishes itself through your own packages does not care how careful you were last quarter. Here is what I changed in our pipelines afterwards, and which parts of it were worth the trouble.

The thing that made the npm supply chain attacks of the last year different was not the payload. It was the speed. A compromised maintainer account was used to publish a package that, on install, went looking for credentials on the build machine — and then used whatever it found to publish itself into other packages. No human in the loop, no waiting for the next release cycle. The blast radius grew while people slept.

The lockfile stopped being an answer

A lockfile protects you from a version changing underneath you. It does nothing about a version that is malicious the moment it is published. If the window between a bad publish and its takedown is measured in hours, and your CI installs dependencies fifty times a day, pinning is not a defence — it is a record of what you installed.

So I stopped treating the dependency list as the source of truth and started trusting two things instead: what is inside the artifact I built, and what is running in the cluster right now. Those are the only two questions a scanner can answer honestly.

An SBOM only counts if something rejects the build

For a long time our SBOMs were compliance artifacts — generated, uploaded, never read. The change was small: generate it from the built image rather than from source, store it against the image digest, and put a gate next to it that can fail.

# generate the SBOM from the image you just built, not from the repo
trivy image --format cyclonedx --output sbom.json app:$GIT_SHA

# fail the pipeline on anything serious that has a fix available
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 app:$GIT_SHA

--ignore-unfixed is the part people argue with, and it is the part that makes the gate survive. A build that fails on findings nobody can act on teaches the team to click through the failure. Within a month the gate means nothing.

A scanner that always fails is the same as no scanner.

The registry is not production

Scanning at build time gives you a photograph of the day you shipped. Images then run for months. A vulnerability published on a Thursday applies to something you built in March, and nothing in your pipeline will ever look at that image again. So the second job scans what is actually running:

# nightly: scan the images the cluster is really running
kubectl get pods -A -o jsonpath='{..image}' | tr ' ' '\n' | sort -u \
  | xargs -n1 trivy image --severity CRITICAL --quiet

The first time I ran that against a cluster I had been maintaining for two years, it found three images nobody had rebuilt since their first deploy and one that was not in any repository I could find. That is not a scanner result, it is an inventory problem — but the scanner is how you learn you have one.

What I actually run

1. A build-time gate with a fix filter. High and critical, only where a patched version exists. Trivy or Snyk — the choice matters far less than running the same one everywhere.
2. One SBOM per image, keyed by digest. When the next advisory lands, "do we ship this package" becomes a query someone answers in a minute instead of an afternoon of guessing.
3. A nightly scan of running workloads, with results going to the same channel as deploys. Security findings in a separate dashboard are findings nobody reads.
4. No long-lived registry tokens in CI. Short-lived credentials scoped to one job. This is the unglamorous one, and it is the one that stops a worm from spreading through you.

None of this is clever. It is plumbing, and it took about two weeks across our pipelines. It does not make anything safe — it makes you fast at finding out. Given how quickly these attacks move now, being fast at finding out is most of the job.