The Black October Patching: Who Owns It and the Best Practices That Matter

TL;DR

• This is the practical half of what the October 2026 disclosures actually were. That post sorts the incidents; this one is the work.
• The security patch drill below is the list to run now and again next time: inventory, then internet-facing items on CISA's Known Exploited Vulnerabilities (KEV) catalog, then the fixed versions, then appliances as investigations rather than patches. Nothing in it assumes you were hit.
• Awareness is everyone's; the work is not. Three duties need a name next to them: someone who reads the advisories weekly, someone who owns the inventory, and someone other than the author who approves dependency and release changes.
• A secret scanner changes detection, not exposure. A key that reached a remote is burned, and only revocation fixes that.
• For AI-assisted development, the rule is the same as for any third party: an agent can propose a dependency or a pipeline change, a human approves it, and the agent never holds publish or production credentials.

Last verified: November 12, 2026. The version numbers below were checked against vendor advisories on that date; the sourcing notes live in the companion post.

The short version of what happened

The full account, with sources and the parts that are still disputed, is in the companion post. The one-paragraph version, because the advice below keeps referring back to it:

Between roughly October 17 and 22, 2026, several unrelated things landed at once. A malicious litellm==1.107.0 was published to PyPI and Docker registries after an admin key committed in May and never revoked was used to subvert the release pipeline. Node.js shipped ten CVEs, one of them a Brotli decoder flaw that CISA later listed as known exploited. Two unauthenticated PHP RCEs ("PHP2Shell") were patched and exploited within days. GitLab patched a CVSS 10.0 flaw in the regex engine Ruby bundles. BeyondTrust, Ivanti and Fortinet appliances were confirmed exploited, in at least two cases weeks before disclosure. And eight eMudhra intermediate CA certificates turned out to have private keys recoverable from the public certificates.

Different causes, different responses. What follows is what to do about all of it.

The security patch drill: what to check, in order

This is the list to work down now, and again the next time a cluster of disclosures lands. The order matters more than the specific CVEs: the first items are cheap and tell you what the rest of the work is. Nothing here assumes you were hit.

The shape of it, for one advisory:

flowchart TD
    A["New advisory lands"] --> B{"Do we run it? Ask the inventory"}
    B -->|No| N["Record the check and stop"]
    B -->|Yes| C{"Listed in CISA KEV?"}
    C -->|Yes| D{"Internet-facing?"}
    D -->|Yes| E["Patch now, ahead of planned work"]
    D -->|No| F["Patch this week"]
    C -->|No| G{"Unauthenticated RCE or a malicious release?"}
    G -->|Yes| F
    G -->|No| H["Normal patch cadence"]
    E --> I{"An appliance, or a host that held credentials?"}
    F --> I
    I -->|Yes| J["Investigate and rotate - do not only patch"]
    I -->|No| K["Verify the version and record what was checked"]

The two decisions that do the most work are the first and the last. The first stops you spending a day on software you do not run; the last is the difference between patching a box and evicting whoever is already on it.

1. Inventory. Find where each of these runs, including inside container images and vendor appliances: PHP, Node.js, GitLab and its Ruby, LiteLLM, BeyondTrust, Ivanti, Fortinet. SBOMs make this a query instead of a hunt; see SBOM-Based Security for GitHub Repositories.

2. Patch internet-facing KEV items first. The KEV catalog lists CVEs with evidence of exploitation in the wild, so it is a filter on reality rather than on severity — a mid-severity flaw someone is actually using outranks a CVSS 10.0 nobody has touched. Anything on it that faces the internet comes before everything else: PHP, the Brotli flaw (Node.js and anything else embedding Brotli), BeyondTrust, Ivanti, Fortinet.

3. Patch to the fixed versions.

Technology	Fixed versions
PHP	8.5.12, 8.4.27, 8.3.36, 8.2.35 (backports for 7.4, 8.0, 8.1)
Node.js	22.23.4, 24.22.1, 26.12.1
GitLab	18.11.13, 19.0.10, 19.1.9, 19.2.9, 19.3.6, 19.4.3, 19.5.1
Ruby (self-compiled GitLab)	3.3.13, 3.4.12, 4.0.9 or later
LiteLLM	Pin 1.106.2 or an explicit earlier version
BeyondTrust Remote Support	Patch BT26-05-RS

4. LiteLLM: check dates and digests. Search lockfiles, build logs and image registries for 1.107.0. Compare image digests against the compromised one. If you find it, follow the maintainers' remediation steps and rotate credentials.

5. Appliances: investigate, do not just patch. Collect network and memory evidence, hunt for unrecognised privileged sessions and log gaps, and rebuild from a known-good state if compromise is suspected.

6. Rotate after any compromise. Everything the affected host or appliance could reach, including cloud roles, provider keys, database passwords and process-environment secrets.

7. Check trust stores for the eMudhra removal in your runtimes and images.

Printable checklist (keep it for next time)

• Inventory complete: PHP, Node.js, GitLab+Ruby, LiteLLM, BeyondTrust, Ivanti, Fortinet
• Internet-facing CISA KEV items identified and scheduled first
• PHP upgraded (8.5.12 / 8.4.27 / 8.3.36 / 8.2.35 or official backport)
• Node.js (Brotli, CISA KEV) upgraded (22.23.4 / 24.22.1 / 26.12.1), base images rebuilt
• GitLab upgraded; Ruby upgraded where self-compiled (3.3.13 / 3.4.12 / 4.0.9+)
• LiteLLM 1.107.0 searched for in lockfiles, venvs, image tags and digests
• If found: malicious files removed before pip, daemon killed, rebuilt, credentials rotated
• BeyondTrust patched; Ivanti and Fortinet reviewed against KEV entries
• Appliance forensics done (network and memory), not only patching
• Credentials rotated for any host or appliance suspected compromised
• Trust stores checked for eMudhra intermediate CA removal
• Record kept of what was checked, when, and by whom

Who should be watching this, and who owns it

"Everyone is responsible for security" is true as a statement about awareness and useless as an assignment. If no name is attached to who reads the advisory feed on Monday morning, nobody reads it. Shared awareness, named owner — that is the split that works.

Here is how the duties usually land in a team of any size. Titles differ; the duties do not.

Role	What they need to be aware of	What they actually own
Developers / feature teams	Advisories for the languages and libraries they write in; what an AI tool just added	Their lockfiles, their service's runtime version, reviewing new dependencies
Tech lead / repo maintainer	Who can publish, what the release workflow trusts, which branch rules are relaxed	Two-person approval on releases, pinning policy, branch protection
Platform / DevOps engineer	Base images, CI runners, registries, SBOM coverage	An inventory you can query, and rebuilding and rolling out patched base images
Security champion (in the team)	New CISA KEV entries, vendor advisories, which ones touch this team's stack	The weekly scan, and turning an advisory into a ticket with a priority
Security team / CISO office	Org-wide exposure, severity policy, disclosure handling, exceptions	The policy and the exception register — not the patching itself
SRE / on-call	Blast radius and rollback path of an emergency patch, error-budget cost	The change window and a safe rollout
IT / infrastructure (appliances)	BeyondTrust, Ivanti, Fortinet, VPNs, trust stores — the things no app team thinks about	Edge patching, and calling in forensics instead of only patching
Engineering manager	That patch work needs real capacity, not goodwill	Making the time exist and protecting it
Product owner	That a wave like this costs a sprint some capacity	The trade-off decision, recorded rather than implied
CTO / leadership	Accumulated risk, vendor concentration, when to leave a vendor	Risk acceptance, and the budget for the boring controls

The appliance gap is the one that bites. In most of the teams I see, every row above has an owner except the edge appliances. Those sit with IT, the application teams never think about them, and they were the entry point in two of the October incidents. Decide out loud who owns the VPN and the remote-support appliance before you need to know.

The smallest version that works

A three to five person team cannot staff that table, and does not need to. Three duties must have a name next to them:

1. A watcher. One person reads the KEV feed and the advisories for your stack on a fixed day each week. Fifteen minutes. Rotate it monthly if you like, but it is on one calendar.
2. An inventory owner. One person can answer "where does this version run" without a week of archaeology — which in practice means they own SBOM generation.
3. An approver. Dependency additions, CI changes and release-pipeline changes need a human who is not the author. In the LiteLLM chain, this is exactly the control that had been relaxed.

One person can hold two of these. Nobody should hold all three, because the point of the third is that it is a second pair of eyes.

What awareness looks like in practice

Awareness is not a mindset, it is a subscription list and a slot in the calendar:

• Subscribe to the CISA KEV catalog updates, and to vendor advisories for every appliance and hosted tool you run.
• Turn on repository advisory and dependency alerts, and send them to a channel a human reads — not an inbox rule.
• Give it a recurring slot. An advisory that arrives with no scheduled time to read it is a notification, not a control.
• Write down what was checked, when, and by whom. During the next wave, that record is the difference between an hour of work and a week of it.
• Run a dry run once. Pick a dependency you do use and ask who would notice, who would patch it, and how long the inventory step takes. Teams usually discover the inventory is the slow part, which is useful to learn on a quiet day.

Earlier 2026 npm campaigns (context, not October)

These were not part of Black October, but they show the same patterns and inform the pipeline advice below.

• Red Hat npm "Miasma" (Microsoft Security Blog, June 2, 2026): 32 packages across more than 90 versions in the @redhat-cloud-services scope were trojanised. Attackers published through the project's legitimate GitHub Actions OIDC publishing workflow, so the malicious versions carried authentic-looking provenance. The malware then spread by republishing other packages with forged SLSA provenance. A preinstall hook ran a dropper that harvested GitHub, npm, AWS, Azure, GCP, Vault and Kubernetes credentials. Lesson: provenance proves where a package was built, not that the build was trustworthy.
• Typosquatted npm packages (Microsoft, May 28, 2026): 14 malicious packages published within four hours by one actor, using install-time hooks to steal AWS, Vault, GitHub Actions and npm credentials. Installing with --ignore-scripts helps against install hooks, but Microsoft notes it would not stop a different attack (AsyncAPI) that triggered at import time.

Hardening your build and release pipeline

The LiteLLM chain and the npm campaigns share a theme: the attacker did not break the code, they used the process around it. The first group of items below comes straight from the LiteLLM chain. The rest is general good practice, not a claim about what these incidents proved.

From the LiteLLM chain:

• Never commit keys, and revoke old ones. A key committed in May was still valid in October. Deleting it from the repository is not revocation.
• Use short-lived, least-privilege tokens. An organisation-owner CLI token and CI secrets were the pivot points.
• Keep two-person approval on production releases, with branch protection that tokens cannot override. The relaxed rule was part of the chain.
• Protect release workflows from being pointed at forks. A workflow input that changes the build source is a high-value target.

Why a secret scanner is not enough

The usual answer to "never commit keys" is to add a secret-scanning hook and consider the problem solved. Worth doing, but be clear about what it buys you: a scanner changes detection. It does not change exposure.

Once a key reaches a remote it is burned, not merely at risk. Deleting the commit does not undo it, because by then it exists in the reflog, in forks and in every clone already taken, in CI logs and build caches, and in the host's events API. Public pushes are scraped continuously, so the right working assumption is minutes, not days.

LiteLLM is the whole argument in one line: the key was committed on May 27 and used by the attacker on October 10. A scanner that fired on May 27 would have changed nothing by itself. What was missing was revocation.

There are three gaps a scanner alone leaves:

• Bespoke formats. The LiteLLM key was an admin key for an internal alpha service. Scanners are good at AWS, GitHub and Stripe-shaped strings and poor at your own token format, at anything base64-encoded or split across lines, and at secrets inside notebooks, fixtures and test data.
• Client-side hooks are advisory. git commit --no-verify skips them, a fresh clone has none until someone installs them, and a colleague who never ran the setup step is simply unprotected.
• The diff is not the history. Scanning what changed finds nothing in the years of commits already there.

So the layers, in order of how much risk each one actually removes:

1. Have no long-lived key to leak. OIDC federation for CI, short-lived tokens, secret-manager references instead of values. This deletes the class of problem rather than policing it, and it is by far the biggest win.
2. A gate that cannot be skipped. Server-side push protection, plus a secrets-scan job in CI that the branch rules require, so a bypassed local hook still fails the pull request. The local hook stays — it is the cheapest, fastest feedback — but it is the convenience layer, not the control.
3. Scan the whole history once, across all refs, not just incoming diffs. Most repositories of any age have something in them nobody remembers putting there.
4. A written rotation path per credential: who owns it, how to revoke it, what breaks when you do. Then check whether it was used, in CloudTrail or its equivalent, because the question is never "was it pushed" but "was it used".

A minimal local hook, as the first layer and nothing more:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.2
    hooks: [{ id: gitleaks }]

The best practice that matters more than any of the tooling: treat every scanner hit as this credential is now public, rotate it today, never as remove the line and move on. The second reading is what turns a bad afternoon into an October.

General good practice:

• Pin dependencies and Docker images by digest, and use lockfiles with frozen installs. See Secure Package Version Pinning and JavaScript Dependency Security.
• Consider a dependency cooldown: do not adopt a brand-new release for a few days. LiteLLM's malicious PyPI window was under two hours.
• Use --ignore-scripts (or your package manager's equivalent) where it works, and know its limits: it does not stop import-time attacks.
• Verify provenance and also check who can publish. Provenance is evidence of where a package was built, not of whether the build was trustworthy.
• Separate publish credentials from developer machines, and prefer short-lived publishing identities.
• Monitor for unexpected new releases of your own packages and images.
• Lock down GitHub Actions: see Supply Chain Security in GitHub Actions.

Precautions for AI-assisted development

Start with what is verified. AI-assisted research reportedly helped find PHP2Shell, and reports say AI shortens the time from patch to exploit, so patch windows are shrinking. The npm worms target developer and CI environments. Install hooks and import-time code run whatever they contain.

None of that is specific to AI coding tools, but AI tools make some of it faster: they add dependencies quickly and can run commands. The advice below is general best practice, not something these incidents proved.

Dependencies

• Review every dependency an AI tool suggests or adds. Check that it exists, its age, maintainers and download history. Be wary of plausible-sounding names that do not exist or are typosquats. See How to Review an AI-Suggested npm Package and Supply-Chain Security in the AI Coding Era.
• Do not let an agent run installs or add dependencies without review.
• Require human approval for dependency, CI and release-pipeline changes.

Privilege and secrets

• Give agents least privilege: no publish tokens, no production credentials, no organisation-wide GitHub tokens.
• Keep secrets out of prompts, logs, config files and model context.
• Run agents in a sandbox or dev container, so a bad command has a small blast radius.

Configuration and tooling

• Treat agent config files, rules files and MCP or tool configuration as code that needs review.
• Pin tool and extension versions, and be wary of IDE or agent extensions from unofficial sources.
• Log and review what agents ran.

Generated code

• Review generated code for the same issues as human code: input validation, injection, authentication. See Why AI Won't Build Secure Software Unless You Instruct It.
• Run SAST, dependency and secret scans in CI, so the checks do not depend on anyone remembering them. See DevSecOps for AI-Built Applications.

A note on sourcing

Version numbers and KEV listings above were checked against vendor advisories and the CISA KEV catalog on November 12, 2026. The disputed and secondary-sourced claims — the age of the PHP bug, the exploitation evidence behind the Brotli KEV listing, the Onigmo mechanism, the appliance entry paths — are listed in full in the companion post's verification section. The ownership table and the pipeline advice are my own practice, not findings from these incidents, except where a bullet names the LiteLLM chain explicitly.

Last verified: November 12, 2026. Check the linked advisories before acting.

Closing

None of the work above is clever. An inventory you can query, a fixed slot to read advisories, a named approver, pinned dependencies, publish credentials off developer machines, and a rotation path per credential. The reason it is worth doing on a quiet week is that every one of these controls is cheaper to build when nothing is burning, and each one shortens the next wave from a week of archaeology to an afternoon of patching.

The incidents are the subject of the companion post. The best practices are the part you keep.

Related reading

• Black October 2026: What Engineering Teams Should Know Before the Next One
• Supply-Chain Security in the AI Coding Era
• Supply Chain Security in GitHub Actions
• SBOM-Based Security for GitHub Repositories
• JavaScript Dependency Security: npm, Bun, and pnpm
• Secure Package Version Pinning for npm, Bun, and pnpm
• DevSecOps for AI-Built Applications