What happened: A new wave of Sha1‑Hulud supply‑chain attacks trojanized hundreds of npm packages (uploaded Nov 21–23, 2025) to run malicious code during the preinstall phase. The payload installs or locates the Bun runtime, runs a bundled script that executes TruffleHog‑style secret scanning, and then exfiltrates credentials and tokens. The campaign also registers infected hosts as self‑hosted GitHub runners named SHA1HULUD, creates ephemeral workflows to extract GitHub secrets, and deletes those workflows to hide activity. Some variants include destructive fallback logic that wipes a user’s home directory if exfiltration fails.
Scale and impact: Security vendors report tens of thousands of affected repositories and hundreds of compromised maintainer accounts. Build systems, CI runners, developer workstations, and any environment that runs npm install or builds from source are at risk. Stolen secrets can lead to cloud account takeover, package republishing, lateral supply‑chain spread, and immediate codebase compromise.
Immediate Actions (first 0–6 hours)
- Stop the bleeding
- Block
npm installand other package installs from public registries in CI and production build lanes until packages are validated. - Quarantine CI runners and developer machines that ran builds between Nov 21–23; isolate suspected hosts from the network.
- Block
- Identify exposure
- Inventory repositories and CI pipelines that pulled packages published in the affected window; prioritize high‑privilege repos and orgs.
- Search for
preinstall,setup_bun.js,bun_environment.js, and unexpectedpackage.jsonscript changes in recent commits and package tarballs.
- Credential containment
- Rotate all tokens and credentials that could have been exposed: npm tokens, GitHub tokens, cloud provider keys (AWS/GCP/Azure), and CI secrets. Treat rotation as emergency for high‑privilege keys.
- Revoke self‑hosted runner registrations and any unknown GitHub Actions runners; remove unexpected
.github/workflows/*files.
- Communicate
- Notify engineering, DevOps, and security teams immediately and instruct developers to stop local installs and to preserve forensic artifacts (logs, package tarballs,
node_modules, shell history).
- Notify engineering, DevOps, and security teams immediately and instruct developers to stop local installs and to preserve forensic artifacts (logs, package tarballs,
Detection and Hunting Playbook
- Repository and CI indicators
- Look for new or modified
.github/workflows/*.yamlfiles, especiallydiscussion.yaml,shai-hulud-workflow.yml, or any workflow that triggers ondiscussionorworkflow_dispatch. - Detect creation of self‑hosted runners named SHA1HULUD or other unexpected runner names.
- Look for new or modified
- Package and install indicators
- Flag packages with
preinstallscripts, unusualsetup_bun.jsentries, or bundledbun_environment.js. - Scan package tarballs for embedded TruffleHog usage, network exfil endpoints, or scripts that spawn
curl,wget, orgitto unknown hosts.
- Flag packages with
- Endpoint and network telemetry
- Hunt for outbound connections to unknown domains immediately after
npm installor build steps. - Detect processes invoking
trufflehog,bun, or unusual Node/Python scripts during build time. - Alert on sudden artifact uploads from CI to GitHub (artifacts containing secrets) and immediate deletion of workflows.
- Hunt for outbound connections to unknown domains immediately after
- Behavioral signals
- Look for mass repository creation attempts, unusual discussion openings in GitHub repos, or artifact downloads that correlate with workflow activity.
Remediation and Recovery Steps
- Clean builds and images
- Rebuild all artifacts from trusted, pinned sources and from clean build agents that have never run the compromised packages. Do not reuse infected runners or images.
- Replace any ephemeral images or containers that may have executed malicious preinstall scripts.
- Revoke and rotate
- Rotate all secrets that could be exposed: GitHub Actions secrets, npm tokens, cloud provider keys, service principals, and any CI/CD deploy keys.
- Revoke OAuth tokens and session tokens where possible; require re‑authentication for critical services.
- Remove persistence
- Audit and remove any malicious workflows, unexpected branches, or runner registrations. Remove unknown SSH keys and scheduled tasks on developer machines.
- Reimage developer workstations and CI runners if root or persistent compromise is suspected.
- Restore and validate
- Restore repositories from known‑good commits; verify commit authorship and recent pushes for unauthorized changes.
- Validate that CI pipelines only use pinned package versions and that package integrity checks (checksums, signed packages) are enforced.
Developer and CI/CD Hardening
- Prevent execution in install hooks
- Disallow or heavily restrict
preinstall/postinstallscripts in CI by running installs with--ignore-scriptswhere feasible and using vetted build steps to run only required tooling. - Use package lockfiles and pin exact package versions; verify lockfile integrity before builds.
- Disallow or heavily restrict
- Enforce provenance and integrity
- Use package signing, checksum verification, or a private registry/mirror that only contains vetted packages.
- Cache and serve dependencies from an internal artifact repository (Artifactory, Nexus, GitHub Package Registry) after validation.
- Harden GitHub Actions
- Disable
workflowcreation from untrusted contributors; require branch protection and PR reviews for workflow changes. - Restrict
workflowtriggers that run ondiscussionor other low‑trust events; requireworkflow_runor manual approvals for sensitive workflows. - Limit self‑hosted runner registration to approved hosts and require attestation (MFA, device posture) before runner joins.
- Disable
- Secrets management
- Move secrets out of repository settings where possible; use short‑lived credentials and vaults (HashiCorp Vault, cloud KMS) with strict access controls.
- Enforce least privilege for CI service accounts and require just‑in‑time secrets injection.
- Developer guidance
- Train developers to inspect
package.jsonchanges, avoid running unknown install scripts, and to treat package updates with the same scrutiny as code changes. - Encourage reproducible builds and reproducible dependency trees.
- Train developers to inspect
Forensics, Evidence Preservation, and Legal
- Preserve artifacts
- Collect package tarballs,
node_modulesdirectories, CI logs, workflow YAMLs, and runner registration logs. Preserve timestamps and environment variables where possible. - Capture memory and disk images for any suspected compromised build agents.
- Collect package tarballs,
- Timeline reconstruction
- Map package publish times, CI job runs, artifact uploads, and workflow creation/deletion events to build a timeline of compromise and exfiltration.
- Engage stakeholders
- Notify cloud providers and GitHub support if runner abuse or artifact exfiltration is confirmed. Consider coordinated takedown of malicious packages and domains.
- Legal and compliance
- If customer data or regulated secrets were exposed, follow breach notification rules and consult legal counsel for disclosure obligations.
Short checklist to operationalize now
- Block public package installs in CI and production build lanes.
- Scan repos and CI logs for
preinstallscripts andsetup_bun.jsartifacts. - Revoke and rotate high‑privilege tokens immediately.
- Isolate and reimage compromised runners and developer machines.
- Enforce
--ignore-scriptsin CI where possible and use internal package mirrors. - Audit
.github/workflowsfor unexpected files and remove unknown runners.
Final thought
This second Sha1‑Hulud wave is a textbook escalation: preinstall execution moves the attack earlier in the software supply chain and into build environments that often run with elevated privileges. The defensive posture must be twofold: stop execution of untrusted install hooks and assume secrets can be harvested—rotate them, shorten lifetimes, and move to vaulted, ephemeral credentials. Treat package installs in CI as a high‑risk operation and harden the pipeline now; the cost of a few hours of disruption is far lower than the cost of cloud takeover or a destructive wipe.
Leave a Reply