Sha1‑Hulud Second Wave — Executive Summary and Immediate Risk

What happened: A new wave of Sha1‑Hulud supply‑chain attacks trojanized hundreds of npm packages (uploaded Nov 21–23, 2025) to run malicious code during the preinstall phase. The payload installs or locates the Bun runtime, runs a bundled script that executes TruffleHog‑style secret scanning, and then exfiltrates credentials and tokens. The campaign also registers infected hosts as self‑hosted GitHub runners named SHA1HULUD, creates ephemeral workflows to extract GitHub secrets, and deletes those workflows to hide activity. Some variants include destructive fallback logic that wipes a user’s home directory if exfiltration fails.

Scale and impact: Security vendors report tens of thousands of affected repositories and hundreds of compromised maintainer accounts. Build systems, CI runners, developer workstations, and any environment that runs npm install or builds from source are at risk. Stolen secrets can lead to cloud account takeover, package republishing, lateral supply‑chain spread, and immediate codebase compromise.

Immediate Actions (first 0–6 hours)

  • Stop the bleeding
    • Block npm install and other package installs from public registries in CI and production build lanes until packages are validated.
    • Quarantine CI runners and developer machines that ran builds between Nov 21–23; isolate suspected hosts from the network.
  • Identify exposure
    • Inventory repositories and CI pipelines that pulled packages published in the affected window; prioritize high‑privilege repos and orgs.
    • Search for preinstall, setup_bun.js, bun_environment.js, and unexpected package.json script changes in recent commits and package tarballs.
  • Credential containment
    • Rotate all tokens and credentials that could have been exposed: npm tokens, GitHub tokens, cloud provider keys (AWS/GCP/Azure), and CI secrets. Treat rotation as emergency for high‑privilege keys.
    • Revoke self‑hosted runner registrations and any unknown GitHub Actions runners; remove unexpected .github/workflows/* files.
  • Communicate
    • Notify engineering, DevOps, and security teams immediately and instruct developers to stop local installs and to preserve forensic artifacts (logs, package tarballs, node_modules, shell history).

Detection and Hunting Playbook

  • Repository and CI indicators
    • Look for new or modified .github/workflows/*.yaml files, especially discussion.yaml, shai-hulud-workflow.yml, or any workflow that triggers on discussion or workflow_dispatch.
    • Detect creation of self‑hosted runners named SHA1HULUD or other unexpected runner names.
  • Package and install indicators
    • Flag packages with preinstall scripts, unusual setup_bun.js entries, or bundled bun_environment.js.
    • Scan package tarballs for embedded TruffleHog usage, network exfil endpoints, or scripts that spawn curl, wget, or git to unknown hosts.
  • Endpoint and network telemetry
    • Hunt for outbound connections to unknown domains immediately after npm install or build steps.
    • Detect processes invoking trufflehog, bun, or unusual Node/Python scripts during build time.
    • Alert on sudden artifact uploads from CI to GitHub (artifacts containing secrets) and immediate deletion of workflows.
  • Behavioral signals
    • Look for mass repository creation attempts, unusual discussion openings in GitHub repos, or artifact downloads that correlate with workflow activity.

Remediation and Recovery Steps

  • Clean builds and images
    • Rebuild all artifacts from trusted, pinned sources and from clean build agents that have never run the compromised packages. Do not reuse infected runners or images.
    • Replace any ephemeral images or containers that may have executed malicious preinstall scripts.
  • Revoke and rotate
    • Rotate all secrets that could be exposed: GitHub Actions secrets, npm tokens, cloud provider keys, service principals, and any CI/CD deploy keys.
    • Revoke OAuth tokens and session tokens where possible; require re‑authentication for critical services.
  • Remove persistence
    • Audit and remove any malicious workflows, unexpected branches, or runner registrations. Remove unknown SSH keys and scheduled tasks on developer machines.
    • Reimage developer workstations and CI runners if root or persistent compromise is suspected.
  • Restore and validate
    • Restore repositories from known‑good commits; verify commit authorship and recent pushes for unauthorized changes.
    • Validate that CI pipelines only use pinned package versions and that package integrity checks (checksums, signed packages) are enforced.

Developer and CI/CD Hardening

  • Prevent execution in install hooks
    • Disallow or heavily restrict preinstall/postinstall scripts in CI by running installs with --ignore-scripts where feasible and using vetted build steps to run only required tooling.
    • Use package lockfiles and pin exact package versions; verify lockfile integrity before builds.
  • Enforce provenance and integrity
    • Use package signing, checksum verification, or a private registry/mirror that only contains vetted packages.
    • Cache and serve dependencies from an internal artifact repository (Artifactory, Nexus, GitHub Package Registry) after validation.
  • Harden GitHub Actions
    • Disable workflow creation from untrusted contributors; require branch protection and PR reviews for workflow changes.
    • Restrict workflow triggers that run on discussion or other low‑trust events; require workflow_run or manual approvals for sensitive workflows.
    • Limit self‑hosted runner registration to approved hosts and require attestation (MFA, device posture) before runner joins.
  • Secrets management
    • Move secrets out of repository settings where possible; use short‑lived credentials and vaults (HashiCorp Vault, cloud KMS) with strict access controls.
    • Enforce least privilege for CI service accounts and require just‑in‑time secrets injection.
  • Developer guidance
    • Train developers to inspect package.json changes, avoid running unknown install scripts, and to treat package updates with the same scrutiny as code changes.
    • Encourage reproducible builds and reproducible dependency trees.

Forensics, Evidence Preservation, and Legal

  • Preserve artifacts
    • Collect package tarballs, node_modules directories, CI logs, workflow YAMLs, and runner registration logs. Preserve timestamps and environment variables where possible.
    • Capture memory and disk images for any suspected compromised build agents.
  • Timeline reconstruction
    • Map package publish times, CI job runs, artifact uploads, and workflow creation/deletion events to build a timeline of compromise and exfiltration.
  • Engage stakeholders
    • Notify cloud providers and GitHub support if runner abuse or artifact exfiltration is confirmed. Consider coordinated takedown of malicious packages and domains.
  • Legal and compliance
    • If customer data or regulated secrets were exposed, follow breach notification rules and consult legal counsel for disclosure obligations.

Short checklist to operationalize now

  • Block public package installs in CI and production build lanes.
  • Scan repos and CI logs for preinstall scripts and setup_bun.js artifacts.
  • Revoke and rotate high‑privilege tokens immediately.
  • Isolate and reimage compromised runners and developer machines.
  • Enforce --ignore-scripts in CI where possible and use internal package mirrors.
  • Audit .github/workflows for unexpected files and remove unknown runners.

Final thought

This second Sha1‑Hulud wave is a textbook escalation: preinstall execution moves the attack earlier in the software supply chain and into build environments that often run with elevated privileges. The defensive posture must be twofold: stop execution of untrusted install hooks and assume secrets can be harvested—rotate them, shorten lifetimes, and move to vaulted, ephemeral credentials. Treat package installs in CI as a high‑risk operation and harden the pipeline now; the cost of a few hours of disruption is far lower than the cost of cloud takeover or a destructive wipe.

Be the first to comment

Leave a Reply

Your email address will not be published.


*


This site uses Akismet to reduce spam. Learn how your comment data is processed.