Open cardosec

Case D2DA2E · AI security · L4 Advanced

Model Load Spawns a Shell

Practise as: Incident drill

Live alertEDR on GPU node ml-train-07 alerts at 02:31: python3 spawned /bin/sh, which ran curl to an unknown IP, right after a job called torch.load() on a model fetched from a public hub.

  1. 01 What happened?
  2. 02 What is the impact?
  3. 03 What do you do?
Practise this card
What a strong answer covers

Try it out loud first. Then check yourself:

  1. Isolate the node and suspend the job; the pickle payload ran with the job's identity and any mounted credentials.
  2. Rotate secrets the process could reach: hub tokens, cloud IAM role creds, S3 keys, W&B/MLflow tokens.
  3. Analyze the file safely with pickletools or a scanner to extract the payload, C2 IP and any persistence.
  4. Hunt: other nodes or users who pulled the same repo/revision; block the IP and repo in the internal mirror.
  5. Prevent: safetensors only (weights_only=True needs PyTorch 2.6+), pinned commit hashes, egress limits on training nodes.

If the interviewer pushes back

  • The training cluster shares an NFS volume with datasets and checkpoints. How does that change containment?

cardosec draws a security topic and gives you a clock: explain it out loud with no notes, then see what you covered and what you missed. Free during early access.