Compute Engine, Persistent Disks, and Forensic Acquisition

3 minutes ago
5 min read

How to pull a defensible disk image off a running or compromised GCE instance without touching the original evidence.

Compute Engine is where most GCP incidents eventually lead you — a web app got popped, a cron job got tampered with, a build agent got used to mint tokens it shouldn't have. The good news is that acquiring forensic evidence off a GCE instance is one of the cleaner workflows in cloud IR, because Persistent Disks are separate resources from the VM they're attached to, and snapshotting one doesn't touch the running instance at all.

Disks Are Not the VM

A Compute Engine instance is really a compute allocation plus one or more Persistent Disks attached to it — a boot disk holding the OS, and optionally additional data disks.

The disk exists as its own GCP resource with its own resource name, independent of the instance's lifecycle: you can snapshot it, clone it, or detach it and attach it somewhere else, all without stopping the instance.

That independence is what makes GCE acquisition workable even when you can't afford downtime on a production system — you're never touching the live disk directly, only a point-in-time copy of it.

Disk type matters for one thing forensically: consistency guarantees. A snapshot taken while the instance is running is crash-consistent by default — equivalent to what you'd get from a hard power-off, which is fine for most filesystems but can leave an application mid-write in an inconsistent state. If you need application-consistent state (a database in the middle of a transaction, for instance), you'd normally quiesce writes first, but for IR purposes crash-consistent is usually exactly what you want anyway — it reflects reality at the moment of acquisition rather than a cleaned-up version of it.

The Snapshot-Based Acquisition Workflow

This is the sequence Solstice's IR retainer uses when a compute instance needs to be imaged:

Snapshot the disk in place, without stopping the instance, so the attacker (if still active) sees no change in behavior:

gcloud compute disks snapshot solstice-web01-boot \
  --snapshot-names=ir-2026-08-13-solstice-web01-boot \
  --zone=us-central1-a --storage-location=us-central1

Create a new disk from that snapshot in an isolated forensics project — never the source project, to keep the evidence off the same IAM boundary the attacker may still have access to:

gcloud compute disks create ir-solstice-web01-boot-copy \
  --source-snapshot=ir-2026-08-13-solstice-web01-boot \
  --zone=us-central1-a --project=solstice-forensics

Attach that disk, read-only, to a clean forensics VM you control in the isolated project, and image it from there with dd or dcfldd rather than mounting it directly:

sudo dcfldd if=/dev/sdb of=/mnt/evidence/solstice-web01-boot.dd hash=sha256 hashlog=solstice-web01-boot.sha256

Hash the resulting image and the snapshot's own checksum (Google stores one), and record both in your chain of custody documentation before the raw image ever gets analyzed.

Two Things That Trip People Up

Snapshot IAM permissions are separate from disk permissions — compute.snapshots.create on the disk's project plus compute.disks.createSnapshot are what you actually need, and it's common for an incident responder to have read access to a project but not snapshot-creation rights, which needs to be granted explicitly and shows up itself as an Admin Activity log entry worth noting for your own timeline.
storage-location matters — by default a snapshot inherits the disk's region, but you can pin it elsewhere. For a client with data residency requirements, confirm where the snapshot actually lands before you create it, not after.

Instance Metadata: Configuration Data and a Favorite Persistence Mechanism

Every GCE instance carries a metadata key-value store, reachable from inside the instance at metadata.google.internal and readable via the API from outside it. Two keys matter more than the rest for IR: startup-script and shutdown-script. Whatever's in those keys runs as root (or SYSTEM on Windows) every time the instance boots or shuts down, and because metadata is editable by anyone with compute.instances.setMetadata on the instance, it's a favorite, boring, extremely effective persistence mechanism — no exploit required, just an IAM permission an attacker already has from an earlier step. Pulling the current and, if available, prior metadata values for these keys should be an early step on any compromised instance.

gcloud compute instances describe solstice-web01 --zone=us-central1-a \
  --format='value(metadata.items.filter("key:startup-script OR key:shutdown-script"))'

The corresponding Admin Activity log entry to hunt for is

v1.compute.instances.setMetadata — filter

for it across a project and you'll catch both the legitimate configuration changes and the illegitimate ones in the same short list.

Who Actually Logged In: OS Login vs. Metadata SSH Keys

Linux instances support two different ways to manage SSH access, and they have very different forensic value.

Metadata-based SSH keys are just public keys pasted into instance or project metadata — anyone holding the matching private key can log in as the associated local username, and there's no per-user audit trail beyond the OS-level auth log on the instance itself.

OS Login, by contrast, ties SSH access to the caller's actual Google identity via IAM (roles/compute.osLogin or osAdminLogin), and every login is attributable to a specific principal in Cloud Audit Logs, not just a key that could have been shared or stolen.

If OS Login isn't enabled org-wide, you should expect to lean much more heavily on the guest OS's own auth logs (auth.log, /var/log/secure, or Windows Security event logs) than on Cloud Audit Logs for the actual login attribution.

Serial Port Output: The Log You Get Even When the OS Won't Cooperate

Every instance's serial console output is captured by Google independent of what's happening inside the guest OS, which makes it valuable when an attacker has tampered with or disabled logging inside the instance itself — serial output survives that. It's retained for the life of the instance and pullable even after the instance is stopped, though it clears on deletion, so grab it before you delete anything.

gcloud compute instances get-serial-port-output solstice-web01 --zone=us-central1-a > serial-output.log

Hyperdisk is Google's newer Persistent Disk generation, offering independently tunable IOPS/throughput and faster snapshot/restore performance than the older pd-* disk types — if you're imaging a Hyperdisk-backed instance the workflow is identical, just faster. Shielded VM (secure boot, vTPM, integrity monitoring) and Confidential VM (memory encryption via AMD SEV or Intel TDX) are also both far more commonly the default posture now , and it's worth checking whether an instance is Confidential VM-enabled before you plan a memory-acquisition approach, since some capture techniques that assume readable guest memory don't behave the same way against an encrypted memory space.

🔎 IR tip: Pull the serial port output and the current metadata (especially startup-script) before you do anything else with a suspect instance — both are cheap, non-invasive, and won't be there anymore if someone deletes the instance while you're still setting up the snapshot workflow.

.......................................................................Dean---------------------------------------------------