Where the traffic actually went — and why GCP's network model means one VPC can span the whole company. GCP's networking model has one structural quirk that changes how you scope an investigation compared to AWS: a VPC in Google Cloud is global, not regional, and it's also a project-level resource that can be shared across other projects entirely. Get this wrong and you'll scope an incident to a single project when the network the attacker actually moved through spans half the org. Shared VPC: One Network, Many Projects Solstice Payments runs a Shared VPC out of solstice-shared-net, the host project, with subnets in us-central1 (10.20.1.0/24) and europe-west1 (10.20.2.0/24). Both solstice-prod-core and solstice-prod-data are service projects attached to that host — their instances get IPs out of those same subnets, but IAM permissions over the network itself (who can create firewall rules, who can peer another VPC into it) live in the host project, not the service projects. This is the single most common scoping mistake I see: An investigator pulls IAM and Flow Logs for the compromised service project, sees nothing unusual about the network configuration, and never checks the host project where the actual firewall rule changes happened. First move on any network-touching incident: identify whether the VPC in play is standalone or shared, and if shared, go find the host project. gcloud compute shared-vpc get-host-project solstice-prod-core VPC Flow Logs Flow Logs record a sampled summary of IP traffic to and from VM interfaces — not full packet capture, but enough to answer "did this instance talk to that IP, and how much data moved." They're enabled per-subnet, off by default, and like Data Access audit logs, are one of the things worth confirming is actually turned on before you assume you'll have network evidence. The fields that matter most in a flow log record: connection.src_ip / connection.dest_ip and the matching ports — the basic who-talked-to-whom. bytes_sent and packet counts — useful for spotting a large outbound transfer that doesn't match the instance's normal traffic profile, which is often the first concrete signal of data exfiltration over the network layer rather than an API call. reporter (SRC or DEST) — tells you which side of the connection generated this particular record, since both ends can report the same flow. rtt_msec — round-trip time, occasionally useful for distinguishing traffic that stayed inside Google's backbone (very low RTT) from traffic that left to the public internet and came back. Sampling matters: Flow Logs default to 50% sampling and 5-second aggregation intervals, both configurable. At default settings you will not see every single connection, which is worth stating explicitly in a report if a client asks why a specific short-lived connection isn't in the logs — it may genuinely not have been sampled, not that logging failed. gcloud compute networks subnets update solstice-central-subnet --region=us-central1 \ --enable-flow-logs --logging-flow-sampling=1.0 --logging-aggregation-interval=interval-5-sec Firewall Rule Logging Separate from Flow Logs, individual firewall rules can have their own logging enabled, recording every connection the rule allowed or denied — including the specific rule that matched. This is what you want when you need to know not just that traffic moved, but which firewall rule let it through, especially useful when an attacker (or a misconfiguration) added an overly permissive rule and you need to prove exactly when it started being exploited. Packet Mirroring: Full Packet Capture, Not Just Flow Summaries Flow Logs and firewall logging tell you that traffic moved and which rule let it through — they don't hand you the actual packets. For that, Google Cloud has Packet Mirroring: Full packet capture defined per VM instance, per GKE cluster, or per VPC, which makes it straightforward to scope a capture tightly around whatever's suspected of being compromised rather than pulling traffic for an entire environment. GCP's implementation works differently from a typical on-prem tap. A Mirror Policy defines which source instances to capture from, but the mirrored traffic doesn't land in a bucket directly — Google Cloud has no mechanism to write a PCAP straight to Cloud Storage. Instead, the mirrored packets are handed to a load balancer forwarding rule, which distributes them across a backend instance group exactly the way a normal load balancer distributes inbound requests, except here the "requests" are copies of live traffic. For DFIR purposes, that backend instance group is where you'd stand up Arkime, Zeek, or another NSM tool to actually capture and analyze what's mirrored. Setting it up takes two pieces. First, an internal load balancer forwarding rule flagged as a mirroring collector — this is what the mirrored packets actually get delivered to. Solstice already has an NSM backend service (running Zeek) fronted by this rule: gcloud compute forwarding-rules create solstice-mirror-collector \ --region=us-central1 --load-balancing-scheme=INTERNAL \ --backend-service=solstice-nsm-backend --is-mirroring-collector Then the mirroring policy itself, pointing at that collector and naming what to mirror — in an active incident you'd usually scope this to the one suspect instance, not the whole subnet: gcloud compute packet-mirrorings create solstice-prod-core-mirror \ --region=us-central1 --network=solstice-shared-net \ --collector-ilb=solstice-mirror-collector \ --mirrored-instances=solstice-prod-core --filter-direction=BOTH The catch for incident response: Packet Mirroring is forward-looking only. It captures traffic sent after the policy goes live — there's no way to retroactively pull packet-level data for a window before the mirror was configured. If Solstice's SRE team suspects solstice-prod-core is actively being used for data exfiltration and Flow Logs alone aren't giving enough detail, standing up a Mirror Policy against that instance immediately is the move — but it only helps for what happens next, not what already happened. Cloud DNS Logging DNS query logging for a VPC's managed zones is also opt-in and separately configured from Flow Logs. It records the query name, response, and source, and is often the fastest way to confirm C2 activity when an attacker is using domain-based command and control rather than a raw IP — Flow Logs will show you the resolved IP, but Cloud DNS logs are what tie that IP back to the domain name that was actually queried, which matters when the IP itself sits behind shared infrastructure like a CDN or the attacker rotates IPs faster than domains. Load Balancer Logs For anything sitting behind a Google Cloud Load Balancer — which is most public-facing Solstice services — the load balancer's own request logs are frequently more useful than the backend instance's application logs, because they're generated regardless of what happened to the backend afterward. Each entry includes the client IP, request path, response code, latency, and, for HTTPS(S) load balancers, the negotiated TLS details. If a web app was compromised through a specific malicious request, the load balancer log is often the cleanest place to find the initial request that did it, especially if the attacker's later steps involved tampering with or disabling the application's own logging. Firewall rules moved toward a hierarchical model — Firewall Policies at the org and folder level, evaluated alongside (and able to override) legacy per-VPC firewall rules, giving central security teams a way to enforce network controls that individual project owners can't loosen. If you're investigating a network permissiveness question, check for hierarchical Firewall Policies in addition to the classic per-VPC rules — the effective policy is a merge of both, and looking at only one gives an incomplete picture. Cloud NGFW (next-generation firewall, with intrusion prevention) also expanded significantly since 2022 and may be in play for environments with more mature network security postures. 🔎 IR tip: Before concluding "no evidence of network exfiltration," confirm three things were actually on: Flow Logs at an adequate sampling rate, firewall rule logging on the relevant rules, and Cloud DNS logging. Absence of evidence across all three is a real finding worth reporting on its own — but a gap in even one of them can produce a false negative that looks identical to a clean network. ..-------------------------------------Dean---------------------------------------------------

Dont forget to check out completle SentinelOne series https://www.cyberengage.org/courses-1/mastering-sentinelone%3A-a-comprehensive-guide-to-deep-visibility%2C-threat-hunting%2C-and-advanced-querying%22 Introduction Ran into something during a routine SentinelOne rollout that's worth sharing, because it doesn't look like a SentinelOne problem when you first see it — it looks like the third-party app is broken. An employee's VPN client wouldn't connect on their managed Mac. Install was clean, app opened fine, connection just sat on "establishing" forever. If you've deployed SentinelOne's default macOS configuration profile fleet-wide, you may be sitting on the same landmine without knowing it yet. The Issue The VPN app needs to register a macOS System Extension to function. That registration was failing silently — no approval prompt, no actionable error, just a permanent hang at connect time. Standard move for SentinelOne Users : disable the SentinelOne agent for a few minutes, have the user retest, see if the agent is the cause. No change. Agent off, same failure. That result normally clears SentinelOne — except this time it didn't, and the reason why is the actual finding here. What I Found The user checked the macOS System Extension policy database directly: sudo plutil -p /Library/SystemExtensions/db.plist It returned an active extensionPolicies entry scoped to one Team ID only: "allowedTeamIDs" => [ "******" ] // SentinelOne's Team ID(Redacted) "allowUserOverrides" => false That policy did not change when the agent was disabled — which makes sense once you know how it's enforced: MDM pushes SentinelOne's default macOS configuration profile, which includes a System Extension payload authorizing SentinelOne's own extension. macOS writes that payload into /Library/SystemExtensions/db.plist as an extensionPolicies entry. Once that entry declares an allowedTeamIDs array, macOS treats it as exclusive — anything not listed is denied at the OS level, silently, with no user prompt. The SentinelOne agent process has nothing to do with enforcement at that point. Killing or disabling the agent doesn't touch the profile sitting on disk, so the block persists agent-on or agent-off. So the "SentinelOne is broken" theory and the "SentinelOne isn't involved" theory were both half right: the running agent wasn't the problem, but a profile it shipped with was The Fix I left SentinelOne's default profile untouched and pushed a second, separate profile adding the VPN vendor's Team ID to the allowlist, with allowUserOverrides flipped back to true. One reboot later, the VPN connected on the first retry. If You're Facing This Third-party app with a system extension / kernel component silently fails to load on a SentinelOne-managed Mac — no prompt, no clear error Disabling the SentinelOne agent doesn't fix it Run sudo plutil -p /Library/SystemExtensions/db.plist and look for allowedTeamIDs If it lists only SentinelOne's Team ID (******) with allowUserOverrides: false, that's your cause Fix: a separate MDM profile allowlisting the blocked vendor's Team ID — don't edit SentinelOne's default profile directly Final Thoughts SentinelOne wasn't misbehaving — its default macOS profile did exactly what it was configured to do. The gap was on our side: we deployed that profile fleet-wide without reading the System Extension payload closely enough to notice it also locks out every other vendor. Worth a five-minute audit of your own profile before this shows up as a VPN, webcam, or remote-access ticket with no obvious cause.

GCS buckets are both a common source of evidence and, misconfigured, one of the most common ways data actually leaves a GCP environment. Cloud Storage shows up in almost every incident twice: once as a target — the thing the attacker was actually after — and once as a source of evidence, if you configured it right beforehand. This article covers both sides: how bucket access actually works, how to tell if an attacker exfiltrated data through a bucket, and how to collect GCS objects in a way that holds up as evidence. IAM vs. Legacy ACLs Buckets can be governed two ways, and it's important to know which one a given bucket uses before you try to reason about who could access it. Uniform bucket-level access applies IAM policy consistently across every object in the bucket — one policy, no per-object exceptions, and it's what Google now recommends and defaults new buckets to. The older model, fine-grained access, layers legacy ACLs on top of IAM at the individual object level, meaning a bucket's overall IAM policy can look reasonable while a specific object inside it has an ACL granting allUsers read access that nobody would find just by checking the bucket policy. Solstice's security team standardized on uniform bucket-level access org-wide for exactly this reason — with fine-grained buckets, checking "is anything in this bucket public" means enumerating every object's ACL individually, not just reading one policy document. If you inherit an investigation involving a fine-grained bucket, budget the extra time for that enumeration. gcloud storage buckets describe gs://solstice-prod-invoices-archive --format='value(iamConfiguration)' Access Logging: Two Different Features, Only One Still Worth Using GCS has two separate logging mechanisms, and it's easy to assume the newer one replaces the older one — it doesn't, not fully. Data Access audit logs give you per-object-read/write attribution tied to a principal, in the same audit log pipeline as everything else, and that's the right default for tracking authenticated access. But Data Access audit logs only capture activity from an identified principal — they have a blind spot for exactly the traffic you most need during a public-exposure investigation: anonymous requests from allUsers or allAuthenticatedUsers. For that, you still need the older, bucket-native Usage Logs and Storage Logs — CSV files written into a destination bucket, and the only mechanism that tracks unauthenticated object access at all. Two catches worth knowing before you rely on them: they can only be enabled via gsutil or the API, not the console they don't route through Cloud Logging or the Log Routing pipeline at all — you have to retrieve them manually from wherever they're written, or process them with a Cloud Function. Usage Logs are produced hourly and record every access from allUsers/allAuthenticatedUsers (Google explicitly doesn't guarantee no duplicates between hourly files — dedupe on the s_request_id field). Storage Logs are written once a day and just report total storage consumed over the prior 24 hours, useful for spotting unexplained growth or shrinkage rather than individual access events. gsutil mb gs://solstice-access-logs gsutil iam ch group:cloud-storage-analytics@google.com:legacyBucketWriter gs://solstice-access-logs gsutil logging set on -b gs://solstice-access-logs gs://solstice-prod-invoices-archive If Usage Logs weren't enabled on a bucket before an incident, there's no way to retroactively see who accessed it anonymously — which makes enabling them upfront, on any bucket handling sensitive data, one of the cheapest insurance policies in this article. Versioning, Soft Delete, and Making Deletion Not Mean Deletion Two separate features protect against data loss — including an attacker deliberately deleting objects to cover their tracks — and it's worth knowing both because they solve slightly different problems. Object versioning keeps prior versions of an object when it's overwritten or deleted, and those noncurrent versions are retrievable by generation number until a lifecycle rule cleans them up. Soft delete, a newer and separate feature, retains any deleted or overwritten object for a configurable retention window (seven days by default) regardless of whether versioning is even enabled — so even a bucket that was never configured for versioning still gives you a recovery window if soft delete is on, which it now is by default for new buckets. For an investigation, this means an attacker's gsutil rm -r doesn't necessarily mean the objects are gone. Check both features before writing off deleted evidence as unrecoverable. gcloud stora ge ls --with-versions gs://solstice-prod-invoices-archive/ gcloud storage objects restore gs://solstice-prod-invoices-archive/q3-report.csv --generation=1723484921003912 Bucket Lock: Making Your Own Evidence Tamper-Proof If you're the one setting up evidence retention rather than investigating after the fact, a retention policy combined with Bucket Lock gives you a genuine WORM (write-once-read-many) guarantee — once locked, not even a project Owner can shorten the retention period or delete objects before it expires, short of deleting the entire project. Solstice's sec-ops team locks their audit-log export bucket this way specifically so that a compromised Owner-level identity — the worst case scenario — still can't destroy the evidence trail that would expose the compromise. The Public-Exposure Problem A bucket or an individual object becomes public the moment allUsers or allAuthenticatedUsers appears in an IAM binding or a legacy ACL entry. This remains one of the most common ways sensitive data actually leaves a GCP environment — not through a sophisticated exfiltration technique, but through a bucket that was made public temporarily for a legitimate reason and never locked back down. Public Access Prevention, an org-policy or bucket-level setting, blocks these bindings outright regardless of what IAM policy says, and checking whether it's enforced organization-wide is one of the fastest ways to rule out this entire class of exposure in an investigation. This same public-exposure pattern has an off-the-shelf attacker tool built around it — GCPBucketBrute automates exactly this recon: dictionary-guessing bucket names (which aren't secret, just unlisted), checking which ones respond as valid, then testing whether objects inside can be listed publicly. If a bucket comes back positive on all three, downloading its contents is trivial. Worth checking your own naming conventions against this reality — a predictable scheme (solstice-prod-*, solstice-backup-*) is exactly what a dictionary attack is built to find. gcloud storage buckets list --format='table(name, iamConfiguration.publicAccessPrevention)' \ --project=solstice-prod-data Collecting GCS Objects as Evidence Every object carries Google-computed CRC32C and MD5 hashes as metadata, generated at write time — pull and record these alongside your own hash of the downloaded copy rather than only hashing after the fact, since it gives you an independent value to cross-check against. Always copy by generation number when the bucket has versioning or soft delete enabled, so you're provably pulling the exact version relevant to your timeline rather than whatever happens to be current at download time. gcloud storage objects describe gs://solstice-prod-invoices-archive/q3-report.csv --format='value(md5Hash,crc32c,generation)' gcloud storage cp gs://solstice-prod-invoices-archive/q3-report.csv#1723484921003912 ./evidence/ No Logging At All? Check the Bill Sometimes you inherit an investigation where neither Data Access audit logs nor Usage/Storage Logs were ever enabled on the bucket in question — more common than it should be. You're not entirely out of options: Google Cloud tracks data egress for billing purposes regardless of whether bucket-level logging was configured, since data leaving a Cloud zone/region gets billed back to the customer either way. The Billing section's Cost Table sub-menu breaks down cost per service per month, and that includes network egress from Cloud Storage. It won't give you a filename or a source IP — this is a blunt instrument — but it can tell you approximately when a spike in egress happened and roughly how much data moved, which is often the difference between having a rough exfil estimate for a client's insurer and having nothing at all to hand them. One real limitation: Cost Table data only finalizes at the end of a billing cycle (typically monthly) — if the incident is mid-cycle, this data isn't available yet and you'll be waiting on the cycle to close before you can pull it. IR tip: check Cost Table egress by month before telling a client billing data can't help — it's a last resort, but it beats telling them there's simply no record at all. 📌 Soft delete moving to generally available and on-by-default for new buckets is the biggest practical change — it means "the attacker deleted the evidence" is a weaker claim than it used to be, and worth actively checking before accepting it. Public Access Prevention has also moved from an opt-in setting to something Google actively pushes organizations toward enforcing by default via org policy, which has measurably reduced (though far from eliminated) the accidental-public-bucket class of incident industry-wide. IR tip: Before you tell a client their exfiltrated data is unrecoverable because the attacker deleted the bucket contents, check soft delete and object versioning status first — in a meaningful fraction of cases, the objects are still sitting there in a recoverable state and nobody's checked. ----------------------------------------------Dean----------------------------------------------------

How to pull a defensible disk image off a running or compromised GCE instance without touching the original evidence. Compute Engine is where most GCP incidents eventually lead you — a web app got popped, a cron job got tampered with, a build agent got used to mint tokens it shouldn't have. The good news is that acquiring forensic evidence off a GCE instance is one of the cleaner workflows in cloud IR, because Persistent Disks are separate resources from the VM they're attached to, and snapshotting one doesn't touch the running instance at all. Disks Are Not the VM A Compute Engine instance is really a compute allocation plus one or more Persistent Disks attached to it — a boot disk holding the OS, and optionally additional data disks. The disk exists as its own GCP resource with its own resource name, independent of the instance's lifecycle: you can snapshot it, clone it, or detach it and attach it somewhere else, all without stopping the instance. That independence is what makes GCE acquisition workable even when you can't afford downtime on a production system — you're never touching the live disk directly, only a point-in-time copy of it. Disk type matters for one thing forensically: consistency guarantees. A snapshot taken while the instance is running is crash-consistent by default — equivalent to what you'd get from a hard power-off, which is fine for most filesystems but can leave an application mid-write in an inconsistent state. If you need application-consistent state (a database in the middle of a transaction, for instance), you'd normally quiesce writes first, but for IR purposes crash-consistent is usually exactly what you want anyway — it reflects reality at the moment of acquisition rather than a cleaned-up version of it. The Snapshot-Based Acquisition Workflow This is the sequence Solstice's IR retainer uses when a compute instance needs to be imaged: Snapshot the disk in place, without stopping the instance, so the attacker (if still active) sees no change in behavior: gcloud compute disks snapshot solstice-web01-boot \ --snapshot-names=ir-2026-08-13-solstice-web01-boot \ --zone=us-central1-a --storage-location=us-central1 Create a new disk from that snapshot in an isolated forensics project — never the source project, to keep the evidence off the same IAM boundary the attacker may still have access to: gcloud compute disks create ir-solstice-web01-boot-copy \ --source-snapshot=ir-2026-08-13-solstice-web01-boot \ --zone=us-central1-a --project=solstice-forensics Attach that disk, read-only, to a clean forensics VM you control in the isolated project, and image it from there with dd or dcfldd rather than mounting it directly: sudo dcfldd if=/dev/sdb of=/mnt/evidence/solstice-web01-boot.dd hash=sha256 hashlog=solstice-web01-boot.sha256 Hash the resulting image and the snapshot's own checksum (Google stores one), and record both in your chain of custody documentation before the raw image ever gets analyzed. Two Things That Trip People Up Snapshot IAM permissions are separate from disk permissions — compute.snapshots.create on the disk's project plus compute.disks.createSnapshot are what you actually need, and it's common for an incident responder to have read access to a project but not snapshot-creation rights, which needs to be granted explicitly and shows up itself as an Admin Activity log entry worth noting for your own timeline. storage-location matters — by default a snapshot inherits the disk's region, but you can pin it elsewhere. For a client with data residency requirements, confirm where the snapshot actually lands before you create it, not after. Instance Metadata: Configuration Data and a Favorite Persistence Mechanism Every GCE instance carries a metadata key-value store, reachable from inside the instance at metadata.google.internal and readable via the API from outside it. Two keys matter more than the rest for IR: startup-script and shutdown-script. Whatever's in those keys runs as root (or SYSTEM on Windows) every time the instance boots or shuts down, and because metadata is editable by anyone with compute.instances.setMetadata on the instance, it's a favorite, boring, extremely effective persistence mechanism — no exploit required, just an IAM permission an attacker already has from an earlier step. Pulling the current and, if available, prior metadata values for these keys should be an early step on any compromised instance. gcloud compute instances describe solstice-web01 --zone=us-central1-a \ --format='value(metadata.items.filter("key:startup-script OR key:shutdown-script"))' The corresponding Admin Activity log entry to hunt for is v1.compute.instances.setMetadata — filter for it across a project and you'll catch both the legitimate configuration changes and the illegitimate ones in the same short list. Who Actually Logged In: OS Login vs. Metadata SSH Keys Linux instances support two different ways to manage SSH access, and they have very different forensic value. Metadata-based SSH keys are just public keys pasted into instance or project metadata — anyone holding the matching private key can log in as the associated local username, and there's no per-user audit trail beyond the OS-level auth log on the instance itself. OS Login, by contrast, ties SSH access to the caller's actual Google identity via IAM (roles/compute.osLogin or osAdminLogin), and every login is attributable to a specific principal in Cloud Audit Logs, not just a key that could have been shared or stolen. If OS Login isn't enabled org-wide, you should expect to lean much more heavily on the guest OS's own auth logs (auth.log, /var/log/secure, or Windows Security event logs) than on Cloud Audit Logs for the actual login attribution. Serial Port Output: The Log You Get Even When the OS Won't Cooperate Every instance's serial console output is captured by Google independent of what's happening inside the guest OS, which makes it valuable when an attacker has tampered with or disabled logging inside the instance itself — serial output survives that. It's retained for the life of the instance and pullable even after the instance is stopped, though it clears on deletion, so grab it before you delete anything. gcloud compute instances get-serial-port-output solstice-web01 --zone=us-central1-a > serial-output.log Hyperdisk is Google's newer Persistent Disk generation, offering independently tunable IOPS/throughput and faster snapshot/restore performance than the older pd-* disk types — if you're imaging a Hyperdisk-backed instance the workflow is identical, just faster. Shielded VM (secure boot, vTPM, integrity monitoring) and Confidential VM (memory encryption via AMD SEV or Intel TDX) are also both far more commonly the default posture now , and it's worth checking whether an instance is Confidential VM-enabled before you plan a memory-acquisition approach, since some capture techniques that assume readable guest memory don't behave the same way against an encrypted memory space. 🔎 IR tip: Pull the serial port output and the current metadata (especially startup-script) before you do anything else with a suspect instance — both are cheap, non-invasive, and won't be there anymore if someone deletes the instance while you're still setting up the snapshot workflow. .......................................................................Dean---------------------------------------------------

When people say "compute" in Google Cloud, they're really talking about anything that burns CPU. If it runs code, processes data, or executes workloads — it falls under compute. For investigations, though, not all compute is equal. Some services give you deep visibility. Others hide almost everything. Here's the case we'll come back to throughout this article: a billing alert flags an unexpected spike in Compute Engine usage on a project called fernbridge-prod. Nobody on the team provisioned new VMs. That's your incident — and everything below is how you'd actually work it. Compute Types (Quick Reality Check) Google Cloud groups compute into three big buckets: IaaS (Infrastructure as a Service) — this is where DFIR people feel at home. You control the operating system, installed software, and users, processes, and disks. Examples: Virtual Machines (Compute Engine), Shielded VMs, GPU-backed VMs. 👉 From a forensic standpoint, this is gold. PaaS (Platform as a Service) — here, Google manages the OS and runtime. You configure the app, not the system. Examples: App Engine, Google Kubernetes Engine (GKE), Cloud Run. You still get logs — but host-level evidence is mostly gone. FaaS (Function as a Service) — this is the black box. Examples: Cloud Functions, Cloud Workflows. You deploy code. Google handles everything else. From a DFIR perspective: great logs, almost zero disk or memory visibility. Why We Focus on Compute Engine (VMs) Google keeps adding new compute services all the time. But for DFIR work, Compute Engine VMs are the most valuable. Why? They behave like traditional systems, they generate OS-level artifacts, you can snapshot disks, you can run agents, and you can do "normal" forensics on them. Most of the investigation techniques you use here carry over to other compute types, but VMs give you the deepest access. In the fernbridge-prod case, the mystery workload turns out to be a handful of brand-new Compute Engine VMs — which is exactly why this article focuses here instead of on GKE or Cloud Functions: VMs are where you actually get to do forensics. VM Types (Don't Overthink It) Google offers a huge list of predefined VM types: General-purpose, High-memory, High-CPU, and Extreme performance machines (full list here). For DFIR, the size doesn't matter much — the artifacts do. A few things worth remembering: VMs can run Windows or Linux, they can use custom images, they can host Docker containers, and you can also define custom machine types. Special VM Configurations (DFIR Angle) GPUs are zone-specific add-ons for an existing VM. They're useful when you need to crunch through evidence fast — say, running ML-assisted malware classification — but they're not an especially interesting artifact source themselves. Preemptible VMs are cheap and not guaranteed to run continuously — they can disappear at any time. Picture this: you're investigating a spike in outbound traffic from a web cluster, and half the logs just stop mid-timeline. That's usually not evidence tampering — it's a preemptible VM getting reclaimed. These are about cost optimization, not stability, and they're common in exactly the kind of web clusters where you'd expect to find them. Shielded VMs add Secure Boot, integrity monitoring, and a virtual TPM (vTPM). Security-wise: great. Forensics-wise: mixed — they often come with additional logins that a standard VM wouldn't have, which is one more thing to check during triage. Capturing Evidence: Disk Snapshots (The Cloud Way) In cloud environments, you don't image disks the old-school way. Instead, you: Snapshot the persistent disk attached to the VM Share that snapshot with your DFIR project Copy it Convert it back into a disk Attach it to a forensic VM as read-only Analyze like a normal disk This works whether your DFIR team is in the same org, a different org, or a completely separate third-party firm — it's just permissions. Same concept as AWS and Azure, just different buttons. For fernbridge-prod, day one looks exactly like this — from Cloud Shell or your workstation: # 1. Snapshot the disk on the suspicious VM gcloud compute disks snapshot fernbridge-vm-07 \ --project=fernbridge-prod \ --zone=us-central1-a \ --snapshot-names=incident-snap-01 # 2. Share the snapshot with your DFIR project gcloud compute snapshots add-iam-policy-binding incident-snap-01 \ --project=fernbridge-prod \ --member="user:you@dfir-project.iam.gserviceaccount.com" \ --role="roles/compute.storageAdmin" # 3. From the DFIR project: copy the snapshot into a new disk gcloud compute disks create incident-disk-01 \ --project=dfir-project \ --source-snapshot=projects/fernbridge-prod/global/snapshots/incident-snap-01 \ --zone=us-central1-a # 4. Attach it read-only to your forensic VM gcloud compute instances attach-disk forensic-vm \ --project=dfir-project \ --zone=us-central1-a \ --disk=incident-disk-01 \ --mode=ro From here, it mounts and analyzes like any other read-only disk. But a snapshot alone won't tell you how those VMs got created in the first place — that's where it gets more complicated. The Big Problem With Snapshots Snapshots are single points in time. That's the catch. Attackers don't freeze just because you took a snapshot — processes keep running, credentials keep getting used, lateral movement continues, and data keeps changing. Modern DFIR often means "leave the system running and observe it live." And this is where cloud changes the game. Live Forensics in Google Cloud: Ops Agent Google Cloud provides a built-in way to collect live telemetry from VMs using the Ops Agent. https://docs.cloud.google.com/logging/docs/agent/ops-agent/installation Instead of stopping the VM, you can collect logs, capture system metrics, monitor application behavior, and observe activity over time — crucial when you're scoping an incident, multiple systems are involved, you need timeline visibility, or shutting systems down simply isn't an option. Think of Ops Agent as "your eyes inside a running cloud VM." This is exactly the gap in the fernbridge-prod case — the mining VMs might already be gone by the time you get around to snapshotting anything. If Ops Agent had been running on them, you'd have live telemetry instead of a cold trail. Snapshot vs Live Telemetry (Real Talk) Disk Snapshot — strength: stable evidence. Weakness: single moment in time. Ops Agent — strength: ongoing visibility. Weakness: requires pre-installation. Logs — strength: broad coverage. Weakness: depends on retention. In real investigations, you use all three. Google Cloud Ops Agent — What It Really Does If you've ever investigated a cloud VM and thought "okay, but what's happening right now on this box?" — that's exactly where Google Cloud Ops Agent comes in. At a high level, Ops Agent is just a small agent running inside the VM (Linux or Windows) that watches logs and ships them to Google Cloud Logging. But the important part isn't what it is — it's how it behaves during investigations. Think of Ops Agent Like This Instead of taking one disk snapshot and hoping it captured the right moment, Ops Agent lets you watch the system while it's alive, see logs as they're being generated, and build timelines without ever stopping the VM. For DFIR, that's huge. How Logs Actually Flow (Simple Mental Model) Here's what's happening under the hood: Logs are generated on the VM — syslog, app logs, and auth logs on Linux; .evtx event logs on Windows Ops Agent reads those logs locally It converts them into structured JSON It sends them to the Google Cloud Logging API From there, your normal Log Sinks and Buckets take over So Ops Agent doesn't store logs — it streams them. Fluent Bit Is the Secret Sauce Ops Agent is basically Fluent Bit wearing a Google jacket. Why that matters: Fluent Bit is fast, lightweight, understands a ton of log formats, and can parse messy logs into clean JSON. This is why Ops Agent works well even on busy systems — it's not some bloated collector. What the Config Actually Looks Like Fluent Bit doesn't guess what to watch — it reads a config file that spells it out. A basic Linux syslog receiver looks like this: logging: receivers: syslog: type: files include_paths: - /var/log/messages - /var/log/syslog service: pipelines: default_pipeline: receivers: [syslog] Why this matters for an investigation: this file tells you exactly which paths Ops Agent is watching — and just as importantly, which paths it isn't. If an attacker knows Ops Agent is running, quietly editing this config (or stopping the agent entirely) is a clean way to blind logging without triggering anything obvious. That makes two commands you should run on every suspicious VM, not just the ones where you already suspect tampering: # Is the agent actually running? sudo systemctl status google-cloud-ops-agent # What is it actually configured to collect — and when was this last touched? cat /etc/google-cloud-ops-agent/config.yaml stat /etc/google-cloud-ops-agent/config.yaml (On Windows, the same file lives at C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml.) A config that's been trimmed down, or a modification timestamp that lines up with the intrusion, is itself an indicator. Structured vs Unstructured Logs (Why You Care) Not all logs are born equal. An unstructured line like: Feb 2 10:41:02 sshd[2211]: Failed password for root becomes, once Ops Agent gets to it, a structured record with a timestamp, hostname, process name, severity, and cleanly parsed fields — meaning you can search cleanly, build timelines, and correlate across systems instead of grepping raw text. Ops Agent also adds its own timestamp when the log is received. This matters because formats like syslog often don't include full date or timezone info — that extra timestamp saves investigations more often than people realize. What Logs Get Collected by Default? Once Ops Agent is installed, it doesn't sit idle. Out of the box, it starts collecting OS logs, common application logs, and auth-related events, and each one is tagged with the VM name, project name, and resource identifiers — that's how Google keeps logs from different projects from leaking into each other. Finding Ops Agent Logs in Log Explorer Logs follow a predictable pattern: projects/[project-name]/logs/[log-name]. To narrow it down to one system, combine that with resource_name (the VM name). This is why Log Explorer searches often look a bit "busy" — you're filtering both what happened and where it happened. Back to fernbridge-prod: this is the actual query you'd run, either in Log Explorer or from the CLI, to see whether Ops Agent caught anything before the instances were deleted: gcloud logging read \ 'logName="projects/fernbridge-prod/logs/syslog" AND resource.labels.instance_id="INSTANCE_ID"' \ --project=fernbridge-prod \ --limit=50 \ --format=json Swap syslog for the log name you're after, and INSTANCE_ID for the VM's numeric instance ID (not its name) — that's what resource.labels.instance_id actually stores. Installing Ops Agent (Zero Drama) You've got two main options: Option 1 — from inside the VM (manual, good for locked-down environments): SSH in, then run: curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh sudo bash add-google-cloud-ops-agent-repo.sh --also-install The agent starts automatically. Confirm it: sudo systemctl status google-cloud-ops-agent Option 2 — via Agent Monitoring Service (most common): from the Compute Engine or Cloud Monitoring console, click Install Agent — Google hands you the same script above, pre-filled, to run on the target VM (installation docs). Even if you install it manually, Google Cloud still detects the agent, shows its status, and tells you which VMs are covered — so you always have visibility into coverage. Why Ops Agent Beats Snapshots (In Many Cases) Snapshots are useful, but they're frozen in time. Ops Agent gives you continuous visibility, ongoing timelines, and evidence while the attacker is still active — and in modern cloud incidents, you rarely investigate just one VM, you rarely have the luxury of stopping systems, and you need to observe rather than interrupt. Ops Agent fits that reality. Multi-Cloud Bonus (This Part Is Underrated) Ops Agent isn't limited to Google Cloud — you can run it on AWS Linux and Windows VMs and send those logs into Google Cloud Logging, using Google Cloud as a central log hub. This is surprisingly common in hybrid environments, migrations, and organizations tired of juggling tools. You just need an AWS connector project — after that, logs flow the same way. What About the Legacy Agent? You'll still see it in the wild. Reality check: it's older, heavier, based on fluentd, and slowly being phased out. Ops Agent uses Fluent Bit, uses less memory, scales better, and is clearly the future. If you see the Legacy Agent during an investigation, don't panic — just know it behaves differently and may produce slightly different logs. Compute Engine Attacks Aren't Theoretical (2025 Data) If the fernbridge-prod scenario sounds oddly specific, that's because it's not far from real life. Since 2023, Google has been tracking a financially motivated actor it calls TRIPLESTRENGTH doing almost exactly this: stealing credentials and session cookies (some sourced from Racoon infostealer logs), hijacking cloud service accounts, and spinning up Compute Engine instances purely to mine cryptocurrency. Their playbook got smarter over time. Early campaigns just abused compromised accounts to create mining VMs directly. Later, they escalated — abuse a highly privileged account to add an attacker-controlled account as a billing contact, then use that billing access to spin up much larger compute resources for mining. Tooling of choice: the unMiner app (which bundles several popular cryptocurrency miners into one package) paired with the unMineable pool, cashing out in TRX. Google's own 2025 telemetry backs this up — in observed post-compromise activity, "Creation/Deletion of cloud instances" showed up as its own tracked category (4.4% of incidents), smaller than lateral movement (62.2%) but exactly the kind of spike Ops Agent and audit logs are built to catch. DFIR takeaway: an unexpected compute spike or a new billing contact on a project is the pattern to check for first. DFIR Takeaway Ops Agent = live visibility Snapshots = point-in-time evidence Logs + telemetry = modern cloud forensics Fluent Bit parsing = clean timelines Project-level tagging = containment and scoping If snapshots tell you what existed, Ops Agent tells you what's happening. fernbridge-prod, resolved: it turned out to be a compromised service account with billing-contact privileges, used to spin up compute for mining — the TRIPLESTRENGTH pattern almost exactly. Ops Agent hadn't been installed on the new VMs (they were too new), but the Admin Activity audit log for the IAM change on the billing account was enough to reconstruct the timeline — the kind of query that finds it: gcloud logging read \ 'protoPayload.methodName="SetIamPolicy" AND resource.type="billing_account"' \ --project=fernbridge-prod \ --freshness=30d \ --format=json That single IAM change — an unfamiliar account added as a billing contact — was the actual root cause. Everything after it, including every VM Ops Agent never got a chance to watch, followed from that one event. ----------------------------------------------------Dean--------------------------------------------

Why Google Cloud Matters for DFIR Most enterprise workloads still run on-premise, not because cloud platforms are weak, but because migration introduces architectural and operational complexity. Unlike traditional environments, investigators in Google Cloud cannot rely on physical access, predictable network paths, or full host-level visibility. Instead, identity events, service-level logs, and resource metadata become the primary evidence sources. Today, most of the world's enterprise computing still happens on-premise. It hasn't moved to the cloud yet, because the path forward is complex and daunting, and full of difficult decisions. How do you modernize in-place without having to jump completely to the cloud? How do you bridge incompatible architectures while you transition? A nd how do you maintain flexibility and avoid lock-in?” — Sundar Pichai, Google's CEO, Google Cloud Next 2019 And the scale behind it: at the time this curriculum was written, Google Cloud was the third-largest cloud provider by market share (behind AWS and Azure) but the fastest-growing of the three, with analysts forecasting overall public cloud spend to grow 18.4% that year. Faster growth means more new, less-mature deployments — exactly why cloud infrastructure keeps landing in threat actors' crosshairs. ------------------------------------------------------------------------------------------------------------ Google Cloud Global Architecture (DFIR Angle) Google Cloud is built on regions and zones: Regions are geographic areas (e.g., Europe-West, US-Central) Zones are isolated deployment areas within regions Why this matters for DFIR: Evidence may be generated in one zone and stored or processed in another Data residency and regulatory requirements apply when collecting evidence Network paths inside Google Cloud are abstracted and not traceable You cannot reconstruct packet paths like on-prem networks. Investigations must focus on what happened, who did it, and which resource was affected, not how traffic flowed. Worth naming the real scale here instead of leaving “regions and zones” abstract: Google Cloud runs 25 regions and 76 zones, with more added regularly. And there's a concrete reason network paths are abstracted: Google interconnects those regions using a mix of its own dedicated network links and shared submarine cables. Redundancy varies a lot by geography too — the US has heavy ingress/egress redundancy, while regions like Australia have comparatively limited links. That's part of why you can't reconstruct a packet path the way you would on-prem — the routing is spread across infrastructure you don't get visibility into, by design. ------------------------------------------------------------------------------------------------------------ Core Google Cloud Services for Incident Response & Forensics These five services provide the highest forensic value during cloud incidents: Identity & Access Management (IAM) Compute Engine (VMs) Cloud Logging Cloud Networking Cloud Storage Buckets Almost every cloud compromise touches IAM + Logging + Compute ------------------------------------------------------------------------------------------------------------ Google Cloud Resource Hierarchy (Critical for Investigators) Google Cloud follows a strict hierarchy: Organization └── Folder(s) └── Project(s) └── Resource(s) Breakdown Organization Highest level in Google Cloud Usually mapped to a company’s domain Often linked with Google Workspace Central point for IAM and security policies From a DFIR perspective, this is where global controls and investigation-wide visibility are established. Folders Used to group teams, environments, or functions Policies applied here affect everything beneath them Can have multiple layers (sub-folders) DFIR benefit: Separate DFIR resources from production Apply investigation-specific policies Limit blast radius during incidents Projects Where services actually run Billing and resource ownership boundary Minimum requirement for creating resources Most compromises are identified inside projects, making them the primary focus during investigations. Resources Compute, Storage, Networking, Logging, etc. Inherit permissions and constraints from above Generate most forensic artifacts Example Organization: novaharbor.io (the company's primary domain) Folder: Incident-Response — isolates DFIR tooling from production Sub-folder: dfir-sandbox — a further split for one-off investigation VMs Project: evidence-processing-prod — where the actual acquisition/processing VMs and storage buckets live Resources: a Compute VM running Plaso, a Storage Bucket holding disk images Policies applied at the Incident-Response folder level — read-only visibility for DFIR staff, a ban on disabling logging — automatically apply down through dfir-sandbox and evidence-processing-prod without touching each resource individually. That's the practical payoff of the hierarchy: set it once at the folder, and every project and resource under it inherits it. ------------------------------------------------------------------------------------------------------------ IAM vs Policies (Very Important Distinction) Concept Controls IAM Who can access resources Policies / Constraints What resources can do Example: IAM: Who can access a VM Policy: Whether logging can be disabled on that VM Use policies to: Prevent attackers from: Disabling logs Creating resources in rogue regions Lock down compromised assets instantly ------------------------------------------------------------------------------------------------------------ Google Cloud Organization Benefits for DFIR What DFIR Teams Can Do Quickly Apply constraints globally Allowed VM templates Approved regions only Grant DFIR read-only visibility everywhere Prevent destructive actions: Turning off logging Deleting evidence Isolate compromised projects without downtime ------------------------------------------------------------------------------------------------------------ Key Google Cloud Resources Attackers Abuse Focus first on these during investigations: Compute & Platform Compute Engine App Engine Kubernetes Engine Storage Cloud Storage Filestore Ops & Visibility Logging Monitoring Networking VPC Network Security Services ------------------------------------------------------------------------------------------------------------ Data Transfer Pricing (Why DFIR Teams Get Burned) Important Cost Rules Same zone + internal IP → No cost Different zone or region → Cost VM-to-VM traffic can incur: Egress Response ingress (TCP) Google Cloud was not charging for ingress traffic into a resource — only egress and cross-zone/region movement carried a cost. That's subject to change, so always check Google Cloud's current pricing page before assuming it still holds; don't bank an evidence-transfer plan on a pricing detail that could shift. DFIR Best Practices ✅ Place DFIR VMs in: Same region Same zone Use internal IPs only Avoid: Moving evidence across zones Exporting raw evidence unnecessarily ------------------------------------------------------------------------------------------------------------ Cost Reduction Options for DFIR 1. Preemptible VMs Cheaper No guaranteed CPU Good for: Distributed tasks Bad for: Single-VM processing (e.g., Plaso) 2. Keep Processing Local Same zone = free internal traffic Faster acquisition & analysis 3. Be Smart with BigQuery Charged per bytes processed Reduce cost by: Fewer queries Better SQL Documenting results to avoid reruns BigQuery actually bills on two separate axes: the cost of holding data in a table, and the cost of querying it. The tips above (fewer queries, better SQL, documenting results) only reduce the second one — there isn't much you can do to reduce storage/holding cost once evidence is sitting in BigQuery, so factor that in when deciding how long to keep large exported log sets there versus moving them to cheaper cold storage. ------------------------------------------------------------------------------------------------------------ Key Takeaways IAM + Policies = Cloud Incident Control Plane Folder & project structure directly affects response speed Logging enforcement is your strongest defense Data movement = cost + legal risk DFIR architecture planning matters before incidents happen --------------------------------------------Dean-----------------------------------------------------------

If you’ve worked in Google Workspace long enough, you already know this truth: Google Drive is where data leaks love to happen. Not always malicious. Sometimes it’s just: “Oops, shared it publicly” “Oops, shared it with the wrong domain” “Oops, didn’t realize Anyone with the link means literally anyone” So when data exposure happens, we usually care about two questions: What happened to the file? Can we still access or recover it? That’s where Google Drive investigation tools come in. ------------------------------------------------------------------------------------------------------------- Tool 1: Google Drive Log Events (Your Timeline, Not Your Files) Think of the Drive Log Events as your CCTV footage, not the evidence locker. What it’s good at: Showing who did what Showing when it happened Showing permission changes Near real-time visibility (usually within minutes) What it’s not good at: Accessing files Showing file contents Tracking anonymous viewers or downloads Key Things to Know About Drive Log Events Let’s break this down simply: Keeps 6 months of history Logs actions like: File creation Sharing changes Permission updates Deletions Does NOT give you the file itself CSV export is limited to 100,000 rows Unauthenticated access is only logged for editing Viewing or downloading by anonymous users? ❌ Not logged So if a file was publicly shared and downloaded 1,000 times anonymously — the audit log will not tell you that. Painful, but important to know upfront. Where Are Drive Log events Now? Earlier, Drive log events lived in their own section. That changed. Today, Drive Log Events live inside in the Google Admin Console. Inside Investigator, you can: Filter events Use AND / OR logic Group by fields (user, document, event type) Search using partial matches One warning though ⚠️Even though logs are generated quickly, some events can lag up to 12 hours before showing up. ------------------------------------------------------------------------------------------------------------ Tool 2: Google Vault (This Is Where the Files Live) If Audit Logs are the timeline, Vault is the evidence room. Vault is what you use when: You actually need the document A file was deleted A user “accidentally” removed something important But Vault comes with conditions. What Vault Can Do Access files in user Drives Access deleted files Apply holds Enforce retention rules What Vault Cannot Do Give you an audit trail Tell you who did what and when It’s access, not visibility. Deletion Timelines (This Matters a LOT) Here’s the reality of deleted files: When a user deletes a file → it goes to Trash Trash keeps files for 30 days Once removed from Trash: Admins have 25 more days to recover (without Vault) ·With Vault, that window extends to 25–40 days total from the original deletion date — the extra 15 days depends on when Google actually purges the file from its own storage Custom retention rules or holds = files stay longer If Vault is enabled, you can often recover files without restoring the user account. ------------------------------------------------------------------------------------------------------------ Alternate File Recovery Scenarios (The “Oh No” Cases) Case 1: Active User Deleted Files Trash keeps files for 30 days After Trash deletion: Admins have 25 days to restore Restore options: Original location Shared Drive No Vault license? After 25 days — game over. Case 2: Deleted User Account This one catches teams off guard. Deleted user accounts can be restored for 20 days Files can only be recovered if: The user account is restored first Or Vault is used Ownership transfer is another option: Files move to another user’s Drive Again — Vault makes life easier here. ------------------------------------------------------------------------------------------------------------ Exporting Drive Logs Once you've found the events that matter, you don't want to sit there scrolling — you want them out of the console and into something you can search, filter, or hand off. Where to export: From the Investigation Tool (Admin Console → Security → Investigation Tool), run your filtered search, then use Export → CSV. Vault has its own export flow (Vault → Search → Export), if you're pulling files or metadata for a hold rather than just the audit trail. What you get: A CSV of the events matching your filters — actor, timestamp, IP, document ID, visibility change, and the other fields covered above. Capped at 100,000 rows per export. If you're hunting something broad (a domain-wide sharing change, for example), narrow the filter first or you'll quietly lose rows past the ceiling. One more thing to know: Exports don't include file contents — same limitation as the log events themselves. You're exporting the trail, not the evidence. Large exports can take a few minutes to generate; Google emails you a download link rather than streaming it back instantly. Clunky, but it's the difference between "I remember seeing something" and having a file you can actually attach to a ticket. Once you've got that CSV, don't just open it in Excel and start scrolling. Eric Zimmerman's Timeline Explorer handles this kind of exported log data far better — filter, sort, and pivot without losing your place, and you're not fighting Excel's row limits or crash-on-large-file quirks. Especially useful if you're keeping the analysis off the (possibly compromised) Workspace tenant entirely. ------------------------------------------------------------------------------------------------------------ Important Fields You’ll Actually Use During an Investigation Let’s translate the useful ones into investigator language: Document ID This is gold. Unique across all Google Workspace tenants Same ID you see in the document URL Perfect for matching phishing URLs to actual Drive files Owner vs Actor Owner: Who owns the file Actor (User field): Who performed the action These are often not the same person. Visibility & Prior Visibility This tells the real story. Prior Visibility → what access looked like before Visibility → what access looks like now This is how you catch: Private → Public changes Internal → External sharing Domain-wide exposure IP Address Extremely useful for: Geo anomalies Impossible travel Correlation with other Workspace logs One more field worth knowing: ### Billable · Shows whether the document counts toward the user's storage quota · Only actually populated on the Essentials edition Small field, easy to skip — but if you're chasing a storage-quota mystery on an Essentials tenant, this is the column that explains it. ------------------------------------------------------------------------------------------------------------ Final Thoughts (The Big Picture) Drive log events — who did what, and when Vault — the actual file, if you need to see or recover it Visibility & Prior Visibility — how exposure changed, and when it changed An export — so the timeline survives the investigation, not just your memory of it No single tool gives you the full picture. Put them together, and you've got one. ---------------------------------------------Dean-----------------------------------------------------------

Up until now, most Google Workspace investigations start in one of two places: The Admin Console Or Workspace APIs Both are useful. Both have limits. At some point though, especially in larger or more mature environments, logs don’t just live inside Workspace anymore — they’re exported into Google Cloud. And once that happens, the Admin Console alone isn’t enough. That’s where gcloud comes in. ------------------------------------------------------------------------------------------------------ What gcloud Actually Is (And What It Isn’t) Let’s clear this up first. gcloud is not some hacking tool or special DFIR-only utility. It’s the official command-line interface for Google Cloud, bundled as part of the Google Cloud SDK. Think of it as: The terminal version of Google Cloud Console It works on: Windows macOS Linux And it’s designed for: Automation Scripting Command-line access to Cloud services What it’s not designed for: Being embedded into applications Acting like a full SDK inside other tools For investigators, that’s perfect — because we want read-only, controlled, scriptable access. ------------------------------------------------------------------------------------------------------ Why Use gcloud Instead of the Web Console? You can view logs in the Google Cloud web UI. That works fine for quick checks. But gcloud gives us a few big advantages: Better control over log extraction Easier time-based filtering Clean JSON output (huge win for DFIR) Works well from isolated IR workstations Easy to feed into tools like SOF-ELK If you’re doing serious timeline reconstruction or long-range log analysis, gcloud is simply more practical. ---------------------------------------------------------------------------------------------------------- Authentication: Who Is gcloud Acting As? Before gcloud can do anything, it needs an identity. You have two main options: 1. Service Account Good for automation, repeatable workflows, and controlled access. 2. User Account More common during investigations, especially when speed matters. In both cases, OAuth is used to authorize access to Google Cloud. And here’s the key permission you need to remember: Private Logs Viewer ---------------------------------------------------------------------------------------------------------- Why “Private Logs Viewer” Matters Google Cloud has two commonly confused roles: Logs Viewer Private Logs Viewer For investigations, Logs Viewer is not enough. Private Logs Viewer gives you access to: Audit logs Logs containing IP addresses Sensitive user activity metadata That’s exactly what we care about during IR. The good news? Both roles are read-only. You cannot modify or delete logs with either. So you get visibility without risk. ---------------------------------------------------------------------------------------------------------- Installing and Preparing gcloud https://docs.cloud.google.com/sdk/docs/install-sdk Once you install the gcloud CLI on your investigation host, there are three things you always need to tell it: Which project you’re working in Who you are (authentication) What logs you want Projects matter because logs live inside Google Cloud Projects, not “Workspace” directly. Workspace just sends logs there. One flag worth knowing if your investigation host doesn't have a GUI or a browser: gcloud init --no-browser. It walks you through the same setup, but instead of popping open a browser window, it hands you a URL to open somewhere else and a verification code to paste back in — exactly what you need on a locked-down, headless IR box. ---------------------------------------------------------------------------------------------------------- Logging Buckets: Where Your Logs Actually Live In Google Cloud, logs are stored in Logging Buckets. Buckets are not sized by data volume — they’re sized by retention days. In almost every project, you’ll see at least: _Required _Default Anything beyond that was created intentionally by admins. As an investigator, one field becomes very important here: retention_days Because it defines: How far back you can go Whether old evidence still exists ---------------------------------------------------------------------------------------------------------- Narrowing Down to Google Workspace Logs Not all logs in Google Cloud are Workspace logs. So instead of searching everything, we filter by service name. Workspace-related logs usually come from: admin.googleapis.com cloudidentity.googleapis.com login.googleapis.com oauth2.googleapis.com This alone removes a ton of noise. ---------------------------------------------------------------------------------------------------------- Time Ranges: Never Forget This (Seriously) This is one of those gcloud “gotchas” that burns people. If you don’t specify a time range, gcloud will: Return only 10 log entries Not 10 pages. Not 10 minutes. Just 10 lines. So every serious query must include a timestamp filter. ---------------------------------------------------------------------------------------------------------- Pulling Logs the Right Way (And Why JSON Matters) When we extract logs, we don’t want pretty output — we want machine-consumable evidence. That’s why we force JSON output. Here’s what a real-world Workspace log pull looks like: gcloud logging read "protoPayload.serviceName=(admin.googleapis.com OR cloudidentity.googleapis.com OR login.googleapis.com OR oauth2.googleapis.com) AND timestamp>=\"2026-01-01T00:00:00Z\" AND timestamp<=\"2026-01-30T00:00:00Z\"" --format=json > gws_logs_in_gcp.json What’s happening here: We limit results to Workspace-related services We define a clear investigation window We output everything as JSON We write it to a file for offline analysis ---------------------------------------------------------------------------------------------------------- Why gcloud Is So Useful in DFIR The real value of gcloud isn’t just “getting logs”. It’s that you can: Re-run queries consistently Adjust time windows precisely Preserve raw evidence Avoid UI-based filtering mistakes Work even when the web console feels slow or limited And once authenticated, gcloud can do anything your account is authorized to do — we just happen to care about logs. ---------------------------------------------------------------------------------------------------------- One Last Thing: Logging Out Matters This sounds basic, but it’s important. When you authenticate with gcloud, you’re opening an active API session. Google Cloud has no way of knowing you’re “done” unless you explicitly log out. From an investigation hygiene perspective: Always revoke or log out Especially on shared IR systems Especially after admin-level access ---------------------------------------------------------------------------------------------------------- Final Thoughts Using gcloud for Google Workspace investigations is one of those skills that feels optional — until it suddenly isn’t. When logs move to Google Cloud: The Admin Console becomes secondary APIs don’t always give full visibility CLI access becomes your best friend ----------------------------------------Dean------------------------------------------------------------

The core idea In Google Cloud, Service Accounts are identities for machines, not humans .They are used by resources like VMs, Cloud Functions, Kubernetes, etc. to talk to other Google Cloud services. Unlike AWS (where users can directly generate API keys), Google Cloud forces you to use Service Accounts when you want: Programmatic access Static credentials Non-interactive authentication So:👉 If code needs access, it almost always runs as a Service Account. ---------------------------------------------------------------------------------------------- What actually happens when you create a VM When you create a VM: Google Cloud automatically creates (or assigns) a Service Account That Service Account: Appears in IAM Can be granted roles just like a user Is used by the VM to access other resources (Storage, APIs, databases) You can delete this Service Account from IAM —but then your VM will break if it needs to talk to anything else. Best practice in reality : Don’t delete it — restrict its permissions. ---------------------------------------------------------------------------------------------- The real danger: Basic Roles (Owner & Editor) On paper Google Cloud has Basic Roles: Viewer Editor Owner They were created early on to make things “easy”. In practice (this is where things go wrong) The Editor role is dangerously powerful: An account with Editor can: Modify resources Create API keys Use actAs to impersonate other accounts Create credentials for other accounts Key insight: Editor ≈ Owner (from an attacker’s point of view) ---------------------------------------------------------------------------------------------- Why attackers love Editor accounts If a threat actor compromises any account with Editor: They can create an API key They can impersonate (actAs) higher-privileged accounts They can effectively privilege-escalate to Owner This is not theoretical — it’s abused in real incidents. ---------------------------------------------------------------------------------------------- Real-world lateral movement scenario Let’s walk through the actual attack flow. Step 1 – Environment setup (normal behavior) Infra team creates a Development Project VMs are deployed for developers Each VM has a Service Account Everything is isolated. Looks safe. Step 2 – Developer needs storage (very common) Developer needs persistent storage Creates a Cloud Storage Bucket Grants the VM’s Service Account Editor access to the bucket From the developer’s perspective: “It works, job done.” Step 3 – Credentials exposure One of the following happens: Service Account key committed to GitHub Credentials stored in code VM is compromised and metadata server is abused Now the attacker has:👉 Service Account credentials with Editor permissions Step 4 – Privilege escalation With Editor access, the attacker can: Create new API keys Impersonate other IAM accounts Take over Owner accounts in the same project Step 5 – Organization-level impact If any Org-level bound account exists in that project: The attacker can impersonate it Escalate to the Organization Gain control over: All projects All resources Entire cloud environment Single Service Account compromise → Full Org takeover Someone already automated this exact chain: gcploit, built by Dylan Ayrey. It maps every resource reachable via actAs starting from a single compromised Service Account — turning the manual 5-step walkthrough above into a single tool run. Worth knowing whether you're investigating a breach (this may be exactly what the attacker ran) or hardening an environment (run it against your own org first). ---------------------------------------------------------------------------------------------- Why this is hard to detect (investigation challenge) The IAM visibility problem In Google Cloud: Resource owners decide access There’s no single place that shows: “What does this account have access to across the Org?” This creates: Hidden trust relationships Accidental cross-project access Silent privilege escalation paths ---------------------------------------------------------------------------------------------- TInvestigator & Defender mindset When you’re investigating or hardening Google Cloud: Red flags to look for Service Accounts with Editor role Shared Service Accounts across projects Exposed Service Account keys Unexpected actAs activity API keys created by non-human identities ---------------------------------------------------------------------------------------------- Defensive mindset shift ❌ “Editor is fine for dev”✅ “Editor is a privilege escalation waiting to happen” ---------------------------------------------------------------------------------------------- One-line summary In Google Cloud, Service Accounts with Editor permissions act as silent trust bridges—once compromised, they enable privilege escalation, lateral movement, and even full organization takeover without deploying malware. -----------------------------------------------Dean---------------------------------------------

Let’s talk about something that often comes up during Google Workspace investigations: how do we actually collect logs and evidence properly? If you’ve ever worked an incident involving Google Workspace, you already know that the platform gives you a lot of data—but not all of it is equally easy to collect or analyze. Broadly speaking, there are two main ways to collect evidence from Google Workspace: Using the Workspace Admin interface (UI) Using the Workspace Admin SDK / APIs On paper, both give you access to similar information. In reality, they behave quite differently—and those differences really matter during forensic analysis Let’s break this down in a simple, practical way. ------------------------------------------------------------------------------------------------------------- Option 1: Using the Google Workspace Admin Interface The Admin interface is usually where everyone starts—and honestly, it’s not a bad place to begin. It gives you a visual and human-friendly way to explore logs. You can click through different sections, filter events, and clearly see what’s going on. ' This is especially useful when: You’re doing a quick triage You need to show evidence to a manager, legal team, or client You want to visually confirm suspicious activity The downside? All the useful data is scattered across different screens. If you want to investigate a full Workspace compromise, you’ll likely need to: Jump between Drive logs Check login and authentication activity Review OAuth and third‑party app access Inspect Admin console changes Each of these lives in a different place. That means a lot of clicking, filtering, exporting, and repeating the process again and again. It works—but it’s slow. Export limitations There are a few important limitations to keep in mind: You can only export 10,000 or 100,000 events per log type. If you exceed that limit, you must split your search into smaller time ranges Logs are exported only as Google Sheets (GSheet) from the UI You can later convert those sheets into CSV, but it’s an extra step—and not ideal if you’re planning to ingest logs into a SIEM or timeline tool. Name the tool for working with those CSV exports If you go the UI/GSheet route, don't just open these in Excel — there's a purpose-built free tool for this: Eric Zimmerman's Timeline Explorer. It imports the CSV export directly and lets you search and filter it far faster than Excel ever will. One thing worth checking every time you use it: look at the highest row number Timeline Explorer shows you. If it lands right at 10,000 or 100,000, that's not a coincidence — you've hit the UI export cap, and there's more data you haven't seen yet. Split your time range and pull the rest. Option 2: Collecting Logs via the Workspace Admin SDK (API) Now this is where things get really interesting for forensic work. The Workspace Admin SDK allows you to collect logs programmatically using API calls. Once set up, this becomes the fastest and most consistent way to gather evidence. Yes, the initial setup takes some effort—you’ll need: A Service Account The right Workspace permissions Some basic scripting knowledge But once that’s done, everything becomes repeatable and scalable. Types of reports you can collect Using the API, you can pull two main types of reports: 1. Activity Reports These tell you what actually happened across Workspace services, including: Google Drive activity Authentication and login events OAuth and third‑party application access Admin console changes These are gold during investigations because they help you track changes, abuse, and attacker actions. 2. Usage Reports These focus more on how user accounts are being used over time. They’re great for spotting anomalies or misuse patterns. Why investigators prefer API logs There are several big advantages here: No event limits like the UI exports Logs are returned in JSON format, which is perfect for: SIEM ingestion Timeline creation Custom parsing and analysis All timestamps are in UTC, which avoids time zone confusion Collection can be fully scripted, ensuring consistency every time In short: if you’re doing a serious investigation, the API approach is hard to beat. ------------------------------------------------------------------------------------------------------------- Option 3: Sending Google Workspace Logs to Google Cloud Logging There’s a third option that often gets overlooked—but it’s extremely powerful. Google Workspace can send certain logs directly to Google Cloud Logging. This allows you to: Retain logs for a much longer period Query them using Cloud Log Explorer Correlate Workspace logs with other Google Cloud activity You Must enabled the sharing which is disabeld by default The catch Not all Workspace logs are sent to Google Cloud. Only five log types are forwarded—and while these are some of the most valuable ones for investigations, they don’t always tell the full story. Name the five log types that actually get forwarded Specifically, these five: Admin Activities, Enterprise Groups, Login Audit, OAuth Token, and SAML. These map to four service names you'll actually query in Cloud Logging (admin.googleapis.com, cloudidentity.googleapis.com, login.googleapis.com, oauth2.googleapis.com) — which lines up with the query further down this article. Knowing the five by name up front makes it obvious at a glance whether the data you need is even in scope for this method, instead of finding out the hard way mid-investigation. For example: Email transit and email access logs are not included You cannot customize which logs are sent Google decides what gets forwarded—you only choose whether forwarding is enabled or not So while this method is fantastic for long‑term visibility, it should be seen as a complement, not a replacement, for API‑based collection. ------------------------------------------------------------------------------------------------------------- Permissions: A Common Roadblock If you try to search Workspace logs in Google Cloud and run into permission errors—don’t panic. This usually means your account doesn’t have enough rights to query logs. https://docs.cloud.google.com/logging/docs/access-control The fix is simple: Go to IAM & Admin in Google Cloud Grant the appropriate role (typically Logging Admin or equivalent) Once that’s done, Log Explorer will start behaving as expected. Log Explorer Example Querying Workspace Logs in Google Cloud When Workspace logs arrive in Google Cloud, they are spread across a few service names. To search them together, you can use a query like this in Log Explorer: protoPayload.serviceName = ( "admin.googleapis.com" OR "cloudidentity.googleapis.com" OR "login.googleapis.com" OR "oauth2.googleapis.com" ) Example: protoPayload.serviceName = ( "login.googleapis.com" ) One important thing to remember: you need to be viewing logs at the root organization level in Google Cloud. ------------------------------------------------------------------------------------------------------------- Final Thoughts If we simplify everything: Admin UI → great for quick checks and visual walkthroughs Admin SDK / API → best for fast, consistent, forensic‑grade evidence collection Google Cloud Logging → excellent for long‑term retention and centralized querying In real investigations, the strongest approach is usually a combination of all three. ------------------------------------------Dean-------------------------------------------------------------

Let me be honest upfront: this setup looks scary the first time you see it. Google makes you jump back and forth between Google Cloud Console and Google Workspace Admin, and it feels like you’re doing something wrong the entire time. You’re not. That’s just how Google designed it. Once you understand the full flow, everything suddenly clicks. This walkthrough assumes: You are a Google Workspace Super Admin You want to collect audit / activity logs using the Admin SDK – Reports API Big picture first (so you don’t get lost) You will work in two places: Google Cloud Console Create a project Enable APIs Create a service account Google Workspace Admin Console Trust that service account using domain-wide delegation Google Workspace itself does not have service accounts. That’s why Google Cloud is involved at all. We’re basically borrowing Google Cloud’s identity system to talk to Workspace. Step 1: Create a Google Cloud Project (same org as Workspace) Start here: 👉 https://console.cloud.google.com Click the project dropdown in the top bar Select New Project Set: Project name: workspace-log-collection Organization: must be the same org as your Workspace tenant Click Create That’s it. Important thing to understand: you are not deploying servers, VMs, or storage. This project is just a container to hold APIs and a service account. Step 2: Enable the required APIs (this is mandatory) Google locks everything by default, so we have to explicitly enable what we need. Inside your new project: Go to APIs & Services → Library Search for and enable: Admin SDK API (this is the key one) Optional (only if you plan to query these later): Google Drive API Gmail API Calendar API The optional Vault API scope One more optional one worth knowing about: G Suite Vault API. Enable this too if you ever want the same service account to also pull Vault-specific data (holds, exports, search results) instead of just audit/activity logs. It's outside the scope of a basic log-pulling setup like this one, but it's the exact same pattern if you decide to expand later — same project, same service account, just one more API enabled. For audit and activity logs, Admin SDK alone is enough. If this API is not enabled, your script will fail even if every permission looks perfect. Step 3: Configure OAuth Consent Screen (yes, even for service accounts) This step confuses almost everyone. Even though we’re using a service account, Google still requires an OAuth consent screen to exist. Go to APIs & Services → OAuth consent screen Choose Internal You only see this option because the project is under a Workspace org Fill in the basics: App name: Workspace Log Collector User support email: your admin email Developer contact email: your admin email Click Save and Continue On the Scopes page → just click Save and Continue Finish You do not need to publish the app externally. Think of this as telling Google: “Yes, this project is allowed to request Workspace APIs.” Step 4: Create the Service Account Now we create the identity that will actually pull logs. Go to IAM & Admin → Service Accounts Click Create Service Account Set: Name: workspace-log-reader Click Create and Continue Skip role assignment (no GCP roles required) Click Done At this point, the service account exists—but it can’t do anything yet. Step 5: Enable Domain-Wide Delegation (critical step) This is where most people miss a checkbox and everything breaks. Click the service account you just created Open the Details tab Click Show domain-wide delegation Check Enable Google Workspace Domain-wide Delegation Save Now copy the Client ID. You’ll need it immediately. This setting allows the service account to act on behalf of users in your domain—but only for scopes you explicitly allow. Step 6: Trust the Service Account in Google Workspace Now we jump back to Workspace. Go to Google Admin Console 👉 https://admin.google.com Navigate to: Security → API controls → Domain-wide delegation Click Add new Enter: Client ID: (from the service account) OAuth scopes: https://www.googleapis.com/auth/admin.reports.audit.readonly https://www.googleapis.com/auth/admin.reports.usage.readonly Click Authorize This is the trust handshake between Workspace and Google Cloud. Without this step, every API call will be denied. Step 7: Create and download a Service Account key You’ll need credentials for your script or tool. Go back to Google Cloud Console → Service Accounts Select your service account Open Keys → Add key → Create new key Choose JSON Download the file ⚠️ This JSON file is effectively a password. Store it securely. Step 8: Using the Service Account to pull logs When you actually query the Admin SDK API: Authenticate using the JSON key Enable domain-wide delegation Impersonate a Workspace admin user (very important) Example conceptually: Delegated user: admin@yourdomain.com API: Admin SDK – Reports API Logs belong to the domain, not the service account, which is why impersonation is required. The actual tool this walkthrough is building toward (biggest gap) You don't have to build this script yourself — there's already a purpose-built one for exactly this job. The script is gws-get-logs.py, written by Megan Roddie, in the https://github.com/dlcowen/sansfor509 (under the GWS folder). It handles the JSON-key auth, the delegation, and the impersonation for you — you just point it at your setup. Before running it, create a config.json alongside the script: { "creds_path": "./credentials.json", "delegated_creds": "admin@yourdomain.com", "output_path": "./output" } creds_path — the JSON key file you downloaded in Step 7 delegated_creds — the admin account being impersonated (the same idea as Step 8 above) output_path — where the collected logs land Run it, then check the output folder — you should see one JSON file per log type: admin_logs.json, login_logs.json, user_logs.json, calendar_logs.json, chat_logs.json, and drive_logs.json. That's Admin, Login, User, Calendar, Chat, and Drive audit logs, all pulled in one pass. Why investigators like this method Once this is set up, you can: Pull all Workspace logs in JSON Avoid UI export limits Build repeatable, defensible evidence collection Feed logs directly into SIEMs, timelines, or DFIR tooling ----------------------------------------------------------------------------------------------------------- Final thought Yes, the setup feels painful the first time. But once it’s done, you’ve essentially built a forensic-grade log pipeline for Google Workspace—and that’s incredibly powerful during incident response. After the first run, most analysts say the same thing: “Oh… that actually wasn’t that bad.” ------------------------------------------------------------Dean----------------------------------------

If you’ve ever had to investigate a Google Workspace account takeover, you already know one thing: it’s not about one log — it’s about connecting multiple logs and understanding how Google thinks. The Two Logs You Must Know When it comes to tracking user behavior (and especially account compromise), there are four core log types you’ll always come back to: Admin log events User log events (Previously it was seperated into two logs) (Login Audit Log + User Accounts Audit Log) Security Reports Think of these as different camera angles. One log alone never tells the full story — but together, they usually do. Log Retention: The 6-Month Trap By default, Google Workspace retains these logs for six months. And here’s the annoying part: You cannot extend retention inside the Admin Console There is no “keep logs longer” checkbox If you want long-term visibility (and you absolutely should), the only solution is to: Export logs to Google Cloud Logging Configure extended retention there Google Cloud allows log storage for up to 10 years, which is a lifesaver for compliance, threat hunting, and delayed investigations. Log Lag Time: Why “Too Early” Is a Real Problem One thing that trips up a lot of investigators is log availability delay. Each of logs has a different lag time before events become searchable. And that lag time should be treated as the minimum waiting period, not a guarantee. So if you search immediately after an incident and think, “This doesn’t make sense…” …it probably doesn’t — yet. Rule of thumb: Never rely on searches run shorter than the documented log lag times. Some events just arrive late. Real lag times Here's roughly what those lag times look like in practice: Admin Audit Log — a few minutes Login Audit Log — a few hours User Accounts Audit Log — tens of minutes Security Reports — 1 to 3 days (yes, this is the fourth log — easy to forget it exists, but it's official and it's slow) And this isn't theoretical — during testing, password resets have shown up in these logs before the login event that triggered them, when someone searched inside the lag window. If the timeline looks impossible, it's not broken. You just searched too early. Admin log events: Start Here for Admin Compromise The Admin Log evets is your go-to log for anything that happens inside the Google Admin Console. It tracks: Admin actions Configuration changes Policy updates Organization-wide modifications If you suspect an admin account compromise, don’t overthink it — this is the first log you check. It tells you exactly what changes were made and by which admin account. User log events: Where the Action Is The User log events is where most account takeover investigations spend their time. This log captures: Successful and failed logins Re-authentication prompts MFA changes Security challenges triggered by Google It doesn’t just tell you that someone logged in — it tells you how, why, and under what conditions. The "often empty" caveat for User Accounts events One quirk worth knowing before you panic: the User Accounts side of this log is often completely empty — sometimes zero events across a full six-month window. That's not a broken pipeline or a missed export. It's genuinely common when there hasn't been much high-risk account activity, and it's called out explicitly as "not an uncommon scenario." An empty log here is a data point, not a failure. Understanding Login Types (This Matters) Each login event includes a Login Type, which explains how the authentication happened. Some common ones you’ll see: Google Password – Standard username + password login ReAuth – Google forced the user to re-authenticate SAML – Login via SSO Exchange – OAuth or existing token-based session Unknown – Login occurred using an unidentified method (always worth a closer look ) When you’re hunting suspicious activity, “Unknown” and unusual patterns in login types are often gold. Warning Icons = Pay Attention In the User log events, some events show a warning icon. These usually indicate unusual or suspicious logins, such as: New IP addresses Unfamiliar locations Behavior Google flags as risky Instead of scrolling endlessly, a smart approach is to hunt by event type. Login Event Types Investigators Care About Here are some high-value event types you should always keep an eye on: 2-step verification disabled – Big red flag Account password change – Especially if unexpected Failed login – Useful for brute-force patterns Government-backed attack – Google explicitly flagged a known threat actor Leaked password – Password found in credential dumps Suspicious login – Unusual characteristics detected Out-of-domain email forwarding enabled – Common data exfil trick User suspended – Often triggered by Google due to abuse or compromise Event-type nuances worth knowing before you escalate A few of these deserve a second look before you read too much into them: Login challenge vs. Login verification — these look similar but mean different things. A challenge means Google already thought the sign-in was suspicious; a verification means it didn't, but asked anyway. Know which one you're looking at before you escalate. Logout events are always logged with Login Type = Google Password, even if the original session started via Exchange, ReAuth, SAML, or Unknown. Don't read that as the user switching auth methods mid-session — it's just how Google records logout. User suspended isn't one event, it's three: suspended for spam, suspended for spam relay, and suspended for suspicious activity. Which variant you get tells you a lot about what Google actually detected. Important note: Some details (like why a login failed) are not visible in the Admin Console and require pulling logs via the API. OAuth Let’s be honest — OAuth sounds way more complicated than it actually is. At its core, OAuth is just a permission slip. Instead of giving an app your username and password (which is a terrible idea), OAuth lets you say: “Hey, this app can read my emails, but nothing else.” That’s it. That’s the magic. So What Exactly Is OAuth? OAuth is an authorization mechanism — not authentication. It doesn’t prove who you are It proves what an app is allowed to do When an application wants to access your data through an API (emails, Drive files, contacts, calendar, etc.), OAuth sits in the middle and asks you for permission. If you say yes, the app gets a token. That token is like a digital key that says: “This app is allowed to access these specific things, on behalf of this user.” No password sharing. No repeated logins. Cleaner and safer. Why OAuth Exists (And Why Everyone Uses It) Imagine if every app you used asked for your Gmail password. Nightmare. OAuth solves a few big problems: You don’t have to re-authenticate every time Apps never see your actual credentials Access can be limited (scope-based) Tokens can be revoked anytime That’s why OAuth is everywhere — Google Workspace, Microsoft, GitHub, Slack, Twitter (X), basically everything modern. OAuth in Google Workspace (What Users Actually See) Inside Google Workspace, OAuth usually shows up as that familiar screen: “This app wants access to your : GmailDrive filesContacts” That list? Those are called scopes. Scopes define exactly what the app can touch. Nothing more. Once the user clicks Allow, Google generates an OAuth token, and the app can start making API calls using that token. Important point: OAuth is enabled by default in Google Workspace unless admins restrict it. Where Things Go Wrong: OAuth Abuse Here’s the problem — OAuth is secure, but humans are optimistic. Threat actors figured out something clever: “Why steal passwords when we can just ask nicely?” The Basic OAuth Attack Chain Attacker creates a malicious app Victim gets a phishing email with a link Victim clicks → sees a legit Google OAuth screen Victim clicks Allow Attacker now has access — no password needed No malware. No credential theft. No MFA bypass required. Just consent. The two real OAuth attacks (replacing the generic chain with named case studies) This isn't a hypothetical either — it's happened at scale, twice, and both cases are worth knowing by name: In May 2017, a fake app calling itself "Google Docs" (not the real thing) spread through Gmail like a worm — a contact would send you what looked like a shared doc, you'd click it, authorize "Google Docs" for email and contacts access, and then your own contacts would get the exact same email from you. Google eventually banned the app and mass-revoked tokens, but not before it spread fast. Between 2015 and 2016, the threat actor group Fancy Bear (APT28) ran a more targeted version against political parties: a phishing email claiming a security system had flagged suspicious sign-ins, urging the target to install "Google Defender" for protection. The "app" requested OAuth access to the victim's Gmail and their entire Drive — full read access to email and shared documents, no password required. Both attacks prove the same point: OAuth abuse doesn't need a stolen credential, a malware payload, or an MFA bypass. It just needs one click on Allow. Why Threat Actors Love OAuth OAuth attacks are attractive because: No credentials to steal MFA doesn’t stop it Looks completely legitimate Uses official Google infrastructure And the scariest part? OAuth does NOT give attackers more access than the user already has — but that’s usually more than enough. Detecting OAuth Abuse in Google Workspace Google Workspace actually gives us solid visibility here. OAuth Log events These logs show: Which user authorized which app Application ID Scopes granted API activity performed using the token Technically, this log records three distinct types of entries, and it helps to know which is which: Activity (an API call the app made, using its token) Authorize (the moment a user granted the app access), Revoke (access being pulled, whether by the user, an admin, or a password change) One limitation worth flagging before you go looking for it: Activity entries in this log are only recorded on Enterprise and Education editions — on other editions, you may simply not have this visibility at all, and that's a licensing gap, not a missing export. If you pull these logs via API, you get even more gold: Source of the request Which Workspace service was accessed How much data was returned Client type and product bucket This is huge for investigations and retroactive analysis. Killing the Access: Revoking OAuth Tokens A few important things defenders should know: Changing a user’s password revokes OAuth tokens IMAP tokens can take up to an hour to expire Admins can: Review all third-party apps See who authorized them Block apps org-wide In the Admin Console, you can quickly identify sketchy apps by: Unusual scopes Non-verified apps Excessive permissions Block once — and it impacts the whole org. The Big Takeaway OAuth isn’t insecure. Blind trust is. OAuth attacks succeed because: Users trust the Google consent screen App names look legitimate No passwords are involved (so alarms don’t go off) Defenders need to: Monitor Token Audit Logs Restrict third-party apps Educate users that “Allow” is a powerful action Because sometimes, clicking Allow is worse than typing your password. ------------------------------------------------------------------------------------------------------------- Final Thoughts If there’s one takeaway here, it’s this: Understand: What each log shows When data becomes available Which events actually matter Once you get comfortable with these logs, Google Workspace investigations stop feeling messy — and start feeling methodical. ------------------------------------------Dean--------------------------------------------------------------