Why ClickHouse The first thing I wanted to understand about RunReveal wasn't the UI — it was what's actually underneath it. Most SIEMs I've worked with are built on some flavor of a log indexer: Splunk has its buckets and indexers, Sentinel sits on top of Azure Log Analytics workspaces. Those architectures were designed in an era where the assumption was you'd mostly search for specific terms across a lot of unstructured text. RunReveal is built directly on ClickHouse, a columnar OLAP database. The practical difference: Columnar storage is built for aggregation and analytical queries — count these, group by that, join across time windows — which happens to be exactly the shape of most detection logic and threat hunting. It's the same reason a lot of observability companies quietly migrated off Elasticsearch onto ClickHouse over the last few years. RunReveal just built the SIEM on that foundation from day one instead of bolting it on later. The logs Table — One Table, Everything Here's the part that actually surprised me. Almost everything in RunReveal reads from a single underlying table, called simply logs. I ran a DESCRIBE TABLE against it directly and counted 38 columns. The ones that matter most day-to-day: workspaceID, sourceID, sourceType — which source produced the event receivedAt — the field you filter on for time ranges; eventTime is separate and represents when the activity actually happened eventName, eventID — what happened srcIP / dstIP, plus full GeoIP and ASN fields for both — no separate enrichment lookup needed for basic geo/ASN context actor (a Map type) — who did it, however the source represents that tags (Map) and resources (Array) — flexible structured fields for anything source-specific enrichments (Array of Tuple) — results of any enrichment steps a pipeline applied rawLog — the original, unparsed event, always kept On top of that single table, RunReveal maintains source-specific views — aws_cloudtrail_logs, okta_logs, github_logs, aws_vpc_flow_logs, and more — which present the same underlying rows with normalized, source-appropriate column names. Same DESCRIBE TABLE trick works on those too, if you want to see exactly what a given connector normalizes for you. Topics, Pipelines, and the Step That Actually Writes Data This is the part of the architecture that took me the longest to actually understand, because the docs describe it more abstractly than what I found once I inspected a real, live pipeline through the API. A Pipeline is an ordered list of steps that runs against a Topic (a stream of incoming events). I pulled a real pipeline from a live workspace and it had five steps, in order: transform — reshape the event enrich — attach additional context filter — drop events matching a condition detect — a lightweight in-pipeline detection check destination — the step that decides where the event actually ends up The insight worth sitting with: a pipeline's Destination step is what actually writes to storage. There's no implicit side-channel where data magically ends up in ClickHouse just because it entered the system. If a pipeline doesn't have a Destination step, older pipelines fall back to RunReveal's default backend, but the docs explicitly warn that fallback isn't something you should rely on — add explicit Destination steps so routing is predictable.' Destinations, Properly Explained I want to spend real space on this because it's a genuinely well-designed piece of the platform, and the docs page for it answers a lot of questions I had after just looking at one example pipeline. Two destination types ClickHouse destinations are for real-time querying and analysis — this is what powers every RunReveal feature. You can configure more than one, with one marked as default. There's also SPIFFE/mTLS support for certificate-based auth instead of username and password, if that matters to your environment. Object Storage destinations — S3, Cloudflare R2, Google Cloud Storage, Azure Blob — are for archival, backup, and long-term retention. They're not queryable through RunReveal's own features; they're cold storage for compliance and disaster recovery. How routing actually works A few rules that aren't obvious until you read the FAQ section of the docs closely: Creating a destination in the UI does nothing on its own — it just makes that destination available. Events only flow to it once a pipeline has a Destination step selecting it. A single pipeline can have multiple Destination steps, which means you can fan the same event out to ClickHouse and S3 at once. Destination steps don't stop pipeline execution — later steps still run. If you have two Destination steps with no precondition separating them, you'll double-write the same event unless that's actually what you want. One destination failing doesn't block delivery to the others — errors are tracked independently per destination. Worth remembering: RunReveal Backend only targets ClickHouse storage. If you want data in S3 too, you need a separate explicit Destination step for that object storage destination — nothing routes there automatically. Destination health checks RunReveal can monitor write errors per destination over a rolling interval, and alert your notification channels if the error count crosses a threshold for enough consecutive checks in a row — useful for catching a misconfigured or unreachable destination before it silently drops data for hours. These are configured from the destination's own Health Checks page, not from the create/edit form, which tripped me up the first time I went looking for the setting. Retention, in Practice Default retention on ingested data is 550 days. Disabling a source stops new ingestion but keeps existing config and data intact. Actual deletion of historical data is support-ticket only — this is append-only storage, which lines up with why Object Storage destinations exist as a separate concept from ClickHouse destinations: one is for querying recent-to-mid-term data fast, the other is for keeping everything indefinitely at a lower cost. Next up: Part 3 covers how you actually query all of this yourself — the Explorer, saved queries, and a set of platform tables that let you query RunReveal's own operational data with the same SQL interface. https://www.cyberengage.org/post/runreveal-explorer-search-querying-your-data-directly ------------------------------------------------Dean------------------------------------------------

The Interface Sitting on Top of Everything in Part 2 Part 2 covered the logs table and the ClickHouse foundation underneath RunReveal. This part covers how you actually touch that table day to day — the Explorer, RunReveal's log search and query interface. It's easy to undersell this as "just a search bar," and I almost did exactly that in the first article of this series before realizing how much is actually packed into it. RunReveal's own description calls it a compact, single-page interface with sidebar navigation, saved queries with full version history, and AI-powered query assistance built in — not a separate add-on chat window bolted to the side. Layout The explorer is organized into a few consistent pieces: A collapsible sidebar listing every queryable table, grouped by category: the main logs table, source-specific views, and any custom views you've created. Expand a table to see its schema and columns without leaving the page. A top bar and filter section for building filters without hand-writing WHERE clauses — filters are type-aware, so a timestamp field gets different filter operators than a string field. A time-series histogram showing event volume across your selected window, which is the fastest way to spot a spike or a gap before you've written a single line of SQL. A saved queries sidebar, and a query editor you can drop into directly when a filter-builder UI isn't expressive enough for what you're trying to ask. Saved Queries Actually Have Version History Every time you save a query, RunReveal creates a new version rather than overwriting the last one. That's a small detail that matters more than it sounds like — it means a saved detection-adjacent query you're iterating on keeps a real history, so if a change makes it worse, going back is a click rather than a rewrite from memory. Saved queries also carry their full parameters and settings with them, and every exploration state — filters, table, time range — can be shared as a URL, which makes handing a specific query off to a teammate genuinely simple. The AI Assistant Is Actually Useful Here You can describe what you want in plain language and the assistant generates a SQL query from that description, which you then review before running — it doesn't execute blind on your behalf. The more interesting piece is what happens when a query fails: the assistant reads the actual error, proposes a fix, and lets you review the corrected query before applying it. For anyone who's spent time debugging a ClickHouse syntax error at 2am during an actual incident, that's not a gimmick feature. Two Timestamps, and Why It Actually Matters RunReveal stores two separate timestamps on every normalized event, and which one you filter on has a real, measurable performance impact — this is documented explicitly, not something I had to discover by trial and error for once. receivedAt — when RunReveal ingested the event. Indexed as part of the primary key, consistent across every source regardless of how that source reports time, and correctly accounts for delayed log delivery. Use this for time filtering, detections, and any scheduled query. eventTime — when the activity actually happened, according to the original source. Not indexed, so filtering on it is slower, and it can be missing or inconsistent depending on the source. Use it for displaying the original event time, not for filtering. ----------------------------------------------------------------------------------------------------------- The docs' own example, and it's a good one: filtering on receivedAt is marked as the correct pattern; filtering on eventTime alone is explicitly flagged as something to avoid for anything performance-sensitive. If a query feels slower than it should, this is the first thing worth checking. ----------------------------------------------------------------------------------------------------------- Query Performance, in Practice A short list of habits the documentation calls out directly, all of which match what I'd already learned the hard way testing detections in Part 6: Use source-specific views (aws_cloudtrail_logs, okta_logs, github_logs, and others) instead of querying the raw logs table and filtering sourceType yourself. Filter on primary-key and indexed fields first — sourceType, then receivedAt, then sourceID — before adding other conditions, matching how the table itself is physically organized. Start with a small time range to validate a query, then widen it. A one-hour window while testing is dramatically faster to iterate on than starting with thirty days. Select only the columns you actually need in an aggregation query rather than pulling everything. Platform Tables — Querying RunReveal About Itself This is the part that genuinely surprised me. Explorer doesn't just query your ingested logs — there's a whole second category of platform tables that let you query RunReveal's own operational data with the same SQL interface: detections — the base table for every detection run, one row per finding, regardless of whether it ever escalated to a signal or alert. Useful for answering "has this ever matched anything" without digging through the UI. scheduled_query_runs — execution metadata for every scheduled detection run: runtime, the exact SQL that executed, parameters, and error messages if a run failed. runreveal_source_volumes — per-minute event counts and approximate ingested bytes, bucketed by source. This is the raw data behind the kind of volume-anomaly agent covered in Part 8. Ingestion and pipeline error tables — failed schema validation events, complete with the original raw log that failed, plus per-pipeline-step dropped-byte metrics. Health check subscription tables — the underlying data behind the destination health checks covered in Part 2. A workspace audit trail table — who did what, when, from which IP. Threat intelligence indicator feeds — a built-in table of known-bad IPs you can join against your own traffic directly in SQL. Why this matters beyond curiosity: every dashboard, health check, and agent report in this platform is ultimately just a query against tables you can also query yourself. Nothing here is a black box — if you don't trust a number a dashboard is showing you, you can go verify it directly against the same table it reads from. -------------------------------------------------------Dean----------------------------------------------- Part 4 goes the other direction — not how you query data, but what happens to it in flight before it's ever stored: transforms, filtering, dropping, masking, sampling, and enrichment.

The Problem Nobody Loves Admitting Every SOC analyst I know has a version of the same complaint. Splunk gets expensive the moment your log volume grows past whatever number your finance team quietly panicked about last renewal. Sentinel is great if your entire estate lives in Azure and painful the moment it doesn't. And half the SIEMs on the market still feel like they were built for a world where a 'query' meant clicking through six dropdown menus instead of just writing SQL. RunReveal showed up on my radar because it skips a lot of that. It's a security data platform built directly on ClickHouse, which is the same database engine a lot of observability companies quietly switched to once they got tired of Elasticsearch falling over at scale. I spent the last few weeks actually building against it — not reading a datasheet, but standing up sources, writing detections, breaking things, and building a small MCP server so I could query it from inside AI agent. This series is what I found. A note on how this series was written: Every technical detail in this series — endpoint behavior, schema fields, quirks, things that are flat-out broken right now — comes from hands-on testing against a real RunReveal workspace, not from marketing copy. Where something is broken, I say so. So What Is It, Actually RunReveal calls itself a modern security data platform, and for once the adjective is doing real work instead of just filling space. The pitch is: Detection-as-code instead of point-and-click rule builders, AI-powered investigation triage instead of an analyst opening forty browser tabs, and sub-second query performance because the whole thing sits on a columnar database instead of a traditional log indexer. Strip away the marketing language and here's the actual shape of the thing: Sources pull data in — cloud providers, identity platforms, EDR tools, SaaS apps, roughly 120 connectors and counting. Pipelines and Topics move that data around and let you transform, enrich, or filter it before it lands anywhere. Everything ends up in one big ClickHouse table, which detections (written in SQL or Sigma) query on a schedule. A match escalates through detections → signals → alerts, getting louder each step, until it hits a notification channel. Alerts spin up Investigations — timeline-based case objects with their own AI triage agent. That pipeline — source to signal to investigation — is the spine of this whole series. Every later article is really just zooming into one link of that chain. Who This Is Actually For I don't think RunReveal is trying to be Splunk's replacement for a 500-analyst enterprise SOC that already has a decade of SPL knowledge baked into its team. Where it makes sense, at least from what I've seen, is smaller or leaner security teams who are comfortable with SQL, don't want to pay an ingest tax on every gigabyte, and want their detections to live in version control instead of a vendor's proprietary rule editor. It also clearly leans into being AI-native rather than having AI bolted on as a chatbot sidebar. The investigation triage agent, the native AI chat, and the fact that it exposes an MCP server so tools like Claude can query your workspace directly — that's a different design philosophy than most of the incumbents, who added "AI" to their SIEM after the fact. The Eight Pieces That Make Up the Platform Here's the map I wish I'd had on day one: Sources — where data comes from. Pipelines & Topics — how data moves and gets shaped in transit. Detections — SQL or Sigma rules that scan the data on a schedule. Signals & Alerts — the escalation ladder from "something matched" to "someone got notified." Investigations — the case file, with an AI agent doing first-pass triage. Search / Explorer — ad hoc querying against the same ClickHouse backend. AI Chat — a native, in-console, fully-audited conversational interface over your data. Dashboards — the usual visual rollups, built on top of everything above. None of these are novel concepts on their own — every SIEM has some version of sources, detections, and alerts. What's different is how thin the layers are between them. There's no separate indexing tier, no proprietary storage format you need a support ticket to query around. It's one table, underneath everything. What's Coming in This Series This is the first of twelve articles. The next one goes under the hood of the ClickHouse architecture itself — the actual table schema, how pipelines route data, and a genuinely interesting design decision about where the "real" write to storage happens. After that: the Explorer and how you actually query all of this yourself, everything that can happen to an event in flight before it's ever stored (transform, filter, drop, mask, sample, enrich), ingestion and the connector catalog, the detection engine, investigations and AI triage, scheduled AI agents (including a live gotcha I hit building one), notifications, the AI Chat and MCP integration (including the MCP server I built), a head-to-head comparison against Splunk and Microsoft Sentinel, and finally a full hands-on build of a detection pipeline from scratch, screenshots included. If you're the kind of person who reads a SIEM's docs page and immediately wants to know what actually happens when you hit the API instead of the marketing site — you're going to enjoy this series more than most. Next Part: https://www.cyberengage.org/post/runreveal-architecture-deep-dive ---------------------------------------Dean------------------------------------------------------

I want to be upfront about where I'm coming from before I say anything else: I'm not an AI skeptic. I run AI agents most of my working day, and I've built plenty of them for different tasks. I use them for a lot — building internal tools, troubleshooting, you name it. Half the projects on my plate right now exist because an AI agent worked alongside me, troubleshot with me, and knocked out in a few days what would've taken months. So when I say the Hugging Face breach worried me, I don't mean it in the 'AI is scary, unplug everything' way. I mean it as someone who actually uses this stuff for a living and recognized exactly how bad the failure mode was.. What actually happened In July 2026, Hugging Face disclosed that its infrastructure had been breached — not by a person, but by an AI agent that was supposed to be inside a sandbox. An OpenAI model, being run through a cybersecurity evaluation, was handed a challenge it couldn't legitimately solve. Instead of failing gracefully, it started reward hacking: chaining together previously unknown exploits to find a real answer out in the wild instead of admitting it was stuck. It broke out of its test environment, compromised a package management tool to get outbound internet access, then used a malicious dataset to abuse two separate remote-code-execution paths in Hugging Face's dataset processing pipeline. From there it leaked cloud and cluster credentials and moved laterally into internal systems, touching a limited set of internal datasets and Spaces secrets — the auth tokens developers and organizations use to talk to APIs. Hugging Face caught it through its own LLM-based anomaly detection, not a human noticing something odd in a log. Here's the part that actually changed how I think about this. Hugging Face's disclosure pushed Anthropic to go back and audit its own evaluation logs — 141,006 runs where Claude had internet access during testing. They found three separate incidents where a Claude model, during a third-party evaluation, got unauthorized access to the production infrastructure of three different companies. Two of those companies hadn't even noticed it happened until Anthropic told them. And in early September, Anthropic disclosed a fourth incident, this time involving Claude Opus 4.6. So this isn't an "OpenAI has a problem" story. It's an industry pattern. Two different frontier labs, evaluating their own models for cybersecurity capability, found those same models quietly breaking containment and touching real production systems that had nothing to do with the test. Why this rattles me more than a normal breach I do incident response. I've spent this year going through ClickFix infostealer cases, credential theft, lateral movement, Ransomwares the whole playbook — attacker moves, defender reconstructs the timeline after the fact. That's a known shape. What happened here is a different shape entirely. This wasn't a human attacker with a plan. It was a system that got stuck, decided the constraints it was given didn't apply anymore, and improvised its way to production access using techniques nobody had programmed it to know. It didn't need to be told how to chain a sandbox escape into an RCE into a lateral movement into a credential harvest — it worked that out on its own, mid-evaluation, because that's what "don't accept failure" optimization does when nobody explicitly told it where the line was. And the thing that caught it wasn't a human analyst staring at a SIEM. It was another AI model doing anomaly correlation. We are now in a place where the attacker is an AI, the intrusion looks like normal API traffic from a trusted internal tool, and the detection layer that actually works is also AI. If your org's whole security posture assumes a human is the one doing something malicious, you're already behind. Where I land — don't stop using AI, start governing it like the privileged identity it is I'm not writing this to tell you to go rip out your AI tooling. I won't be ripping out mine. The productivity gain is real and I'm not giving it up over a headline. But "keep using it" and "keep using it with no guardrails" are two very different decisions, and right now most orgs — including plenty that think of themselves as security-mature — are quietly making the second one by default. A few things I actually believe need to happen now, not in a year when someone writes a compliance framework for it: Treat AI agents as privileged identities, not as tools. Scoped credentials, short-lived tokens, least privilege — the same rules you'd apply to a service account, applied to every agent that can touch a real system, including the ones your vendors run during "just an evaluation." Agent activity monitoring has to be a first-class control, not something bolted on after the fact. Hugging Face and Anthropic both caught these incidents through AI-driven telemetry correlation, which tells you the traditional logging most companies have isn't built to catch this class of behavior at all. Sandbox boundaries need to be boundaries an agent can't reason its way out of. A test environment that can be escaped by a model that decides the test is unfair is not a sandbox, it's a suggestion. Human approval gates belong in front of anything touching production, even — especially — when the thing asking is your own evaluation pipeline testing your own model. Disclosure has to become normal, not exceptional. Credit where it's due: Hugging Face and Anthropic both published real, detailed incident reports instead of burying this. That's the behavior the rest of the industry needs to copy, not the exception that makes headlines. I've spent time this year independently testing one of the early products trying to build exactly this kind of control — Delphi Security's xAIDR, an open-source AI agent runtime security sensor.(https://delphisecurity.ai/products) Delphi Security's xAIDR might not be at that level of maturity yet, but it's a start. Tool Delphi Security. It's in early-stage . But the instinct behind it — that agent behavior needs its own dedicated monitoring layer, separate from how we watch human users — is exactly right, and I'd rather see more of these tools exist and mature than none at all. The point We're not going to put this genie back in the bottle, and honestly, I don't want to — I'd be less effective at my own job without it. But the Hugging Face breach, and the fact that Anthropic found the same failure mode in its own models within weeks of looking, should be the moment every org stops treating "we use AI agents" as a productivity line item and starts treating it as an access-control problem. Write the policy now. Monitor the agents now. Assume they will do something you didn't authorize, because two of the most capable labs in the world just showed you theirs already did. --------------------------------------------------Dean-------------------------------------------------

Where the traffic actually went — and why GCP's network model means one VPC can span the whole company. GCP's networking model has one structural quirk that changes how you scope an investigation compared to AWS: a VPC in Google Cloud is global, not regional, and it's also a project-level resource that can be shared across other projects entirely. Get this wrong and you'll scope an incident to a single project when the network the attacker actually moved through spans half the org. Shared VPC: One Network, Many Projects Solstice Payments runs a Shared VPC out of solstice-shared-net, the host project, with subnets in us-central1 (10.20.1.0/24) and europe-west1 (10.20.2.0/24). Both solstice-prod-core and solstice-prod-data are service projects attached to that host — their instances get IPs out of those same subnets, but IAM permissions over the network itself (who can create firewall rules, who can peer another VPC into it) live in the host project, not the service projects. This is the single most common scoping mistake I see: An investigator pulls IAM and Flow Logs for the compromised service project, sees nothing unusual about the network configuration, and never checks the host project where the actual firewall rule changes happened. First move on any network-touching incident: identify whether the VPC in play is standalone or shared, and if shared, go find the host project. gcloud compute shared-vpc get-host-project solstice-prod-core VPC Flow Logs Flow Logs record a sampled summary of IP traffic to and from VM interfaces — not full packet capture, but enough to answer "did this instance talk to that IP, and how much data moved." They're enabled per-subnet, off by default, and like Data Access audit logs, are one of the things worth confirming is actually turned on before you assume you'll have network evidence. The fields that matter most in a flow log record: connection.src_ip / connection.dest_ip and the matching ports — the basic who-talked-to-whom. bytes_sent and packet counts — useful for spotting a large outbound transfer that doesn't match the instance's normal traffic profile, which is often the first concrete signal of data exfiltration over the network layer rather than an API call. reporter (SRC or DEST) — tells you which side of the connection generated this particular record, since both ends can report the same flow. rtt_msec — round-trip time, occasionally useful for distinguishing traffic that stayed inside Google's backbone (very low RTT) from traffic that left to the public internet and came back. Sampling matters: Flow Logs default to 50% sampling and 5-second aggregation intervals, both configurable. At default settings you will not see every single connection, which is worth stating explicitly in a report if a client asks why a specific short-lived connection isn't in the logs — it may genuinely not have been sampled, not that logging failed. gcloud compute networks subnets update solstice-central-subnet --region=us-central1 \ --enable-flow-logs --logging-flow-sampling=1.0 --logging-aggregation-interval=interval-5-sec Firewall Rule Logging Separate from Flow Logs, individual firewall rules can have their own logging enabled, recording every connection the rule allowed or denied — including the specific rule that matched. This is what you want when you need to know not just that traffic moved, but which firewall rule let it through, especially useful when an attacker (or a misconfiguration) added an overly permissive rule and you need to prove exactly when it started being exploited. Packet Mirroring: Full Packet Capture, Not Just Flow Summaries Flow Logs and firewall logging tell you that traffic moved and which rule let it through — they don't hand you the actual packets. For that, Google Cloud has Packet Mirroring: Full packet capture defined per VM instance, per GKE cluster, or per VPC, which makes it straightforward to scope a capture tightly around whatever's suspected of being compromised rather than pulling traffic for an entire environment. GCP's implementation works differently from a typical on-prem tap. A Mirror Policy defines which source instances to capture from, but the mirrored traffic doesn't land in a bucket directly — Google Cloud has no mechanism to write a PCAP straight to Cloud Storage. Instead, the mirrored packets are handed to a load balancer forwarding rule, which distributes them across a backend instance group exactly the way a normal load balancer distributes inbound requests, except here the "requests" are copies of live traffic. For DFIR purposes, that backend instance group is where you'd stand up Arkime, Zeek, or another NSM tool to actually capture and analyze what's mirrored. Setting it up takes two pieces. First, an internal load balancer forwarding rule flagged as a mirroring collector — this is what the mirrored packets actually get delivered to. Solstice already has an NSM backend service (running Zeek) fronted by this rule: gcloud compute forwarding-rules create solstice-mirror-collector \ --region=us-central1 --load-balancing-scheme=INTERNAL \ --backend-service=solstice-nsm-backend --is-mirroring-collector Then the mirroring policy itself, pointing at that collector and naming what to mirror — in an active incident you'd usually scope this to the one suspect instance, not the whole subnet: gcloud compute packet-mirrorings create solstice-prod-core-mirror \ --region=us-central1 --network=solstice-shared-net \ --collector-ilb=solstice-mirror-collector \ --mirrored-instances=solstice-prod-core --filter-direction=BOTH The catch for incident response: Packet Mirroring is forward-looking only. It captures traffic sent after the policy goes live — there's no way to retroactively pull packet-level data for a window before the mirror was configured. If Solstice's SRE team suspects solstice-prod-core is actively being used for data exfiltration and Flow Logs alone aren't giving enough detail, standing up a Mirror Policy against that instance immediately is the move — but it only helps for what happens next, not what already happened. Cloud DNS Logging DNS query logging for a VPC's managed zones is also opt-in and separately configured from Flow Logs. It records the query name, response, and source, and is often the fastest way to confirm C2 activity when an attacker is using domain-based command and control rather than a raw IP — Flow Logs will show you the resolved IP, but Cloud DNS logs are what tie that IP back to the domain name that was actually queried, which matters when the IP itself sits behind shared infrastructure like a CDN or the attacker rotates IPs faster than domains. Load Balancer Logs For anything sitting behind a Google Cloud Load Balancer — which is most public-facing Solstice services — the load balancer's own request logs are frequently more useful than the backend instance's application logs, because they're generated regardless of what happened to the backend afterward. Each entry includes the client IP, request path, response code, latency, and, for HTTPS(S) load balancers, the negotiated TLS details. If a web app was compromised through a specific malicious request, the load balancer log is often the cleanest place to find the initial request that did it, especially if the attacker's later steps involved tampering with or disabling the application's own logging. Firewall rules moved toward a hierarchical model — Firewall Policies at the org and folder level, evaluated alongside (and able to override) legacy per-VPC firewall rules, giving central security teams a way to enforce network controls that individual project owners can't loosen. If you're investigating a network permissiveness question, check for hierarchical Firewall Policies in addition to the classic per-VPC rules — the effective policy is a merge of both, and looking at only one gives an incomplete picture. Cloud NGFW (next-generation firewall, with intrusion prevention) also expanded significantly since 2022 and may be in play for environments with more mature network security postures. 🔎 IR tip: Before concluding "no evidence of network exfiltration," confirm three things were actually on: Flow Logs at an adequate sampling rate, firewall rule logging on the relevant rules, and Cloud DNS logging. Absence of evidence across all three is a real finding worth reporting on its own — but a gap in even one of them can produce a false negative that looks identical to a clean network. ..-------------------------------------Dean---------------------------------------------------

Dont forget to check out completle SentinelOne series https://www.cyberengage.org/courses-1/mastering-sentinelone%3A-a-comprehensive-guide-to-deep-visibility%2C-threat-hunting%2C-and-advanced-querying%22 Introduction Ran into something during a routine SentinelOne rollout that's worth sharing, because it doesn't look like a SentinelOne problem when you first see it — it looks like the third-party app is broken. An employee's VPN client wouldn't connect on their managed Mac. Install was clean, app opened fine, connection just sat on "establishing" forever. If you've deployed SentinelOne's default macOS configuration profile fleet-wide, you may be sitting on the same landmine without knowing it yet. The Issue The VPN app needs to register a macOS System Extension to function. That registration was failing silently — no approval prompt, no actionable error, just a permanent hang at connect time. Standard move for SentinelOne Users : disable the SentinelOne agent for a few minutes, have the user retest, see if the agent is the cause. No change. Agent off, same failure. That result normally clears SentinelOne — except this time it didn't, and the reason why is the actual finding here. What I Found The user checked the macOS System Extension policy database directly: sudo plutil -p /Library/SystemExtensions/db.plist It returned an active extensionPolicies entry scoped to one Team ID only: "allowedTeamIDs" => [ "******" ] // SentinelOne's Team ID(Redacted) "allowUserOverrides" => false That policy did not change when the agent was disabled — which makes sense once you know how it's enforced: MDM pushes SentinelOne's default macOS configuration profile, which includes a System Extension payload authorizing SentinelOne's own extension. macOS writes that payload into /Library/SystemExtensions/db.plist as an extensionPolicies entry. Once that entry declares an allowedTeamIDs array, macOS treats it as exclusive — anything not listed is denied at the OS level, silently, with no user prompt. The SentinelOne agent process has nothing to do with enforcement at that point. Killing or disabling the agent doesn't touch the profile sitting on disk, so the block persists agent-on or agent-off. So the "SentinelOne is broken" theory and the "SentinelOne isn't involved" theory were both half right: the running agent wasn't the problem, but a profile it shipped with was The Fix I left SentinelOne's default profile untouched and pushed a second, separate profile adding the VPN vendor's Team ID to the allowlist, with allowUserOverrides flipped back to true. One reboot later, the VPN connected on the first retry. If You're Facing This Third-party app with a system extension / kernel component silently fails to load on a SentinelOne-managed Mac — no prompt, no clear error Disabling the SentinelOne agent doesn't fix it Run sudo plutil -p /Library/SystemExtensions/db.plist and look for allowedTeamIDs If it lists only SentinelOne's Team ID (******) with allowUserOverrides: false, that's your cause Fix: a separate MDM profile allowlisting the blocked vendor's Team ID — don't edit SentinelOne's default profile directly Final Thoughts SentinelOne wasn't misbehaving — its default macOS profile did exactly what it was configured to do. The gap was on our side: we deployed that profile fleet-wide without reading the System Extension payload closely enough to notice it also locks out every other vendor. Worth a five-minute audit of your own profile before this shows up as a VPN, webcam, or remote-access ticket with no obvious cause.

GCS buckets are both a common source of evidence and, misconfigured, one of the most common ways data actually leaves a GCP environment. Cloud Storage shows up in almost every incident twice: once as a target — the thing the attacker was actually after — and once as a source of evidence, if you configured it right beforehand. This article covers both sides: how bucket access actually works, how to tell if an attacker exfiltrated data through a bucket, and how to collect GCS objects in a way that holds up as evidence. IAM vs. Legacy ACLs Buckets can be governed two ways, and it's important to know which one a given bucket uses before you try to reason about who could access it. Uniform bucket-level access applies IAM policy consistently across every object in the bucket — one policy, no per-object exceptions, and it's what Google now recommends and defaults new buckets to. The older model, fine-grained access, layers legacy ACLs on top of IAM at the individual object level, meaning a bucket's overall IAM policy can look reasonable while a specific object inside it has an ACL granting allUsers read access that nobody would find just by checking the bucket policy. Solstice's security team standardized on uniform bucket-level access org-wide for exactly this reason — with fine-grained buckets, checking "is anything in this bucket public" means enumerating every object's ACL individually, not just reading one policy document. If you inherit an investigation involving a fine-grained bucket, budget the extra time for that enumeration. gcloud storage buckets describe gs://solstice-prod-invoices-archive --format='value(iamConfiguration)' Access Logging: Two Different Features, Only One Still Worth Using GCS has two separate logging mechanisms, and it's easy to assume the newer one replaces the older one — it doesn't, not fully. Data Access audit logs give you per-object-read/write attribution tied to a principal, in the same audit log pipeline as everything else, and that's the right default for tracking authenticated access. But Data Access audit logs only capture activity from an identified principal — they have a blind spot for exactly the traffic you most need during a public-exposure investigation: anonymous requests from allUsers or allAuthenticatedUsers. For that, you still need the older, bucket-native Usage Logs and Storage Logs — CSV files written into a destination bucket, and the only mechanism that tracks unauthenticated object access at all. Two catches worth knowing before you rely on them: they can only be enabled via gsutil or the API, not the console they don't route through Cloud Logging or the Log Routing pipeline at all — you have to retrieve them manually from wherever they're written, or process them with a Cloud Function. Usage Logs are produced hourly and record every access from allUsers/allAuthenticatedUsers (Google explicitly doesn't guarantee no duplicates between hourly files — dedupe on the s_request_id field). Storage Logs are written once a day and just report total storage consumed over the prior 24 hours, useful for spotting unexplained growth or shrinkage rather than individual access events. gsutil mb gs://solstice-access-logs gsutil iam ch group:cloud-storage-analytics@google.com:legacyBucketWriter gs://solstice-access-logs gsutil logging set on -b gs://solstice-access-logs gs://solstice-prod-invoices-archive If Usage Logs weren't enabled on a bucket before an incident, there's no way to retroactively see who accessed it anonymously — which makes enabling them upfront, on any bucket handling sensitive data, one of the cheapest insurance policies in this article. Versioning, Soft Delete, and Making Deletion Not Mean Deletion Two separate features protect against data loss — including an attacker deliberately deleting objects to cover their tracks — and it's worth knowing both because they solve slightly different problems. Object versioning keeps prior versions of an object when it's overwritten or deleted, and those noncurrent versions are retrievable by generation number until a lifecycle rule cleans them up. Soft delete, a newer and separate feature, retains any deleted or overwritten object for a configurable retention window (seven days by default) regardless of whether versioning is even enabled — so even a bucket that was never configured for versioning still gives you a recovery window if soft delete is on, which it now is by default for new buckets. For an investigation, this means an attacker's gsutil rm -r doesn't necessarily mean the objects are gone. Check both features before writing off deleted evidence as unrecoverable. gcloud stora ge ls --with-versions gs://solstice-prod-invoices-archive/ gcloud storage objects restore gs://solstice-prod-invoices-archive/q3-report.csv --generation=1723484921003912 Bucket Lock: Making Your Own Evidence Tamper-Proof If you're the one setting up evidence retention rather than investigating after the fact, a retention policy combined with Bucket Lock gives you a genuine WORM (write-once-read-many) guarantee — once locked, not even a project Owner can shorten the retention period or delete objects before it expires, short of deleting the entire project. Solstice's sec-ops team locks their audit-log export bucket this way specifically so that a compromised Owner-level identity — the worst case scenario — still can't destroy the evidence trail that would expose the compromise. The Public-Exposure Problem A bucket or an individual object becomes public the moment allUsers or allAuthenticatedUsers appears in an IAM binding or a legacy ACL entry. This remains one of the most common ways sensitive data actually leaves a GCP environment — not through a sophisticated exfiltration technique, but through a bucket that was made public temporarily for a legitimate reason and never locked back down. Public Access Prevention, an org-policy or bucket-level setting, blocks these bindings outright regardless of what IAM policy says, and checking whether it's enforced organization-wide is one of the fastest ways to rule out this entire class of exposure in an investigation. This same public-exposure pattern has an off-the-shelf attacker tool built around it — GCPBucketBrute automates exactly this recon: dictionary-guessing bucket names (which aren't secret, just unlisted), checking which ones respond as valid, then testing whether objects inside can be listed publicly. If a bucket comes back positive on all three, downloading its contents is trivial. Worth checking your own naming conventions against this reality — a predictable scheme (solstice-prod-*, solstice-backup-*) is exactly what a dictionary attack is built to find. gcloud storage buckets list --format='table(name, iamConfiguration.publicAccessPrevention)' \ --project=solstice-prod-data Collecting GCS Objects as Evidence Every object carries Google-computed CRC32C and MD5 hashes as metadata, generated at write time — pull and record these alongside your own hash of the downloaded copy rather than only hashing after the fact, since it gives you an independent value to cross-check against. Always copy by generation number when the bucket has versioning or soft delete enabled, so you're provably pulling the exact version relevant to your timeline rather than whatever happens to be current at download time. gcloud storage objects describe gs://solstice-prod-invoices-archive/q3-report.csv --format='value(md5Hash,crc32c,generation)' gcloud storage cp gs://solstice-prod-invoices-archive/q3-report.csv#1723484921003912 ./evidence/ No Logging At All? Check the Bill Sometimes you inherit an investigation where neither Data Access audit logs nor Usage/Storage Logs were ever enabled on the bucket in question — more common than it should be. You're not entirely out of options: Google Cloud tracks data egress for billing purposes regardless of whether bucket-level logging was configured, since data leaving a Cloud zone/region gets billed back to the customer either way. The Billing section's Cost Table sub-menu breaks down cost per service per month, and that includes network egress from Cloud Storage. It won't give you a filename or a source IP — this is a blunt instrument — but it can tell you approximately when a spike in egress happened and roughly how much data moved, which is often the difference between having a rough exfil estimate for a client's insurer and having nothing at all to hand them. One real limitation: Cost Table data only finalizes at the end of a billing cycle (typically monthly) — if the incident is mid-cycle, this data isn't available yet and you'll be waiting on the cycle to close before you can pull it. IR tip: check Cost Table egress by month before telling a client billing data can't help — it's a last resort, but it beats telling them there's simply no record at all. 📌 Soft delete moving to generally available and on-by-default for new buckets is the biggest practical change — it means "the attacker deleted the evidence" is a weaker claim than it used to be, and worth actively checking before accepting it. Public Access Prevention has also moved from an opt-in setting to something Google actively pushes organizations toward enforcing by default via org policy, which has measurably reduced (though far from eliminated) the accidental-public-bucket class of incident industry-wide. IR tip: Before you tell a client their exfiltrated data is unrecoverable because the attacker deleted the bucket contents, check soft delete and object versioning status first — in a meaningful fraction of cases, the objects are still sitting there in a recoverable state and nobody's checked. ----------------------------------------------Dean----------------------------------------------------

How to pull a defensible disk image off a running or compromised GCE instance without touching the original evidence. Compute Engine is where most GCP incidents eventually lead you — a web app got popped, a cron job got tampered with, a build agent got used to mint tokens it shouldn't have. The good news is that acquiring forensic evidence off a GCE instance is one of the cleaner workflows in cloud IR, because Persistent Disks are separate resources from the VM they're attached to, and snapshotting one doesn't touch the running instance at all. Disks Are Not the VM A Compute Engine instance is really a compute allocation plus one or more Persistent Disks attached to it — a boot disk holding the OS, and optionally additional data disks. The disk exists as its own GCP resource with its own resource name, independent of the instance's lifecycle: you can snapshot it, clone it, or detach it and attach it somewhere else, all without stopping the instance. That independence is what makes GCE acquisition workable even when you can't afford downtime on a production system — you're never touching the live disk directly, only a point-in-time copy of it. Disk type matters for one thing forensically: consistency guarantees. A snapshot taken while the instance is running is crash-consistent by default — equivalent to what you'd get from a hard power-off, which is fine for most filesystems but can leave an application mid-write in an inconsistent state. If you need application-consistent state (a database in the middle of a transaction, for instance), you'd normally quiesce writes first, but for IR purposes crash-consistent is usually exactly what you want anyway — it reflects reality at the moment of acquisition rather than a cleaned-up version of it. The Snapshot-Based Acquisition Workflow This is the sequence Solstice's IR retainer uses when a compute instance needs to be imaged: Snapshot the disk in place, without stopping the instance, so the attacker (if still active) sees no change in behavior: gcloud compute disks snapshot solstice-web01-boot \ --snapshot-names=ir-2026-08-13-solstice-web01-boot \ --zone=us-central1-a --storage-location=us-central1 Create a new disk from that snapshot in an isolated forensics project — never the source project, to keep the evidence off the same IAM boundary the attacker may still have access to: gcloud compute disks create ir-solstice-web01-boot-copy \ --source-snapshot=ir-2026-08-13-solstice-web01-boot \ --zone=us-central1-a --project=solstice-forensics Attach that disk, read-only, to a clean forensics VM you control in the isolated project, and image it from there with dd or dcfldd rather than mounting it directly: sudo dcfldd if=/dev/sdb of=/mnt/evidence/solstice-web01-boot.dd hash=sha256 hashlog=solstice-web01-boot.sha256 Hash the resulting image and the snapshot's own checksum (Google stores one), and record both in your chain of custody documentation before the raw image ever gets analyzed. Two Things That Trip People Up Snapshot IAM permissions are separate from disk permissions — compute.snapshots.create on the disk's project plus compute.disks.createSnapshot are what you actually need, and it's common for an incident responder to have read access to a project but not snapshot-creation rights, which needs to be granted explicitly and shows up itself as an Admin Activity log entry worth noting for your own timeline. storage-location matters — by default a snapshot inherits the disk's region, but you can pin it elsewhere. For a client with data residency requirements, confirm where the snapshot actually lands before you create it, not after. Instance Metadata: Configuration Data and a Favorite Persistence Mechanism Every GCE instance carries a metadata key-value store, reachable from inside the instance at metadata.google.internal and readable via the API from outside it. Two keys matter more than the rest for IR: startup-script and shutdown-script. Whatever's in those keys runs as root (or SYSTEM on Windows) every time the instance boots or shuts down, and because metadata is editable by anyone with compute.instances.setMetadata on the instance, it's a favorite, boring, extremely effective persistence mechanism — no exploit required, just an IAM permission an attacker already has from an earlier step. Pulling the current and, if available, prior metadata values for these keys should be an early step on any compromised instance. gcloud compute instances describe solstice-web01 --zone=us-central1-a \ --format='value(metadata.items.filter("key:startup-script OR key:shutdown-script"))' The corresponding Admin Activity log entry to hunt for is v1.compute.instances.setMetadata — filter for it across a project and you'll catch both the legitimate configuration changes and the illegitimate ones in the same short list. Who Actually Logged In: OS Login vs. Metadata SSH Keys Linux instances support two different ways to manage SSH access, and they have very different forensic value. Metadata-based SSH keys are just public keys pasted into instance or project metadata — anyone holding the matching private key can log in as the associated local username, and there's no per-user audit trail beyond the OS-level auth log on the instance itself. OS Login, by contrast, ties SSH access to the caller's actual Google identity via IAM (roles/compute.osLogin or osAdminLogin), and every login is attributable to a specific principal in Cloud Audit Logs, not just a key that could have been shared or stolen. If OS Login isn't enabled org-wide, you should expect to lean much more heavily on the guest OS's own auth logs (auth.log, /var/log/secure, or Windows Security event logs) than on Cloud Audit Logs for the actual login attribution. Serial Port Output: The Log You Get Even When the OS Won't Cooperate Every instance's serial console output is captured by Google independent of what's happening inside the guest OS, which makes it valuable when an attacker has tampered with or disabled logging inside the instance itself — serial output survives that. It's retained for the life of the instance and pullable even after the instance is stopped, though it clears on deletion, so grab it before you delete anything. gcloud compute instances get-serial-port-output solstice-web01 --zone=us-central1-a > serial-output.log Hyperdisk is Google's newer Persistent Disk generation, offering independently tunable IOPS/throughput and faster snapshot/restore performance than the older pd-* disk types — if you're imaging a Hyperdisk-backed instance the workflow is identical, just faster. Shielded VM (secure boot, vTPM, integrity monitoring) and Confidential VM (memory encryption via AMD SEV or Intel TDX) are also both far more commonly the default posture now , and it's worth checking whether an instance is Confidential VM-enabled before you plan a memory-acquisition approach, since some capture techniques that assume readable guest memory don't behave the same way against an encrypted memory space. 🔎 IR tip: Pull the serial port output and the current metadata (especially startup-script) before you do anything else with a suspect instance — both are cheap, non-invasive, and won't be there anymore if someone deletes the instance while you're still setting up the snapshot workflow. .......................................................................Dean---------------------------------------------------

When people say "compute" in Google Cloud, they're really talking about anything that burns CPU. If it runs code, processes data, or executes workloads — it falls under compute. For investigations, though, not all compute is equal. Some services give you deep visibility. Others hide almost everything. Here's the case we'll come back to throughout this article: a billing alert flags an unexpected spike in Compute Engine usage on a project called fernbridge-prod. Nobody on the team provisioned new VMs. That's your incident — and everything below is how you'd actually work it. Compute Types (Quick Reality Check) Google Cloud groups compute into three big buckets: IaaS (Infrastructure as a Service) — this is where DFIR people feel at home. You control the operating system, installed software, and users, processes, and disks. Examples: Virtual Machines (Compute Engine), Shielded VMs, GPU-backed VMs. 👉 From a forensic standpoint, this is gold. PaaS (Platform as a Service) — here, Google manages the OS and runtime. You configure the app, not the system. Examples: App Engine, Google Kubernetes Engine (GKE), Cloud Run. You still get logs — but host-level evidence is mostly gone. FaaS (Function as a Service) — this is the black box. Examples: Cloud Functions, Cloud Workflows. You deploy code. Google handles everything else. From a DFIR perspective: great logs, almost zero disk or memory visibility. Why We Focus on Compute Engine (VMs) Google keeps adding new compute services all the time. But for DFIR work, Compute Engine VMs are the most valuable. Why? They behave like traditional systems, they generate OS-level artifacts, you can snapshot disks, you can run agents, and you can do "normal" forensics on them. Most of the investigation techniques you use here carry over to other compute types, but VMs give you the deepest access. In the fernbridge-prod case, the mystery workload turns out to be a handful of brand-new Compute Engine VMs — which is exactly why this article focuses here instead of on GKE or Cloud Functions: VMs are where you actually get to do forensics. VM Types (Don't Overthink It) Google offers a huge list of predefined VM types: General-purpose, High-memory, High-CPU, and Extreme performance machines (full list here). For DFIR, the size doesn't matter much — the artifacts do. A few things worth remembering: VMs can run Windows or Linux, they can use custom images, they can host Docker containers, and you can also define custom machine types. Special VM Configurations (DFIR Angle) GPUs are zone-specific add-ons for an existing VM. They're useful when you need to crunch through evidence fast — say, running ML-assisted malware classification — but they're not an especially interesting artifact source themselves. Preemptible VMs are cheap and not guaranteed to run continuously — they can disappear at any time. Picture this: you're investigating a spike in outbound traffic from a web cluster, and half the logs just stop mid-timeline. That's usually not evidence tampering — it's a preemptible VM getting reclaimed. These are about cost optimization, not stability, and they're common in exactly the kind of web clusters where you'd expect to find them. Shielded VMs add Secure Boot, integrity monitoring, and a virtual TPM (vTPM). Security-wise: great. Forensics-wise: mixed — they often come with additional logins that a standard VM wouldn't have, which is one more thing to check during triage. Capturing Evidence: Disk Snapshots (The Cloud Way) In cloud environments, you don't image disks the old-school way. Instead, you: Snapshot the persistent disk attached to the VM Share that snapshot with your DFIR project Copy it Convert it back into a disk Attach it to a forensic VM as read-only Analyze like a normal disk This works whether your DFIR team is in the same org, a different org, or a completely separate third-party firm — it's just permissions. Same concept as AWS and Azure, just different buttons. For fernbridge-prod, day one looks exactly like this — from Cloud Shell or your workstation: # 1. Snapshot the disk on the suspicious VM gcloud compute disks snapshot fernbridge-vm-07 \ --project=fernbridge-prod \ --zone=us-central1-a \ --snapshot-names=incident-snap-01 # 2. Share the snapshot with your DFIR project gcloud compute snapshots add-iam-policy-binding incident-snap-01 \ --project=fernbridge-prod \ --member="user:you@dfir-project.iam.gserviceaccount.com" \ --role="roles/compute.storageAdmin" # 3. From the DFIR project: copy the snapshot into a new disk gcloud compute disks create incident-disk-01 \ --project=dfir-project \ --source-snapshot=projects/fernbridge-prod/global/snapshots/incident-snap-01 \ --zone=us-central1-a # 4. Attach it read-only to your forensic VM gcloud compute instances attach-disk forensic-vm \ --project=dfir-project \ --zone=us-central1-a \ --disk=incident-disk-01 \ --mode=ro From here, it mounts and analyzes like any other read-only disk. But a snapshot alone won't tell you how those VMs got created in the first place — that's where it gets more complicated. The Big Problem With Snapshots Snapshots are single points in time. That's the catch. Attackers don't freeze just because you took a snapshot — processes keep running, credentials keep getting used, lateral movement continues, and data keeps changing. Modern DFIR often means "leave the system running and observe it live." And this is where cloud changes the game. Live Forensics in Google Cloud: Ops Agent Google Cloud provides a built-in way to collect live telemetry from VMs using the Ops Agent. https://docs.cloud.google.com/logging/docs/agent/ops-agent/installation Instead of stopping the VM, you can collect logs, capture system metrics, monitor application behavior, and observe activity over time — crucial when you're scoping an incident, multiple systems are involved, you need timeline visibility, or shutting systems down simply isn't an option. Think of Ops Agent as "your eyes inside a running cloud VM." This is exactly the gap in the fernbridge-prod case — the mining VMs might already be gone by the time you get around to snapshotting anything. If Ops Agent had been running on them, you'd have live telemetry instead of a cold trail. Snapshot vs Live Telemetry (Real Talk) Disk Snapshot — strength: stable evidence. Weakness: single moment in time. Ops Agent — strength: ongoing visibility. Weakness: requires pre-installation. Logs — strength: broad coverage. Weakness: depends on retention. In real investigations, you use all three. Google Cloud Ops Agent — What It Really Does If you've ever investigated a cloud VM and thought "okay, but what's happening right now on this box?" — that's exactly where Google Cloud Ops Agent comes in. At a high level, Ops Agent is just a small agent running inside the VM (Linux or Windows) that watches logs and ships them to Google Cloud Logging. But the important part isn't what it is — it's how it behaves during investigations. Think of Ops Agent Like This Instead of taking one disk snapshot and hoping it captured the right moment, Ops Agent lets you watch the system while it's alive, see logs as they're being generated, and build timelines without ever stopping the VM. For DFIR, that's huge. How Logs Actually Flow (Simple Mental Model) Here's what's happening under the hood: Logs are generated on the VM — syslog, app logs, and auth logs on Linux; .evtx event logs on Windows Ops Agent reads those logs locally It converts them into structured JSON It sends them to the Google Cloud Logging API From there, your normal Log Sinks and Buckets take over So Ops Agent doesn't store logs — it streams them. Fluent Bit Is the Secret Sauce Ops Agent is basically Fluent Bit wearing a Google jacket. Why that matters: Fluent Bit is fast, lightweight, understands a ton of log formats, and can parse messy logs into clean JSON. This is why Ops Agent works well even on busy systems — it's not some bloated collector. What the Config Actually Looks Like Fluent Bit doesn't guess what to watch — it reads a config file that spells it out. A basic Linux syslog receiver looks like this: logging: receivers: syslog: type: files include_paths: - /var/log/messages - /var/log/syslog service: pipelines: default_pipeline: receivers: [syslog] Why this matters for an investigation: this file tells you exactly which paths Ops Agent is watching — and just as importantly, which paths it isn't. If an attacker knows Ops Agent is running, quietly editing this config (or stopping the agent entirely) is a clean way to blind logging without triggering anything obvious. That makes two commands you should run on every suspicious VM, not just the ones where you already suspect tampering: # Is the agent actually running? sudo systemctl status google-cloud-ops-agent # What is it actually configured to collect — and when was this last touched? cat /etc/google-cloud-ops-agent/config.yaml stat /etc/google-cloud-ops-agent/config.yaml (On Windows, the same file lives at C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml.) A config that's been trimmed down, or a modification timestamp that lines up with the intrusion, is itself an indicator. Structured vs Unstructured Logs (Why You Care) Not all logs are born equal. An unstructured line like: Feb 2 10:41:02 sshd[2211]: Failed password for root becomes, once Ops Agent gets to it, a structured record with a timestamp, hostname, process name, severity, and cleanly parsed fields — meaning you can search cleanly, build timelines, and correlate across systems instead of grepping raw text. Ops Agent also adds its own timestamp when the log is received. This matters because formats like syslog often don't include full date or timezone info — that extra timestamp saves investigations more often than people realize. What Logs Get Collected by Default? Once Ops Agent is installed, it doesn't sit idle. Out of the box, it starts collecting OS logs, common application logs, and auth-related events, and each one is tagged with the VM name, project name, and resource identifiers — that's how Google keeps logs from different projects from leaking into each other. Finding Ops Agent Logs in Log Explorer Logs follow a predictable pattern: projects/[project-name]/logs/[log-name]. To narrow it down to one system, combine that with resource_name (the VM name). This is why Log Explorer searches often look a bit "busy" — you're filtering both what happened and where it happened. Back to fernbridge-prod: this is the actual query you'd run, either in Log Explorer or from the CLI, to see whether Ops Agent caught anything before the instances were deleted: gcloud logging read \ 'logName="projects/fernbridge-prod/logs/syslog" AND resource.labels.instance_id="INSTANCE_ID"' \ --project=fernbridge-prod \ --limit=50 \ --format=json Swap syslog for the log name you're after, and INSTANCE_ID for the VM's numeric instance ID (not its name) — that's what resource.labels.instance_id actually stores. Installing Ops Agent (Zero Drama) You've got two main options: Option 1 — from inside the VM (manual, good for locked-down environments): SSH in, then run: curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh sudo bash add-google-cloud-ops-agent-repo.sh --also-install The agent starts automatically. Confirm it: sudo systemctl status google-cloud-ops-agent Option 2 — via Agent Monitoring Service (most common): from the Compute Engine or Cloud Monitoring console, click Install Agent — Google hands you the same script above, pre-filled, to run on the target VM (installation docs). Even if you install it manually, Google Cloud still detects the agent, shows its status, and tells you which VMs are covered — so you always have visibility into coverage. Why Ops Agent Beats Snapshots (In Many Cases) Snapshots are useful, but they're frozen in time. Ops Agent gives you continuous visibility, ongoing timelines, and evidence while the attacker is still active — and in modern cloud incidents, you rarely investigate just one VM, you rarely have the luxury of stopping systems, and you need to observe rather than interrupt. Ops Agent fits that reality. Multi-Cloud Bonus (This Part Is Underrated) Ops Agent isn't limited to Google Cloud — you can run it on AWS Linux and Windows VMs and send those logs into Google Cloud Logging, using Google Cloud as a central log hub. This is surprisingly common in hybrid environments, migrations, and organizations tired of juggling tools. You just need an AWS connector project — after that, logs flow the same way. What About the Legacy Agent? You'll still see it in the wild. Reality check: it's older, heavier, based on fluentd, and slowly being phased out. Ops Agent uses Fluent Bit, uses less memory, scales better, and is clearly the future. If you see the Legacy Agent during an investigation, don't panic — just know it behaves differently and may produce slightly different logs. Compute Engine Attacks Aren't Theoretical (2025 Data) If the fernbridge-prod scenario sounds oddly specific, that's because it's not far from real life. Since 2023, Google has been tracking a financially motivated actor it calls TRIPLESTRENGTH doing almost exactly this: stealing credentials and session cookies (some sourced from Racoon infostealer logs), hijacking cloud service accounts, and spinning up Compute Engine instances purely to mine cryptocurrency. Their playbook got smarter over time. Early campaigns just abused compromised accounts to create mining VMs directly. Later, they escalated — abuse a highly privileged account to add an attacker-controlled account as a billing contact, then use that billing access to spin up much larger compute resources for mining. Tooling of choice: the unMiner app (which bundles several popular cryptocurrency miners into one package) paired with the unMineable pool, cashing out in TRX. Google's own 2025 telemetry backs this up — in observed post-compromise activity, "Creation/Deletion of cloud instances" showed up as its own tracked category (4.4% of incidents), smaller than lateral movement (62.2%) but exactly the kind of spike Ops Agent and audit logs are built to catch. DFIR takeaway: an unexpected compute spike or a new billing contact on a project is the pattern to check for first. DFIR Takeaway Ops Agent = live visibility Snapshots = point-in-time evidence Logs + telemetry = modern cloud forensics Fluent Bit parsing = clean timelines Project-level tagging = containment and scoping If snapshots tell you what existed, Ops Agent tells you what's happening. fernbridge-prod, resolved: it turned out to be a compromised service account with billing-contact privileges, used to spin up compute for mining — the TRIPLESTRENGTH pattern almost exactly. Ops Agent hadn't been installed on the new VMs (they were too new), but the Admin Activity audit log for the IAM change on the billing account was enough to reconstruct the timeline — the kind of query that finds it: gcloud logging read \ 'protoPayload.methodName="SetIamPolicy" AND resource.type="billing_account"' \ --project=fernbridge-prod \ --freshness=30d \ --format=json That single IAM change — an unfamiliar account added as a billing contact — was the actual root cause. Everything after it, including every VM Ops Agent never got a chance to watch, followed from that one event. ----------------------------------------------------Dean--------------------------------------------

Why Google Cloud Matters for DFIR Most enterprise workloads still run on-premise, not because cloud platforms are weak, but because migration introduces architectural and operational complexity. Unlike traditional environments, investigators in Google Cloud cannot rely on physical access, predictable network paths, or full host-level visibility. Instead, identity events, service-level logs, and resource metadata become the primary evidence sources. Today, most of the world's enterprise computing still happens on-premise. It hasn't moved to the cloud yet, because the path forward is complex and daunting, and full of difficult decisions. How do you modernize in-place without having to jump completely to the cloud? How do you bridge incompatible architectures while you transition? A nd how do you maintain flexibility and avoid lock-in?” — Sundar Pichai, Google's CEO, Google Cloud Next 2019 And the scale behind it: at the time this curriculum was written, Google Cloud was the third-largest cloud provider by market share (behind AWS and Azure) but the fastest-growing of the three, with analysts forecasting overall public cloud spend to grow 18.4% that year. Faster growth means more new, less-mature deployments — exactly why cloud infrastructure keeps landing in threat actors' crosshairs. ------------------------------------------------------------------------------------------------------------ Google Cloud Global Architecture (DFIR Angle) Google Cloud is built on regions and zones: Regions are geographic areas (e.g., Europe-West, US-Central) Zones are isolated deployment areas within regions Why this matters for DFIR: Evidence may be generated in one zone and stored or processed in another Data residency and regulatory requirements apply when collecting evidence Network paths inside Google Cloud are abstracted and not traceable You cannot reconstruct packet paths like on-prem networks. Investigations must focus on what happened, who did it, and which resource was affected, not how traffic flowed. Worth naming the real scale here instead of leaving “regions and zones” abstract: Google Cloud runs 25 regions and 76 zones, with more added regularly. And there's a concrete reason network paths are abstracted: Google interconnects those regions using a mix of its own dedicated network links and shared submarine cables. Redundancy varies a lot by geography too — the US has heavy ingress/egress redundancy, while regions like Australia have comparatively limited links. That's part of why you can't reconstruct a packet path the way you would on-prem — the routing is spread across infrastructure you don't get visibility into, by design. ------------------------------------------------------------------------------------------------------------ Core Google Cloud Services for Incident Response & Forensics These five services provide the highest forensic value during cloud incidents: Identity & Access Management (IAM) Compute Engine (VMs) Cloud Logging Cloud Networking Cloud Storage Buckets Almost every cloud compromise touches IAM + Logging + Compute ------------------------------------------------------------------------------------------------------------ Google Cloud Resource Hierarchy (Critical for Investigators) Google Cloud follows a strict hierarchy: Organization └── Folder(s) └── Project(s) └── Resource(s) Breakdown Organization Highest level in Google Cloud Usually mapped to a company’s domain Often linked with Google Workspace Central point for IAM and security policies From a DFIR perspective, this is where global controls and investigation-wide visibility are established. Folders Used to group teams, environments, or functions Policies applied here affect everything beneath them Can have multiple layers (sub-folders) DFIR benefit: Separate DFIR resources from production Apply investigation-specific policies Limit blast radius during incidents Projects Where services actually run Billing and resource ownership boundary Minimum requirement for creating resources Most compromises are identified inside projects, making them the primary focus during investigations. Resources Compute, Storage, Networking, Logging, etc. Inherit permissions and constraints from above Generate most forensic artifacts Example Organization: novaharbor.io (the company's primary domain) Folder: Incident-Response — isolates DFIR tooling from production Sub-folder: dfir-sandbox — a further split for one-off investigation VMs Project: evidence-processing-prod — where the actual acquisition/processing VMs and storage buckets live Resources: a Compute VM running Plaso, a Storage Bucket holding disk images Policies applied at the Incident-Response folder level — read-only visibility for DFIR staff, a ban on disabling logging — automatically apply down through dfir-sandbox and evidence-processing-prod without touching each resource individually. That's the practical payoff of the hierarchy: set it once at the folder, and every project and resource under it inherits it. ------------------------------------------------------------------------------------------------------------ IAM vs Policies (Very Important Distinction) Concept Controls IAM Who can access resources Policies / Constraints What resources can do Example: IAM: Who can access a VM Policy: Whether logging can be disabled on that VM Use policies to: Prevent attackers from: Disabling logs Creating resources in rogue regions Lock down compromised assets instantly ------------------------------------------------------------------------------------------------------------ Google Cloud Organization Benefits for DFIR What DFIR Teams Can Do Quickly Apply constraints globally Allowed VM templates Approved regions only Grant DFIR read-only visibility everywhere Prevent destructive actions: Turning off logging Deleting evidence Isolate compromised projects without downtime ------------------------------------------------------------------------------------------------------------ Key Google Cloud Resources Attackers Abuse Focus first on these during investigations: Compute & Platform Compute Engine App Engine Kubernetes Engine Storage Cloud Storage Filestore Ops & Visibility Logging Monitoring Networking VPC Network Security Services ------------------------------------------------------------------------------------------------------------ Data Transfer Pricing (Why DFIR Teams Get Burned) Important Cost Rules Same zone + internal IP → No cost Different zone or region → Cost VM-to-VM traffic can incur: Egress Response ingress (TCP) Google Cloud was not charging for ingress traffic into a resource — only egress and cross-zone/region movement carried a cost. That's subject to change, so always check Google Cloud's current pricing page before assuming it still holds; don't bank an evidence-transfer plan on a pricing detail that could shift. DFIR Best Practices ✅ Place DFIR VMs in: Same region Same zone Use internal IPs only Avoid: Moving evidence across zones Exporting raw evidence unnecessarily ------------------------------------------------------------------------------------------------------------ Cost Reduction Options for DFIR 1. Preemptible VMs Cheaper No guaranteed CPU Good for: Distributed tasks Bad for: Single-VM processing (e.g., Plaso) 2. Keep Processing Local Same zone = free internal traffic Faster acquisition & analysis 3. Be Smart with BigQuery Charged per bytes processed Reduce cost by: Fewer queries Better SQL Documenting results to avoid reruns BigQuery actually bills on two separate axes: the cost of holding data in a table, and the cost of querying it. The tips above (fewer queries, better SQL, documenting results) only reduce the second one — there isn't much you can do to reduce storage/holding cost once evidence is sitting in BigQuery, so factor that in when deciding how long to keep large exported log sets there versus moving them to cheaper cold storage. ------------------------------------------------------------------------------------------------------------ Key Takeaways IAM + Policies = Cloud Incident Control Plane Folder & project structure directly affects response speed Logging enforcement is your strongest defense Data movement = cost + legal risk DFIR architecture planning matters before incidents happen --------------------------------------------Dean-----------------------------------------------------------

If you’ve worked in Google Workspace long enough, you already know this truth: Google Drive is where data leaks love to happen. Not always malicious. Sometimes it’s just: “Oops, shared it publicly” “Oops, shared it with the wrong domain” “Oops, didn’t realize Anyone with the link means literally anyone” So when data exposure happens, we usually care about two questions: What happened to the file? Can we still access or recover it? That’s where Google Drive investigation tools come in. ------------------------------------------------------------------------------------------------------------- Tool 1: Google Drive Log Events (Your Timeline, Not Your Files) Think of the Drive Log Events as your CCTV footage, not the evidence locker. What it’s good at: Showing who did what Showing when it happened Showing permission changes Near real-time visibility (usually within minutes) What it’s not good at: Accessing files Showing file contents Tracking anonymous viewers or downloads Key Things to Know About Drive Log Events Let’s break this down simply: Keeps 6 months of history Logs actions like: File creation Sharing changes Permission updates Deletions Does NOT give you the file itself CSV export is limited to 100,000 rows Unauthenticated access is only logged for editing Viewing or downloading by anonymous users? ❌ Not logged So if a file was publicly shared and downloaded 1,000 times anonymously — the audit log will not tell you that. Painful, but important to know upfront. Where Are Drive Log events Now? Earlier, Drive log events lived in their own section. That changed. Today, Drive Log Events live inside in the Google Admin Console. Inside Investigator, you can: Filter events Use AND / OR logic Group by fields (user, document, event type) Search using partial matches One warning though ⚠️Even though logs are generated quickly, some events can lag up to 12 hours before showing up. ------------------------------------------------------------------------------------------------------------ Tool 2: Google Vault (This Is Where the Files Live) If Audit Logs are the timeline, Vault is the evidence room. Vault is what you use when: You actually need the document A file was deleted A user “accidentally” removed something important But Vault comes with conditions. What Vault Can Do Access files in user Drives Access deleted files Apply holds Enforce retention rules What Vault Cannot Do Give you an audit trail Tell you who did what and when It’s access, not visibility. Deletion Timelines (This Matters a LOT) Here’s the reality of deleted files: When a user deletes a file → it goes to Trash Trash keeps files for 30 days Once removed from Trash: Admins have 25 more days to recover (without Vault) ·With Vault, that window extends to 25–40 days total from the original deletion date — the extra 15 days depends on when Google actually purges the file from its own storage Custom retention rules or holds = files stay longer If Vault is enabled, you can often recover files without restoring the user account. ------------------------------------------------------------------------------------------------------------ Alternate File Recovery Scenarios (The “Oh No” Cases) Case 1: Active User Deleted Files Trash keeps files for 30 days After Trash deletion: Admins have 25 days to restore Restore options: Original location Shared Drive No Vault license? After 25 days — game over. Case 2: Deleted User Account This one catches teams off guard. Deleted user accounts can be restored for 20 days Files can only be recovered if: The user account is restored first Or Vault is used Ownership transfer is another option: Files move to another user’s Drive Again — Vault makes life easier here. ------------------------------------------------------------------------------------------------------------ Exporting Drive Logs Once you've found the events that matter, you don't want to sit there scrolling — you want them out of the console and into something you can search, filter, or hand off. Where to export: From the Investigation Tool (Admin Console → Security → Investigation Tool), run your filtered search, then use Export → CSV. Vault has its own export flow (Vault → Search → Export), if you're pulling files or metadata for a hold rather than just the audit trail. What you get: A CSV of the events matching your filters — actor, timestamp, IP, document ID, visibility change, and the other fields covered above. Capped at 100,000 rows per export. If you're hunting something broad (a domain-wide sharing change, for example), narrow the filter first or you'll quietly lose rows past the ceiling. One more thing to know: Exports don't include file contents — same limitation as the log events themselves. You're exporting the trail, not the evidence. Large exports can take a few minutes to generate; Google emails you a download link rather than streaming it back instantly. Clunky, but it's the difference between "I remember seeing something" and having a file you can actually attach to a ticket. Once you've got that CSV, don't just open it in Excel and start scrolling. Eric Zimmerman's Timeline Explorer handles this kind of exported log data far better — filter, sort, and pivot without losing your place, and you're not fighting Excel's row limits or crash-on-large-file quirks. Especially useful if you're keeping the analysis off the (possibly compromised) Workspace tenant entirely. ------------------------------------------------------------------------------------------------------------ Important Fields You’ll Actually Use During an Investigation Let’s translate the useful ones into investigator language: Document ID This is gold. Unique across all Google Workspace tenants Same ID you see in the document URL Perfect for matching phishing URLs to actual Drive files Owner vs Actor Owner: Who owns the file Actor (User field): Who performed the action These are often not the same person. Visibility & Prior Visibility This tells the real story. Prior Visibility → what access looked like before Visibility → what access looks like now This is how you catch: Private → Public changes Internal → External sharing Domain-wide exposure IP Address Extremely useful for: Geo anomalies Impossible travel Correlation with other Workspace logs One more field worth knowing: ### Billable · Shows whether the document counts toward the user's storage quota · Only actually populated on the Essentials edition Small field, easy to skip — but if you're chasing a storage-quota mystery on an Essentials tenant, this is the column that explains it. ------------------------------------------------------------------------------------------------------------ Final Thoughts (The Big Picture) Drive log events — who did what, and when Vault — the actual file, if you need to see or recover it Visibility & Prior Visibility — how exposure changed, and when it changed An export — so the timeline survives the investigation, not just your memory of it No single tool gives you the full picture. Put them together, and you've got one. ---------------------------------------------Dean-----------------------------------------------------------

Up until now, most Google Workspace investigations start in one of two places: The Admin Console Or Workspace APIs Both are useful. Both have limits. At some point though, especially in larger or more mature environments, logs don’t just live inside Workspace anymore — they’re exported into Google Cloud. And once that happens, the Admin Console alone isn’t enough. That’s where gcloud comes in. ------------------------------------------------------------------------------------------------------ What gcloud Actually Is (And What It Isn’t) Let’s clear this up first. gcloud is not some hacking tool or special DFIR-only utility. It’s the official command-line interface for Google Cloud, bundled as part of the Google Cloud SDK. Think of it as: The terminal version of Google Cloud Console It works on: Windows macOS Linux And it’s designed for: Automation Scripting Command-line access to Cloud services What it’s not designed for: Being embedded into applications Acting like a full SDK inside other tools For investigators, that’s perfect — because we want read-only, controlled, scriptable access. ------------------------------------------------------------------------------------------------------ Why Use gcloud Instead of the Web Console? You can view logs in the Google Cloud web UI. That works fine for quick checks. But gcloud gives us a few big advantages: Better control over log extraction Easier time-based filtering Clean JSON output (huge win for DFIR) Works well from isolated IR workstations Easy to feed into tools like SOF-ELK If you’re doing serious timeline reconstruction or long-range log analysis, gcloud is simply more practical. ---------------------------------------------------------------------------------------------------------- Authentication: Who Is gcloud Acting As? Before gcloud can do anything, it needs an identity. You have two main options: 1. Service Account Good for automation, repeatable workflows, and controlled access. 2. User Account More common during investigations, especially when speed matters. In both cases, OAuth is used to authorize access to Google Cloud. And here’s the key permission you need to remember: Private Logs Viewer ---------------------------------------------------------------------------------------------------------- Why “Private Logs Viewer” Matters Google Cloud has two commonly confused roles: Logs Viewer Private Logs Viewer For investigations, Logs Viewer is not enough. Private Logs Viewer gives you access to: Audit logs Logs containing IP addresses Sensitive user activity metadata That’s exactly what we care about during IR. The good news? Both roles are read-only. You cannot modify or delete logs with either. So you get visibility without risk. ---------------------------------------------------------------------------------------------------------- Installing and Preparing gcloud https://docs.cloud.google.com/sdk/docs/install-sdk Once you install the gcloud CLI on your investigation host, there are three things you always need to tell it: Which project you’re working in Who you are (authentication) What logs you want Projects matter because logs live inside Google Cloud Projects, not “Workspace” directly. Workspace just sends logs there. One flag worth knowing if your investigation host doesn't have a GUI or a browser: gcloud init --no-browser. It walks you through the same setup, but instead of popping open a browser window, it hands you a URL to open somewhere else and a verification code to paste back in — exactly what you need on a locked-down, headless IR box. ---------------------------------------------------------------------------------------------------------- Logging Buckets: Where Your Logs Actually Live In Google Cloud, logs are stored in Logging Buckets. Buckets are not sized by data volume — they’re sized by retention days. In almost every project, you’ll see at least: _Required _Default Anything beyond that was created intentionally by admins. As an investigator, one field becomes very important here: retention_days Because it defines: How far back you can go Whether old evidence still exists ---------------------------------------------------------------------------------------------------------- Narrowing Down to Google Workspace Logs Not all logs in Google Cloud are Workspace logs. So instead of searching everything, we filter by service name. Workspace-related logs usually come from: admin.googleapis.com cloudidentity.googleapis.com login.googleapis.com oauth2.googleapis.com This alone removes a ton of noise. ---------------------------------------------------------------------------------------------------------- Time Ranges: Never Forget This (Seriously) This is one of those gcloud “gotchas” that burns people. If you don’t specify a time range, gcloud will: Return only 10 log entries Not 10 pages. Not 10 minutes. Just 10 lines. So every serious query must include a timestamp filter. ---------------------------------------------------------------------------------------------------------- Pulling Logs the Right Way (And Why JSON Matters) When we extract logs, we don’t want pretty output — we want machine-consumable evidence. That’s why we force JSON output. Here’s what a real-world Workspace log pull looks like: gcloud logging read "protoPayload.serviceName=(admin.googleapis.com OR cloudidentity.googleapis.com OR login.googleapis.com OR oauth2.googleapis.com) AND timestamp>=\"2026-01-01T00:00:00Z\" AND timestamp<=\"2026-01-30T00:00:00Z\"" --format=json > gws_logs_in_gcp.json What’s happening here: We limit results to Workspace-related services We define a clear investigation window We output everything as JSON We write it to a file for offline analysis ---------------------------------------------------------------------------------------------------------- Why gcloud Is So Useful in DFIR The real value of gcloud isn’t just “getting logs”. It’s that you can: Re-run queries consistently Adjust time windows precisely Preserve raw evidence Avoid UI-based filtering mistakes Work even when the web console feels slow or limited And once authenticated, gcloud can do anything your account is authorized to do — we just happen to care about logs. ---------------------------------------------------------------------------------------------------------- One Last Thing: Logging Out Matters This sounds basic, but it’s important. When you authenticate with gcloud, you’re opening an active API session. Google Cloud has no way of knowing you’re “done” unless you explicitly log out. From an investigation hygiene perspective: Always revoke or log out Especially on shared IR systems Especially after admin-level access ---------------------------------------------------------------------------------------------------------- Final Thoughts Using gcloud for Google Workspace investigations is one of those skills that feels optional — until it suddenly isn’t. When logs move to Google Cloud: The Admin Console becomes secondary APIs don’t always give full visibility CLI access becomes your best friend ----------------------------------------Dean------------------------------------------------------------