Networking: VPC, Flow Logs, and Cloud DNS

7 days ago

5 min read

Where the traffic actually went — and why GCP's network model means one VPC can span the whole company.

GCP's networking model has one structural quirk that changes how you scope an investigation compared to AWS: a VPC in Google Cloud is global, not regional, and it's also a project-level resource that can be shared across other projects entirely. Get this wrong and you'll scope an incident to a single project when the network the attacker actually moved through spans half the org.

Shared VPC: One Network, Many Projects

Solstice Payments runs a Shared VPC out of solstice-shared-net, the host project, with subnets in us-central1 (10.20.1.0/24) and europe-west1 (10.20.2.0/24).

Both solstice-prod-core and solstice-prod-data are service projects attached to that host — their instances get IPs out of those same subnets, but IAM permissions over the network itself (who can create firewall rules, who can peer another VPC into it) live in the host project, not the service projects.

This is the single most common scoping mistake I see:

An investigator pulls IAM and Flow Logs for the compromised service project, sees nothing unusual about the network configuration, and never checks the host project where the actual firewall rule changes happened.

First move on any network-touching incident: identify whether the VPC in play is standalone or shared, and if shared, go find the host project.

gcloud compute shared-vpc get-host-project solstice-prod-core

VPC Flow Logs

Flow Logs record a sampled summary of IP traffic to and from VM interfaces — not full packet capture, but enough to answer

"did this instance talk to that IP, and how much data moved."

They're enabled per-subnet, off by default, and like Data Access audit logs, are one of the things worth confirming is actually turned on before you assume you'll have network evidence.

The fields that matter most in a flow log record:

connection.src_ip / connection.dest_ip and the matching ports — the basic who-talked-to-whom.
bytes_sent and packet counts — useful for spotting a large outbound transfer that doesn't match the instance's normal traffic profile, which is often the first concrete signal of data exfiltration over the network layer rather than an API call.
reporter (SRC or DEST) — tells you which side of the connection generated this particular record, since both ends can report the same flow.
rtt_msec — round-trip time, occasionally useful for distinguishing traffic that stayed inside Google's backbone (very low RTT) from traffic that left to the public internet and came back.

Sampling matters:

Flow Logs default to 50% sampling and 5-second aggregation intervals, both configurable. At default settings you will not see every single connection, which is worth stating explicitly in a report if a client asks why a specific short-lived connection isn't in the logs — it may genuinely not have been sampled, not that logging failed.

gcloud compute networks subnets update solstice-central-subnet --region=us-central1 \
  --enable-flow-logs --logging-flow-sampling=1.0 --logging-aggregation-interval=interval-5-sec

Firewall Rule Logging

Separate from Flow Logs, individual firewall rules can have their own logging enabled, recording every connection the rule allowed or denied — including the specific rule that matched.

This is what you want when you need to know not just that traffic moved, but which firewall rule let it through, especially useful when an attacker (or a misconfiguration) added an overly permissive rule and you need to prove exactly when it started being exploited.

Packet Mirroring: Full Packet Capture, Not Just Flow Summaries

Flow Logs and firewall logging tell you that traffic moved and which rule let it through — they don't hand you the actual packets.

For that, Google Cloud has Packet Mirroring:

Full packet capture defined per VM instance, per GKE cluster, or per VPC, which makes it straightforward to scope a capture tightly around whatever's suspected of being compromised rather than pulling traffic for an entire environment.

GCP's implementation works differently from a typical on-prem tap.

A Mirror Policy defines which source instances to capture from, but the mirrored traffic doesn't land in a bucket directly — Google Cloud has no mechanism to write a PCAP straight to Cloud Storage.

Instead, the mirrored packets are handed to a load balancer forwarding rule, which distributes them across a backend instance group exactly the way a normal load balancer distributes inbound requests, except here the "requests" are copies of live traffic.

For DFIR purposes, that backend instance group is where you'd stand up Arkime, Zeek, or another NSM tool to actually capture and analyze what's mirrored.

Setting it up takes two pieces. First, an internal load balancer forwarding rule flagged as a mirroring collector — this is what the mirrored packets actually get delivered to.

Solstice already has an NSM backend service (running Zeek) fronted by this rule:

gcloud compute forwarding-rules create solstice-mirror-collector \
  --region=us-central1 --load-balancing-scheme=INTERNAL \
  --backend-service=solstice-nsm-backend --is-mirroring-collector

Then the mirroring policy itself, pointing at that collector and naming what to mirror — in an active incident you'd usually scope this to the one suspect instance, not the whole subnet:

gcloud compute packet-mirrorings create solstice-prod-core-mirror \
  --region=us-central1 --network=solstice-shared-net \
  --collector-ilb=solstice-mirror-collector \
  --mirrored-instances=solstice-prod-core --filter-direction=BOTH

The catch for incident response:

Packet Mirroring is forward-looking only. It captures traffic sent after the policy goes live — there's no way to retroactively pull packet-level data for a window before the mirror was configured. If Solstice's SRE team suspects solstice-prod-core is actively being used for data exfiltration and Flow

Logs alone aren't giving enough detail, standing up a Mirror Policy against that instance immediately is the move — but it only helps for what happens next, not what already happened.

Cloud DNS Logging

DNS query logging for a VPC's managed zones is also opt-in and separately configured from Flow Logs.

It records the query name, response, and source, and is often the fastest way to confirm C2 activity when an attacker is using domain-based command and control rather than a raw IP — Flow Logs will show you the resolved IP, but Cloud DNS logs are what tie that IP back to the domain name that was actually queried, which matters when the IP itself sits behind shared infrastructure like a CDN or the attacker rotates IPs faster than domains.

Load Balancer Logs

For anything sitting behind a Google Cloud Load Balancer — which is most public-facing Solstice services — the load balancer's own request logs are frequently more useful than the backend instance's application logs, because they're generated regardless of what happened to the backend afterward. Each entry includes the client IP, request path, response code, latency, and, for HTTPS(S) load balancers, the negotiated TLS details.

If a web app was compromised through a specific malicious request, the load balancer log is often the cleanest place to find the initial request that did it, especially if the attacker's later steps involved tampering with or disabling the application's own logging.

Firewall rules moved toward a hierarchical model — Firewall Policies at the org and folder level, evaluated alongside (and able to override) legacy per-VPC firewall rules, giving central security teams a way to enforce network controls that individual project owners can't loosen.

If you're investigating a network permissiveness question, check for hierarchical Firewall Policies in addition to the classic per-VPC rules — the effective policy is a merge of both, and looking at only one gives an incomplete picture. Cloud NGFW (next-generation firewall, with intrusion prevention) also expanded significantly since 2022 and may be in play for environments with more mature network security postures.

🔎 IR tip: Before concluding "no evidence of network exfiltration," confirm three things were actually on: Flow Logs at an adequate sampling rate, firewall rule logging on the relevant rules, and Cloud DNS logging. Absence of evidence across all three is a real finding worth reporting on its own — but a gap in even one of them can produce a false negative that looks identical to a clean network.

..-------------------------------------Dean---------------------------------------------------