The core idea In Google Cloud, Service Accounts are identities for machines, not humans .They are used by resources like VMs, Cloud Functions, Kubernetes, etc. to talk to other Google Cloud services. Unlike AWS (where users can directly generate API keys), Google Cloud forces you to use Service Accounts when you want: Programmatic access Static credentials Non-interactive authentication So:👉 If code needs access, it almost always runs as a Service Account. ---------------------------------------------------------------------------------------------- What actually happens when you create a VM When you create a VM: Google Cloud automatically creates (or assigns) a Service Account That Service Account: Appears in IAM Can be granted roles just like a user Is used by the VM to access other resources (Storage, APIs, databases) You can delete this Service Account from IAM —but then your VM will break if it needs to talk to anything else. Best practice in reality : Don’t delete it — restrict its permissions. ---------------------------------------------------------------------------------------------- The real danger: Basic Roles (Owner & Editor) On paper Google Cloud has Basic Roles: Viewer Editor Owner They were created early on to make things “easy”. In practice (this is where things go wrong) The Editor role is dangerously powerful: An account with Editor can: Modify resources Create API keys Use actAs to impersonate other accounts Create credentials for other accounts Key insight: Editor ≈ Owner (from an attacker’s point of view) ---------------------------------------------------------------------------------------------- Why attackers love Editor accounts If a threat actor compromises any account with Editor: They can create an API key They can impersonate (actAs) higher-privileged accounts They can effectively privilege-escalate to Owner This is not theoretical — it’s abused in real incidents. ---------------------------------------------------------------------------------------------- Real-world lateral movement scenario Let’s walk through the actual attack flow. Step 1 – Environment setup (normal behavior) Infra team creates a Development Project VMs are deployed for developers Each VM has a Service Account Everything is isolated. Looks safe. Step 2 – Developer needs storage (very common) Developer needs persistent storage Creates a Cloud Storage Bucket Grants the VM’s Service Account Editor access to the bucket From the developer’s perspective: “It works, job done.” Step 3 – Credentials exposure One of the following happens: Service Account key committed to GitHub Credentials stored in code VM is compromised and metadata server is abused Now the attacker has:👉 Service Account credentials with Editor permissions Step 4 – Privilege escalation With Editor access, the attacker can: Create new API keys Impersonate other IAM accounts Take over Owner accounts in the same project Step 5 – Organization-level impact If any Org-level bound account exists in that project: The attacker can impersonate it Escalate to the Organization Gain control over: All projects All resources Entire cloud environment Single Service Account compromise → Full Org takeover Someone already automated this exact chain: gcploit, built by Dylan Ayrey. It maps every resource reachable via actAs starting from a single compromised Service Account — turning the manual 5-step walkthrough above into a single tool run. Worth knowing whether you're investigating a breach (this may be exactly what the attacker ran) or hardening an environment (run it against your own org first). ---------------------------------------------------------------------------------------------- Why this is hard to detect (investigation challenge) The IAM visibility problem In Google Cloud: Resource owners decide access There’s no single place that shows: “What does this account have access to across the Org?” This creates: Hidden trust relationships Accidental cross-project access Silent privilege escalation paths ---------------------------------------------------------------------------------------------- TInvestigator & Defender mindset When you’re investigating or hardening Google Cloud: Red flags to look for Service Accounts with Editor role Shared Service Accounts across projects Exposed Service Account keys Unexpected actAs activity API keys created by non-human identities ---------------------------------------------------------------------------------------------- Defensive mindset shift ❌ “Editor is fine for dev”✅ “Editor is a privilege escalation waiting to happen” ---------------------------------------------------------------------------------------------- One-line summary In Google Cloud, Service Accounts with Editor permissions act as silent trust bridges—once compromised, they enable privilege escalation, lateral movement, and even full organization takeover without deploying malware. -----------------------------------------------Dean---------------------------------------------

Let’s talk about something that often comes up during Google Workspace investigations: how do we actually collect logs and evidence properly? If you’ve ever worked an incident involving Google Workspace, you already know that the platform gives you a lot of data—but not all of it is equally easy to collect or analyze. Broadly speaking, there are two main ways to collect evidence from Google Workspace: Using the Workspace Admin interface (UI) Using the Workspace Admin SDK / APIs On paper, both give you access to similar information. In reality, they behave quite differently—and those differences really matter during forensic analysis Let’s break this down in a simple, practical way. ------------------------------------------------------------------------------------------------------------- Option 1: Using the Google Workspace Admin Interface The Admin interface is usually where everyone starts—and honestly, it’s not a bad place to begin. It gives you a visual and human-friendly way to explore logs. You can click through different sections, filter events, and clearly see what’s going on. ' This is especially useful when: You’re doing a quick triage You need to show evidence to a manager, legal team, or client You want to visually confirm suspicious activity The downside? All the useful data is scattered across different screens. If you want to investigate a full Workspace compromise, you’ll likely need to: Jump between Drive logs Check login and authentication activity Review OAuth and third‑party app access Inspect Admin console changes Each of these lives in a different place. That means a lot of clicking, filtering, exporting, and repeating the process again and again. It works—but it’s slow. Export limitations There are a few important limitations to keep in mind: You can only export 10,000 or 100,000 events per log type. If you exceed that limit, you must split your search into smaller time ranges Logs are exported only as Google Sheets (GSheet) from the UI You can later convert those sheets into CSV, but it’s an extra step—and not ideal if you’re planning to ingest logs into a SIEM or timeline tool. Name the tool for working with those CSV exports If you go the UI/GSheet route, don't just open these in Excel — there's a purpose-built free tool for this: Eric Zimmerman's Timeline Explorer. It imports the CSV export directly and lets you search and filter it far faster than Excel ever will. One thing worth checking every time you use it: look at the highest row number Timeline Explorer shows you. If it lands right at 10,000 or 100,000, that's not a coincidence — you've hit the UI export cap, and there's more data you haven't seen yet. Split your time range and pull the rest. Option 2: Collecting Logs via the Workspace Admin SDK (API) Now this is where things get really interesting for forensic work. The Workspace Admin SDK allows you to collect logs programmatically using API calls. Once set up, this becomes the fastest and most consistent way to gather evidence. Yes, the initial setup takes some effort—you’ll need: A Service Account The right Workspace permissions Some basic scripting knowledge But once that’s done, everything becomes repeatable and scalable. Types of reports you can collect Using the API, you can pull two main types of reports: 1. Activity Reports These tell you what actually happened across Workspace services, including: Google Drive activity Authentication and login events OAuth and third‑party application access Admin console changes These are gold during investigations because they help you track changes, abuse, and attacker actions. 2. Usage Reports These focus more on how user accounts are being used over time. They’re great for spotting anomalies or misuse patterns. Why investigators prefer API logs There are several big advantages here: No event limits like the UI exports Logs are returned in JSON format, which is perfect for: SIEM ingestion Timeline creation Custom parsing and analysis All timestamps are in UTC, which avoids time zone confusion Collection can be fully scripted, ensuring consistency every time In short: if you’re doing a serious investigation, the API approach is hard to beat. ------------------------------------------------------------------------------------------------------------- Option 3: Sending Google Workspace Logs to Google Cloud Logging There’s a third option that often gets overlooked—but it’s extremely powerful. Google Workspace can send certain logs directly to Google Cloud Logging. This allows you to: Retain logs for a much longer period Query them using Cloud Log Explorer Correlate Workspace logs with other Google Cloud activity You Must enabled the sharing which is disabeld by default The catch Not all Workspace logs are sent to Google Cloud. Only five log types are forwarded—and while these are some of the most valuable ones for investigations, they don’t always tell the full story. Name the five log types that actually get forwarded Specifically, these five: Admin Activities, Enterprise Groups, Login Audit, OAuth Token, and SAML. These map to four service names you'll actually query in Cloud Logging (admin.googleapis.com, cloudidentity.googleapis.com, login.googleapis.com, oauth2.googleapis.com) — which lines up with the query further down this article. Knowing the five by name up front makes it obvious at a glance whether the data you need is even in scope for this method, instead of finding out the hard way mid-investigation. For example: Email transit and email access logs are not included You cannot customize which logs are sent Google decides what gets forwarded—you only choose whether forwarding is enabled or not So while this method is fantastic for long‑term visibility, it should be seen as a complement, not a replacement, for API‑based collection. ------------------------------------------------------------------------------------------------------------- Permissions: A Common Roadblock If you try to search Workspace logs in Google Cloud and run into permission errors—don’t panic. This usually means your account doesn’t have enough rights to query logs. https://docs.cloud.google.com/logging/docs/access-control The fix is simple: Go to IAM & Admin in Google Cloud Grant the appropriate role (typically Logging Admin or equivalent) Once that’s done, Log Explorer will start behaving as expected. Log Explorer Example Querying Workspace Logs in Google Cloud When Workspace logs arrive in Google Cloud, they are spread across a few service names. To search them together, you can use a query like this in Log Explorer: protoPayload.serviceName = ( "admin.googleapis.com" OR "cloudidentity.googleapis.com" OR "login.googleapis.com" OR "oauth2.googleapis.com" ) Example: protoPayload.serviceName = ( "login.googleapis.com" ) One important thing to remember: you need to be viewing logs at the root organization level in Google Cloud. ------------------------------------------------------------------------------------------------------------- Final Thoughts If we simplify everything: Admin UI → great for quick checks and visual walkthroughs Admin SDK / API → best for fast, consistent, forensic‑grade evidence collection Google Cloud Logging → excellent for long‑term retention and centralized querying In real investigations, the strongest approach is usually a combination of all three. ------------------------------------------Dean-------------------------------------------------------------

Let me be honest upfront: this setup looks scary the first time you see it. Google makes you jump back and forth between Google Cloud Console and Google Workspace Admin, and it feels like you’re doing something wrong the entire time. You’re not. That’s just how Google designed it. Once you understand the full flow, everything suddenly clicks. This walkthrough assumes: You are a Google Workspace Super Admin You want to collect audit / activity logs using the Admin SDK – Reports API Big picture first (so you don’t get lost) You will work in two places: Google Cloud Console Create a project Enable APIs Create a service account Google Workspace Admin Console Trust that service account using domain-wide delegation Google Workspace itself does not have service accounts. That’s why Google Cloud is involved at all. We’re basically borrowing Google Cloud’s identity system to talk to Workspace. Step 1: Create a Google Cloud Project (same org as Workspace) Start here: 👉 https://console.cloud.google.com Click the project dropdown in the top bar Select New Project Set: Project name: workspace-log-collection Organization: must be the same org as your Workspace tenant Click Create That’s it. Important thing to understand: you are not deploying servers, VMs, or storage. This project is just a container to hold APIs and a service account. Step 2: Enable the required APIs (this is mandatory) Google locks everything by default, so we have to explicitly enable what we need. Inside your new project: Go to APIs & Services → Library Search for and enable: Admin SDK API (this is the key one) Optional (only if you plan to query these later): Google Drive API Gmail API Calendar API The optional Vault API scope One more optional one worth knowing about: G Suite Vault API. Enable this too if you ever want the same service account to also pull Vault-specific data (holds, exports, search results) instead of just audit/activity logs. It's outside the scope of a basic log-pulling setup like this one, but it's the exact same pattern if you decide to expand later — same project, same service account, just one more API enabled. For audit and activity logs, Admin SDK alone is enough. If this API is not enabled, your script will fail even if every permission looks perfect. Step 3: Configure OAuth Consent Screen (yes, even for service accounts) This step confuses almost everyone. Even though we’re using a service account, Google still requires an OAuth consent screen to exist. Go to APIs & Services → OAuth consent screen Choose Internal You only see this option because the project is under a Workspace org Fill in the basics: App name: Workspace Log Collector User support email: your admin email Developer contact email: your admin email Click Save and Continue On the Scopes page → just click Save and Continue Finish You do not need to publish the app externally. Think of this as telling Google: “Yes, this project is allowed to request Workspace APIs.” Step 4: Create the Service Account Now we create the identity that will actually pull logs. Go to IAM & Admin → Service Accounts Click Create Service Account Set: Name: workspace-log-reader Click Create and Continue Skip role assignment (no GCP roles required) Click Done At this point, the service account exists—but it can’t do anything yet. Step 5: Enable Domain-Wide Delegation (critical step) This is where most people miss a checkbox and everything breaks. Click the service account you just created Open the Details tab Click Show domain-wide delegation Check Enable Google Workspace Domain-wide Delegation Save Now copy the Client ID. You’ll need it immediately. This setting allows the service account to act on behalf of users in your domain—but only for scopes you explicitly allow. Step 6: Trust the Service Account in Google Workspace Now we jump back to Workspace. Go to Google Admin Console 👉 https://admin.google.com Navigate to: Security → API controls → Domain-wide delegation Click Add new Enter: Client ID: (from the service account) OAuth scopes: https://www.googleapis.com/auth/admin.reports.audit.readonly https://www.googleapis.com/auth/admin.reports.usage.readonly Click Authorize This is the trust handshake between Workspace and Google Cloud. Without this step, every API call will be denied. Step 7: Create and download a Service Account key You’ll need credentials for your script or tool. Go back to Google Cloud Console → Service Accounts Select your service account Open Keys → Add key → Create new key Choose JSON Download the file ⚠️ This JSON file is effectively a password. Store it securely. Step 8: Using the Service Account to pull logs When you actually query the Admin SDK API: Authenticate using the JSON key Enable domain-wide delegation Impersonate a Workspace admin user (very important) Example conceptually: Delegated user: admin@yourdomain.com API: Admin SDK – Reports API Logs belong to the domain, not the service account, which is why impersonation is required. The actual tool this walkthrough is building toward (biggest gap) You don't have to build this script yourself — there's already a purpose-built one for exactly this job. The script is gws-get-logs.py, written by Megan Roddie, in the https://github.com/dlcowen/sansfor509 (under the GWS folder). It handles the JSON-key auth, the delegation, and the impersonation for you — you just point it at your setup. Before running it, create a config.json alongside the script: { "creds_path": "./credentials.json", "delegated_creds": "admin@yourdomain.com", "output_path": "./output" } creds_path — the JSON key file you downloaded in Step 7 delegated_creds — the admin account being impersonated (the same idea as Step 8 above) output_path — where the collected logs land Run it, then check the output folder — you should see one JSON file per log type: admin_logs.json, login_logs.json, user_logs.json, calendar_logs.json, chat_logs.json, and drive_logs.json. That's Admin, Login, User, Calendar, Chat, and Drive audit logs, all pulled in one pass. Why investigators like this method Once this is set up, you can: Pull all Workspace logs in JSON Avoid UI export limits Build repeatable, defensible evidence collection Feed logs directly into SIEMs, timelines, or DFIR tooling ----------------------------------------------------------------------------------------------------------- Final thought Yes, the setup feels painful the first time. But once it’s done, you’ve essentially built a forensic-grade log pipeline for Google Workspace—and that’s incredibly powerful during incident response. After the first run, most analysts say the same thing: “Oh… that actually wasn’t that bad.” ------------------------------------------------------------Dean----------------------------------------

If you’ve ever had to investigate a Google Workspace account takeover, you already know one thing: it’s not about one log — it’s about connecting multiple logs and understanding how Google thinks. The Two Logs You Must Know When it comes to tracking user behavior (and especially account compromise), there are four core log types you’ll always come back to: Admin log events User log events (Previously it was seperated into two logs) (Login Audit Log + User Accounts Audit Log) Security Reports Think of these as different camera angles. One log alone never tells the full story — but together, they usually do. Log Retention: The 6-Month Trap By default, Google Workspace retains these logs for six months. And here’s the annoying part: You cannot extend retention inside the Admin Console There is no “keep logs longer” checkbox If you want long-term visibility (and you absolutely should), the only solution is to: Export logs to Google Cloud Logging Configure extended retention there Google Cloud allows log storage for up to 10 years, which is a lifesaver for compliance, threat hunting, and delayed investigations. Log Lag Time: Why “Too Early” Is a Real Problem One thing that trips up a lot of investigators is log availability delay. Each of logs has a different lag time before events become searchable. And that lag time should be treated as the minimum waiting period, not a guarantee. So if you search immediately after an incident and think, “This doesn’t make sense…” …it probably doesn’t — yet. Rule of thumb: Never rely on searches run shorter than the documented log lag times. Some events just arrive late. Real lag times Here's roughly what those lag times look like in practice: Admin Audit Log — a few minutes Login Audit Log — a few hours User Accounts Audit Log — tens of minutes Security Reports — 1 to 3 days (yes, this is the fourth log — easy to forget it exists, but it's official and it's slow) And this isn't theoretical — during testing, password resets have shown up in these logs before the login event that triggered them, when someone searched inside the lag window. If the timeline looks impossible, it's not broken. You just searched too early. Admin log events: Start Here for Admin Compromise The Admin Log evets is your go-to log for anything that happens inside the Google Admin Console. It tracks: Admin actions Configuration changes Policy updates Organization-wide modifications If you suspect an admin account compromise, don’t overthink it — this is the first log you check. It tells you exactly what changes were made and by which admin account. User log events: Where the Action Is The User log events is where most account takeover investigations spend their time. This log captures: Successful and failed logins Re-authentication prompts MFA changes Security challenges triggered by Google It doesn’t just tell you that someone logged in — it tells you how, why, and under what conditions. The "often empty" caveat for User Accounts events One quirk worth knowing before you panic: the User Accounts side of this log is often completely empty — sometimes zero events across a full six-month window. That's not a broken pipeline or a missed export. It's genuinely common when there hasn't been much high-risk account activity, and it's called out explicitly as "not an uncommon scenario." An empty log here is a data point, not a failure. Understanding Login Types (This Matters) Each login event includes a Login Type, which explains how the authentication happened. Some common ones you’ll see: Google Password – Standard username + password login ReAuth – Google forced the user to re-authenticate SAML – Login via SSO Exchange – OAuth or existing token-based session Unknown – Login occurred using an unidentified method (always worth a closer look ) When you’re hunting suspicious activity, “Unknown” and unusual patterns in login types are often gold. Warning Icons = Pay Attention In the User log events, some events show a warning icon. These usually indicate unusual or suspicious logins, such as: New IP addresses Unfamiliar locations Behavior Google flags as risky Instead of scrolling endlessly, a smart approach is to hunt by event type. Login Event Types Investigators Care About Here are some high-value event types you should always keep an eye on: 2-step verification disabled – Big red flag Account password change – Especially if unexpected Failed login – Useful for brute-force patterns Government-backed attack – Google explicitly flagged a known threat actor Leaked password – Password found in credential dumps Suspicious login – Unusual characteristics detected Out-of-domain email forwarding enabled – Common data exfil trick User suspended – Often triggered by Google due to abuse or compromise Event-type nuances worth knowing before you escalate A few of these deserve a second look before you read too much into them: Login challenge vs. Login verification — these look similar but mean different things. A challenge means Google already thought the sign-in was suspicious; a verification means it didn't, but asked anyway. Know which one you're looking at before you escalate. Logout events are always logged with Login Type = Google Password, even if the original session started via Exchange, ReAuth, SAML, or Unknown. Don't read that as the user switching auth methods mid-session — it's just how Google records logout. User suspended isn't one event, it's three: suspended for spam, suspended for spam relay, and suspended for suspicious activity. Which variant you get tells you a lot about what Google actually detected. Important note: Some details (like why a login failed) are not visible in the Admin Console and require pulling logs via the API. OAuth Let’s be honest — OAuth sounds way more complicated than it actually is. At its core, OAuth is just a permission slip. Instead of giving an app your username and password (which is a terrible idea), OAuth lets you say: “Hey, this app can read my emails, but nothing else.” That’s it. That’s the magic. So What Exactly Is OAuth? OAuth is an authorization mechanism — not authentication. It doesn’t prove who you are It proves what an app is allowed to do When an application wants to access your data through an API (emails, Drive files, contacts, calendar, etc.), OAuth sits in the middle and asks you for permission. If you say yes, the app gets a token. That token is like a digital key that says: “This app is allowed to access these specific things, on behalf of this user.” No password sharing. No repeated logins. Cleaner and safer. Why OAuth Exists (And Why Everyone Uses It) Imagine if every app you used asked for your Gmail password. Nightmare. OAuth solves a few big problems: You don’t have to re-authenticate every time Apps never see your actual credentials Access can be limited (scope-based) Tokens can be revoked anytime That’s why OAuth is everywhere — Google Workspace, Microsoft, GitHub, Slack, Twitter (X), basically everything modern. OAuth in Google Workspace (What Users Actually See) Inside Google Workspace, OAuth usually shows up as that familiar screen: “This app wants access to your : GmailDrive filesContacts” That list? Those are called scopes. Scopes define exactly what the app can touch. Nothing more. Once the user clicks Allow, Google generates an OAuth token, and the app can start making API calls using that token. Important point: OAuth is enabled by default in Google Workspace unless admins restrict it. Where Things Go Wrong: OAuth Abuse Here’s the problem — OAuth is secure, but humans are optimistic. Threat actors figured out something clever: “Why steal passwords when we can just ask nicely?” The Basic OAuth Attack Chain Attacker creates a malicious app Victim gets a phishing email with a link Victim clicks → sees a legit Google OAuth screen Victim clicks Allow Attacker now has access — no password needed No malware. No credential theft. No MFA bypass required. Just consent. The two real OAuth attacks (replacing the generic chain with named case studies) This isn't a hypothetical either — it's happened at scale, twice, and both cases are worth knowing by name: In May 2017, a fake app calling itself "Google Docs" (not the real thing) spread through Gmail like a worm — a contact would send you what looked like a shared doc, you'd click it, authorize "Google Docs" for email and contacts access, and then your own contacts would get the exact same email from you. Google eventually banned the app and mass-revoked tokens, but not before it spread fast. Between 2015 and 2016, the threat actor group Fancy Bear (APT28) ran a more targeted version against political parties: a phishing email claiming a security system had flagged suspicious sign-ins, urging the target to install "Google Defender" for protection. The "app" requested OAuth access to the victim's Gmail and their entire Drive — full read access to email and shared documents, no password required. Both attacks prove the same point: OAuth abuse doesn't need a stolen credential, a malware payload, or an MFA bypass. It just needs one click on Allow. Why Threat Actors Love OAuth OAuth attacks are attractive because: No credentials to steal MFA doesn’t stop it Looks completely legitimate Uses official Google infrastructure And the scariest part? OAuth does NOT give attackers more access than the user already has — but that’s usually more than enough. Detecting OAuth Abuse in Google Workspace Google Workspace actually gives us solid visibility here. OAuth Log events These logs show: Which user authorized which app Application ID Scopes granted API activity performed using the token Technically, this log records three distinct types of entries, and it helps to know which is which: Activity (an API call the app made, using its token) Authorize (the moment a user granted the app access), Revoke (access being pulled, whether by the user, an admin, or a password change) One limitation worth flagging before you go looking for it: Activity entries in this log are only recorded on Enterprise and Education editions — on other editions, you may simply not have this visibility at all, and that's a licensing gap, not a missing export. If you pull these logs via API, you get even more gold: Source of the request Which Workspace service was accessed How much data was returned Client type and product bucket This is huge for investigations and retroactive analysis. Killing the Access: Revoking OAuth Tokens A few important things defenders should know: Changing a user’s password revokes OAuth tokens IMAP tokens can take up to an hour to expire Admins can: Review all third-party apps See who authorized them Block apps org-wide In the Admin Console, you can quickly identify sketchy apps by: Unusual scopes Non-verified apps Excessive permissions Block once — and it impacts the whole org. The Big Takeaway OAuth isn’t insecure. Blind trust is. OAuth attacks succeed because: Users trust the Google consent screen App names look legitimate No passwords are involved (so alarms don’t go off) Defenders need to: Monitor Token Audit Logs Restrict third-party apps Educate users that “Allow” is a powerful action Because sometimes, clicking Allow is worse than typing your password. ------------------------------------------------------------------------------------------------------------- Final Thoughts If there’s one takeaway here, it’s this: Understand: What each log shows When data becomes available Which events actually matter Once you get comfortable with these logs, Google Workspace investigations stop feeling messy — and start feeling methodical. ------------------------------------------Dean--------------------------------------------------------------

(How logs are generated, why they matter, and how investigators actually use them) Big picture Before you can analyze logs, you need to understand where logs even come from in Google Cloud. Google Cloud generates logs in two fundamental ways: Platform-level Audit Logs→ Logs generated automatically by Google Cloud itself Application / workload logs→ Logs generated by what you run (VMs, apps, network traffic, etc.) From a DFIR point of view: Audit Logs tell you “what changed in the cloud control plane” Application logs tell you “what happened inside workloads” You almost always need both during an incident. ------------------------------------------------------------------------------------------------ Platform Audit Logs – what Google logs for you Audit Logs record actions like: Who logged in Who created / modified / deleted resources Who changed IAM permissions Which actions were denied by policy These logs are generated by Google Cloud, not by your apps. Why this matters in practice Audit Logs are: Hard for attackers to tamper with Centralized Often the first place you detect compromise If IAM abuse, privilege escalation, or lateral movement happens —👉 Audit Logs are your ground truth ------------------------------------------------------------------------------------------------ Enforcing logging at the Organization level Concept Google Cloud lets you enforce audit logging: At Organization At Folder At Project Logging rules flow top-down. If something is enforced at the Org level: Projects cannot disable it They can only add more logging ------------------------------------------------------------------------------------------------ Example (real-world) An organization enforces: Admin Write logs at Org level This means: Every admin-level change is logged No project owner can turn it off Even compromised Owner accounts still generate logs This is critical for post-compromise investigations. ------------------------------------------------------------------------------------------------ Audit log types you must understand (not all logs are equal) Required Log Bucket (most important) These logs: Cannot be disabled Stored 400 days Free High-value security events Includes: Admin Activity Logs System Events Enterprise Group Audit Logs Login events Access Transparency logs 👉 From an investigator’s perspective: This is your “black box recorder.” ------------------------------------------------------------------------------------------------ Default Log Bucket These logs: Often capture denied actions Stored 30 days for free Cost money if retained longer Why denied logs matter: Brute-force attempts Repeated IAM failures Early-stage recon attempts In real incidents: The successful login might be one event —the denied attempts tell the full story. ------------------------------------------------------------------------------------------------ Exempted Users – useful but dangerous Concept Google Cloud allows exempted users: Their actions are not logged Why this exists Some service accounts generate massive noise Cost and signal-to-noise ratio matter DFIR risk If misused: An attacker may intentionally target exempted accounts Blind spots are created in audit trails 👉 As an investigator, always ask: “Which accounts are exempted from logging — and why?” ------------------------------------------------------------------------------------------------ Cost model (what actually costs money) Google Cloud logging costs are based on two things, not one: 1. Log Ingestion Logs entering the logging system 50 GiB per project is free Required logs do NOT count toward this 2. Log Storage How long logs are retained Default bucket: 30 days free Required bucket: 400 days free Key insight: You don’t usually pay because you log too much You pay because you retain logs too long. For incident response: Short retention = cheaper Long retention = better historical visibility This is a risk vs cost decision, not just technical. ------------------------------------------------------------------------------------------------ Accessing logs – where investigations actually happen Log Explorer (Google’s built-in “SIEM-lite”) Google significantly upgraded Log Explorer, and today it behaves very much like: Splunk ELK Chronicle-style query systems How investigators use Log Explorer 1. Scope Defines where you’re searching: Project Folder Entire Org (if permissions allow) In real investigations: Start broad → narrow down Scope mistakes = missed evidence 2. Query Builder Uses structured, SQL-like queries. You typically hunt for: IAM permission changes Service account usage API key creation actAs events Login anomalies Very similar mental model to: ELK Splunk SOF-ELK timelines 3. Results Each log entry: Is collapsed by default Must be expanded for full context Important fields often hidden until expanded: Caller IP Principal email Authentication method Resource name Permission granted or denied ------------------------------------------------------------------------------------------------ Investigator mindset shift (important) Traditional IR: “Logs come from servers” Cloud IR: “Logs come from the control plane” If you only look at VM logs and ignore Audit Logs: You miss IAM abuse You miss lateral movement You miss Org takeover paths ------------------------------------------------------------------------------------------------ Query Builder – what it really does Concept Log Explorer’s Query Builder is not magic. It’s simply a UI-assisted way of writing structured queries against JSON logs. You: Pick a resource type Narrow it down using resource labels Add fields relevant to that resource Set a time range The UI then converts your selections into the underlying query syntax. 👉 Important mindset: Log Explorer will only search what you explicitly ask for, and only inside the selected scope. Practical implication (DFIR) If: You forget to include the right resource type Or your scope is wrong (wrong project / folder) Or your time range is too small Then events do not “not exist” — you just didn’t ask correctly. This is a very common cloud IR mistake. ------------------------------------------------------------------------------------------------ Resource-based searching (why it feels backward) Concept Google Cloud logs are resource-centric, not user-centric. So instead of: “Show me everything user X did” You often start with: “Show me everything that happened to resource Y” Example1 : resource.type="gcs_bucket" resource.labels.bucket_name="securitz resource.labels.location="us-east1" Example2 : resource.type="audited_resource" resource.labels.method="google.login.LoginService.riskySensitiveActionAllowed" resource.labels.service="login.googleapis.com" Why this is powerful for investigations This matches Google Cloud’s IAM model: Permissions are attached to resources Members are granted access by the resource owner So if a bucket, VM, or project was abused:👉 Start with the resource, then pivot to the actor. ------------------------------------------------------------------------------------------------ Time range – not just a filter Concept Time range is part of the query logic, not just a display option. You can: Search seconds, minutes, hours, days Use custom absolute ranges (incident window) Investigation workflow A common IR pattern: Start with a tight time window (alert timestamp) Validate suspicious activity Expand the time window without changing the query Watch how activity builds up before and after the incident Log Explorer keeps previously matched results visible when expanding time — this helps you see progression, not just isolated events. ------------------------------------------------------------------------------------------------ Results view – summary vs evidence Concept The default results pane: Shows a condensed summary Hides most fields This is intentional — logs are JSON and very verbose. Investigator reality The real evidence is always inside the expanded event: principalEmail callerIp userAgent timestamp serviceName methodName You rarely care about every field. You care about: Who, from where, did what, to which resource, and when. ------------------------------------------------------------------------------------------------ JSON structure – why queries feel “long” Concept Google Cloud logs are structured JSON. That means: Fields are nested You must specify full paths Example: resource.labels.method="google.login.LoginService.riskySensitiveActionAllowed" Practical tip (this saves time) If you already found a relevant event: Expand it Click a field (e.g., principalEmail) Select “Show matching entries” Log Explorer automatically: Adds the correct field path Adds the value Updates your query This avoids syntax mistakes and speeds up hunting. ------------------------------------------------------------------------------------------------ How investigators actually build queries You rarely write one “perfect” query upfront. Real workflow: Broad resource-based query Identify suspicious event Pivot using fields from that event Narrow down to: User IP Service account API method Expand time window Repeat This is iterative threat hunting, not static searching. ------------------------------------------------------------------------------------------------ Logging pipeline – what happens behind the scenes Conceptual flow Every log follows the same path: Generated (platform or workload) Sent to Google Cloud Logging API Passed through Log Sinks Either: Stored Exported Dropped This happens before you ever see the log in Log Explorer. ------------------------------------------------------------------------------------------------ Log Sinks – control points (and blind spots) Concept Log Sinks exist at: Project level Organization level They decide: Which logs are kept Which logs are excluded Where logs are sent (storage, SIEM, Pub/Sub) DFIR relevance If a log does not appear: It may have been excluded It may have been routed elsewhere It may have been dropped by design During investigations, always confirm: Sink configuration Exclusion rules Retention settings Missing logs ≠ attacker tampering (most of the time). ------------------------------------------------------------------------------------------------ Heading: Log Routing: Where Sinks Actually Send Logs Every log that hits the Cloud Logging API gets pushed to every sink that exists — not just the one you're thinking about. Three sink/bucket pairs exist out of the box: · Required sink → Required bucket · Default sink → Default bucket · Any sink you create → a bucket you choose (or BigQuery / Cloud Storage / Pub/Sub / an external platform) Locked vs. unlocked (this matters more than it sounds) _Required bucket: locked. 400 days, free, cannot be modified or disabled — not even by an org admin. _Default bucket: unlocked. 30 days by default, but an admin can change the retention — anywhere from 1 day up to 3,650 days (~10 years). 👉 “Locked” and “unlocked” are Google's actual terms for this in the console — if you see a bucket flagged unlocked, know its retention could've been changed at any point, and check when. Two things people assume wrong: Log Buckets are not Storage Buckets. You can't browse them in Cloud Storage — they're only accessible through the Log Storage / Log Explorer section. Different system, similarly-named. You can't bulk-export logs already sitting in a bucket after the fact. Log Explorer will let you export up to 10,000 records from a search, but if you need everything, it has to be flowing through a sink to an export destination from the start. One more option worth knowing about, even if you never touch it: log buckets can be encrypted with your own keys (CMEK) instead of Google's default encryption. Not something you'll usually need for an investigation, but worth knowing if a client asks whether their logs are protected with customer-managed keys. ------------------------------------------------------------------------------------------------ Exclusions – useful but dangerous Concept Exclusions reduce noise: Ignore repetitive service account actions Reduce cost Improve signal quality Investigation risk Over-aggressive exclusions can: Remove early attacker recon Hide lateral movement Remove failed attempts that give context Good practice: Exclude volume, not security-relevant behavior. ------------------------------------------------------------------------------------------------ Heading: Creating a Log Sink (Hands-On) If you need logs sent somewhere specific — a bucket you control, BigQuery, Pub/Sub, or off-platform entirely — you create your own sink. Four things you set: A unique sink name (unique within the project) A destination (where the logs actually go) An inclusion query — SQL-like syntax, same as Log Explorer — that decides what this sink actually captures An optional exclusion query, to filter out noise that matched the inclusion query but that you don't want stored Example Say you want a dedicated sink that captures every IAM policy change in a project — useful to have sitting in its own bucket with its own retention, separate from the noise of everything else: protoPayload.methodName="SetIamPolicy" Every log event that hits the sink gets checked against that query. Match, and it flows to wherever you pointed the sink. No match, it's ignored by this sink — though it may still hit others. 👉 Remember: every sink evaluates independently. The same log event can pass through one sink and get filtered out of another. ------------------------------------------------------------------------------------------------ Heading: gcloud CLI: Pulling Logs Without the UI Everything above works through the console, but you can do it all from the CLI too — useful when you're scripting collection across a lot of projects at once. gcloud logging buckets list Shows every bucket in scope — location, retention days, and critically, whether it's locked. This is the fastest way to confirm a bucket's actual retention without digging through console menus. gcloud logging read 'timestamp<="2026-08-16T00:00:00Z" AND timestamp>="2020-08-16T00:00:00Z"' --format="json" > all_gcp_logs.json Pulls logs for a time range. Two things worth knowing before you run this: by default, logging read returns a limited number of results and isn't JSON-formatted — you have to explicitly force both full results and JSON output, or you'll walk away thinking you got everything when you didn't. ------------------------------------------------------------------------------------------------ Heading: The Multi-Project Blind Spot (and Exporting Logs Out) The scenario Picture this: a threat actor didn't just compromise one VM. They're across Compute Engine, Cloud Storage, and containers, spread across multiple projects — and you're not confident the IAM setup was solid to begin with. Single-project Log Explorer scoping doesn't cut it here. You need visibility across the whole organization, fast, before you even know exactly what's been touched — exactly the situation “start broad, then narrow down” was built for. Getting logs out to where you can actually work When you need to centralize logs from Google Cloud into a SIEM or log aggregator (SOF-ELK, Splunk, whatever your team runs), the mechanism is Pub/Sub — a sink routes to a Pub/Sub Topic, and a Subscription attached to that topic is what actually delivers the messages onward. Two delivery methods: Push — Pub/Sub sends the message to your aggregator via HTTP/S POST. Preferred when you can, but it means opening a port on your receiving end. Pull — your aggregator asks Pub/Sub for messages via the API instead. No inbound port needed, useful if you can't expose one. A few knobs worth knowing about on the subscription side: messages can be retained even after successful delivery (useful for replay, costs extra), subscriptions can be set to expire after a period of inactivity, and undeliverable messages can be routed to a dead-letter queue instead of just disappearing. One thing that trips people up the first time: pushed messages arrive wrapped in a Pub/Sub envelope, not as your raw log. The actual log content is base64-encoded inside a data field, alongside Pub/Sub's own messageId, publishTime, and subscription metadata — decode data before you try to parse it as your log. ------------------------------------------------------------------------------------------------ Takeaway Google Cloud Log Explorer queries are built around resource-centric, JSON-structured logs that require investigators to think differently than traditional user-based logging models. By starting with affected resources, iteratively refining queries using nested fields, and understanding how time ranges and log sinks influence visibility, analysts can reconstruct attacker behavior across projects and organizational boundaries. Effective investigations rely not on writing perfect queries upfront, but on pivoting through relevant fields and understanding where logs may be excluded or redirected within the logging pipeline. ------------------------------------------Dean------------------------------------------------

In this new series, we'll be diving deep into investigation and forensics within Google Workspace (the Google ecosystem). So tighten your seatbelt—let's go! When diving into cloud forensics—especially in Google Workspace—there’s a lot more to unravel than just user credentials or login timestamps. One of the most overlooked but crucial areas is how permissions are managed within the environment. The Edition Problem: Why Your Investigation Might Come Up Empty Before you go hunting through OUs and Groups, there's a more basic question that decides whether you'll find anything at all: what Workspace edition is this org actually on? Two orgs can have identical OU structures and completely different forensic visibility — because visibility isn't set by structure, it's set by license. Here's what actually changes between tiers: Starter / Essential — no Drive audit logging, no audit reports at all. Entire categories of evidence simply don't exist to find on this tier. Standard — full audit logging across services — this is the baseline every other article in this series assumes. Plus — everything in Standard, plus Vault and Device Management. Forensic Tip: Check the edition before you check anything else. A blank search result on Standard means nothing happened. The same blank result on Starter might just mean nothing was ever logged. ---------------------------------------------------------------------------------------------------------- Now that you know what tier you're working with, let's break down two key building blocks of Google Workspace that matter a lot when you're investigating suspicious account behavior or responding to an incident: 👉 Organizational Units (OUs) 👉 Groups Why OUs and Groups Matter in Forensics Google Workspace has its own authentication and identity system, sure—but when you're trying to understand how and why a user had access to certain data or features, you need to look beyond just login logs. That’s where Organizational Units (OUs) and Groups come in. These two are the backbone of how permissions are structured and managed in Workspace. And guess what? They can be used independently, so knowing how each works is essential for tracing how permissions are applied—or misapplied. ---------------------------------------------------------------------------------------------------------- Quick Primer: Footprint, Responsibility, and How We Got Here Before we get into OUs and Groups, three quick things worth knowing about the platform itself — they don't show up in permission structures, but they shape what evidence you can expect to find and where. Where your data actually lives. By default, a new Google Workspace instance uses a globally distributed data region — your org's data could be sitting in any of Google's data centers across North America, South America, Europe, or Asia. You can lock it down to just the US or just Europe, but only if your license supports it. Either way, there's no functional difference in the product itself — this is a data-residency question, not a feature question. Who's responsible for what. Google uses the same shared-responsibility model you'd expect from any cloud provider. The good news for DFIR: Google gives you logging as soon as you're on the platform, regardless of which *aaS tier you're using. The catch: the more useful security and protection features — the stuff that actually helps you detect and prevent, not just investigate after the fact — only show up once you're using PaaS/SaaS-level features. A short version of how we got here, because knowing the platform's age tells you how mature (or sparse) documentation and third-party tooling might be for a given feature: · 👉 2006 — First released as Google Apps: Gmail, Talk, Calendar, Page Creator · 👉 2012 — Google Vault launches · 👉 2014 — Drive launches; enterprise offering renamed “Google for Work” · 👉 2016 — Rebranded to G Suite · 👉 2021 — Rebranded again to Google Workspace; Hangouts becomes Chat None of this changes how you investigate — but it explains why you'll still see G Suite, Google Apps, and Workspace all floating around in older documentation, tickets, and Stack Overflow answers. ---------------------------------------------------------------------------------------------------------- The Edition Problem: Why Your Logging Might Not Exist Here's the gap that actually matters: not every Google Workspace edition gives you the same evidence. Starter / Essential — limited logging on user and admin actions, no Drive logging at all, no audit reports, limited data protection. Standard — complete audit logging. This is the tier where real DFIR visibility starts. Plus — everything in Standard, plus Vault, device management (Business only), LDAP, and higher user/size limits. Translation: if you walk into an investigation and the org is on Starter or Essential, don't assume Drive activity logs were rotated out or deleted. They may never have existed in the first place — that tier doesn't generate them. Check the edition before you conclude anything about missing evidence. (Education edition has its own separate tiering that's more generous at the lower end than the commercial editions — worth checking separately if you're investigating a school or university tenant.) ---------------------------------------------------------------------------------------------------------- Organizational Units (OUs): Think Department Bins Let’s start with Organizational Units. Think of them like folders or containers that you put users into based on department, location, or job role. Every user account must belong to one—and only one—OU. From an investigation perspective, this helps narrow things down: If you know the user’s OU, you don’t have to search other units. Also, OUs can be nested—meaning you can have child units inside parent ones. So a user could be in a sub-OU deep in the hierarchy, but they’ll still inherit permissions from the OUs above them. This inheritance is something to watch closely during an investigation. Two practical things worth knowing when you're actually in the Admin Console: you'll find the whole OU structure under Directory → Organizational Units — that's where you go to see the full parent/child hierarchy (it won't show you which users are in which OU, just the structure itself). And if a user account is created without an OU explicitly assigned, it doesn't end up ownerless — it drops into the org's default OU automatically. One more inheritance nuance: A child OU inherits its parent's settings by default, but the moment someone sets a custom setting directly on that child OU, it decouples from the parent for that setting — future changes to the parent won't retroactively touch it. If you're trying to explain why two OUs under the same parent behave differently, that's usually why. Forensic Tip: OU Inheritance Can Create Hidden Access If a user is in a deeply nested OU, don’t forget to trace all the inherited settings and permissions. You might find that access was granted not directly, but from higher up the chain ---------------------------------------------------------------------------------------------------------- One User, Many Groups — And Even More Permissions Unlike OUs where a user can only belong to one, a single user account in Google Workspace can be part of multiple groups at the same time. But here’s where things get interesting—and complicated: Groups can contain other groups. So if User A is in Group X, and Group X is inside Group Y, then User A indirectly inherits all permissions from Group Y too. This is what we call inherited groups, and it’s an important concept for anyone doing incident response or auditing permissions. Forensic Insight: Inherited Groups = Inherited Risk Let’s say you have a group called "IT Users". It’s a member of both the "Log Access" and "Vault Access" groups. That means everyone in IT Users also inherits access to logs and vault data—even if that wasn’t the original intention. This kind of setup is handy for streamlining permissions—but it can also accidentally over-provision users, which is something DFIR teams always need to watch out for. Using Groups Smartly Groups aren’t just for permissions. You can also use them for: Feature access control Mailing lists Managing shared resources (like calendars, drives, etc.) Think of it like Microsoft’s Security Groups and Distribution Groups in Active Directory. In large organizations, using groups makes onboarding and permissioning way easier. You can just drop a new user into the right group and boom—they’ve got the correct access in seconds. But this simplicity can be dangerous if you don’t track what each group actually allows. Real-World Use: Google Drive Sharing Groups Imagine this : You’ve got three groups set up for Google Drive sharing: Internal Sharing Only Sharing to Trusted Domains Open Sharing (Anyone outside the org) During a data breach, it’s so much easier to identify which group allowed risky sharing if these types of groups are clearly defined. You could simply yank a user out of the “Open Sharing” group, and the exfiltration risk goes down instantly. This logic applies not just to Drive—but to all services in Google Workspace. group Roles: Owner, Manager, Member Every group in Workspace has three roles: Owner: Full control—can add/remove members, change settings, etc. Manager: Can manage members, sometimes limited in changing settings Member: Just a regular part of the group Forensics Tip: The group owner isn’t always an IT admin. It could be a team lead, project manager, or anyone else. That means non-IT staff could be controlling access to sensitive groups, so always check who owns what. Heading: The Permission Matrix Groups Actually Control Owner/Manager/Member tells you who has control — but Groups also let you control who can do what, independently for each action. Two axes: Who: Group Owners, Group Managers, Group Members, the whole organization, or anyone external to the organization. What: contact the owners, view members, view conversations, make posts, add new members. Mix and match those and you get a permission matrix, not just a role. A group could let the whole org view posts but restrict adding new members to Owners only — or open posting to anyone external while locking down membership visibility. When you're investigating a group tied to an incident, check the actual matrix, not just who holds which role — the role tells you who's in charge, the matrix tells you what everyone else was actually allowed to do. Heading: Forensic Insight: Dynamic Groups Add Members Without a Human Doing It Not every group gets built by someone clicking “Add Member.” Google Workspace also supports dynamic groups, where membership is assigned automatically based on attributes already sitting in a user's account — department, job title, location, whatever the admin configured as the rule. For an investigation, this matters because group membership history won't always trace back to a deliberate admin action. Someone could pick up group access — and whatever permissions ride along with it — just by having their job title changed, with nobody consciously deciding to grant them anything. Using Admin Console to Inspect Groups Google makes it a bit easier to investigate with features like Inspect Groups, available in the Admin Console. With this, you can: See all groups assigned to a user Know whether group membership is direct or inherited Check which users belong to a specific group For example:You might find that Akash is directly added to the “Vault Access” group but indirectly added to the “IT Users” group through another group membership. That tells you how permissions were layered onto Akash's account. Feature Alert: Only in Enterprise Plus or Cloud Identity Premium Here’s the catch: This level of detail—especially Inspect Groups and dynamic visibility—requires either: Enterprise Plus edition of Google Workspace, or The Cloud Identity Premium Edition add-on If you’re in a budget-conscious environment, the add-on gives you solid forensic capabilities without needing the full Enterprise tier. ---------------------------------------------------------------------------------------------------------- Final Thoughts: Groups = Power and Risk Groups are incredibly powerful, but also easy to overlook during forensic reviews. Always: Map direct vs inherited permissions Watch for non-admin group owners Audit who’s in which group and why Use features like Inspect Groups if your license supports it Getting this right can help you detect, contain, and respond to incidents faster and smarter. ----------------------------------------------Dean----------------------------------------------------- Stay with me—things are going to get more interesting in the upcoming articles!

You've seen Vault mentioned twice already in this series — once as "the evidence room" in the Drive exposure article, once in passing in the Takeout article. Both times, the same line: Vault is what you use when you actually need the file. That's true. It's also not the whole story. Vault is a legal hold and eDiscovery tool first, and an investigator's tool second. Those two jobs pull in different directions, and if you don't know where the seams are, you'll assume Vault sees more than it does. What Vault Actually Is (And Isn't) What it is: a search, hold, and export layer that sits across Workspace data — built for legal discovery and compliance, not day-to-day IR. What it isn't: a second copy of every log you've already seen in the Investigation Tool. Vault doesn't give you actor/IP/visibility fields — it gives you the underlying content. If Drive log events are the timeline and Vault is the evidence room, Vault's door only opens for a few rooms in the building — not the whole floor. Creating a Matter Before you can run a single search, Vault makes you create a matter. Think of it as a case folder — every search, hold, and export you run lives inside it, tied to whatever investigation you opened it for. Two things worth knowing before your first one: One matter per case. Running two unrelated investigations at once? Don't dump both into the same matter — you'll lose the ability to cleanly hand off or close one without touching the other. Matters are shareable. Add another investigator to the matter and they see everything you've run inside it — searches, holds, exports. Useful once a case grows past one person; also means you should know who else has access before you assume a matter is private to you. Search Scope: What Vault Can Actually See This is the part people get wrong most often. Vault doesn't cover "Workspace" as a whole — it covers specific services, and coverage has grown over time, so don't assume last year's scope is this year's. Gmail — mail, including drafts and chats migrated into Gmail Drive — files in My Drive and Shared Drives, including version history Google Chat — spaces and direct messages, if history is turned on for the space Groups — messages sent to Google Groups Meet — recordings saved to Drive, plus limited attendance/metadata Notice what's missing: Calendar isn't a Vault search target. If your case hinges on "what meeting was this person actually in," Vault won't answer that — you're back to Admin Console reports. Gmail Search Syntax Inside Vault The basic search fields get you far, but the real power is in the “Terms” field, which takes Gmail-style search operators: Operator What it does to: / from: / cc: / replyto: Match a specific address in that header in:inbox / in:spam Match a system folder label: Match a specific label is:read / is:unread Match read state larger: / smaller: Match by message size, e.g. larger:5M filename: Match an attachment name has:attachment Match messages with an attachment AND / OR / NOT (or -) Combine or exclude terms * Wildcard AROUND n Match terms within n words of each other Chain them together and you can get specific fast. Investigating a suspected data handoff to a personal account, for example: from:dean@ AND to:*@gmail.com AND has:attachment AND larger: 2M Two limits worth knowing before you rely on this: English words and numbers only — no punctuation. Vault can't search for a literal string with symbols in it. That includes href links, so if you're hunting a phishing email by the URL embedded in it, a Vault text search won't find it — pull the raw message and check manually instead. A blank space between terms means AND, not OR. Don't assume unrelated words in a query are being treated as alternatives. One more thing before you trust a timestamp in your results: Vault displays times in your browser's local system time, not UTC. If you've got investigators in different time zones — or your own machine's clock is off — normalize before you build a timeline. Holds: The Part That Actually Preserves Data A hold is what stops data from being purged — by retention rules, by a user hitting delete, by anything. No hold, no guarantee the thing you need still exists tomorrow. Two ways to scope a hold Org-wide / OU-wide hold: applied across an organizational unit. Broad, blunt, usually for a standing legal or compliance requirement rather than a single investigation. Custodian-based hold: applied to specific accounts — the people actually relevant to a matter. This is the one you'll use mid-investigation: put a hold on the account(s) involved the moment you suspect something, before you've finished scoping the case. Speed matters here. A hold placed after a user has already deleted and purged something doesn't reach back in time — it only stops what happens next. Vault Retention vs. Native Workspace Retention This is where two systems with similar names quietly disagree with each other. Native Workspace retention rules (set in Admin Console) control how long data lives by default — and they purge on schedule. A Vault hold overrides that purge for anything it covers, for as long as the hold is active. Remove the hold, and native retention rules resume — including deleting anything that was only being kept alive by the hold. Two more things that trip people up here: Vault retention and Email Audit Log retention are not the same clock. An email can fall out of the Audit Log's retention window while still being fully searchable in Vault, or the other way around — check both before you conclude something isn't recoverable. And a brand-new Workspace org has no default retention rules configured at all — someone has to turn them on. Until they do, mail just follows the plain mailbox-to-Trash timeline below: nothing purged early, but nothing extra held either. That second 30-day window is the one people miss. Once an email clears a user's Trash, most people assume it's gone. It isn't — Google holds it for another 30 days by default, and during that window it's still searchable in Vault and still shows up in your other investigation tools. After that, it's gone unless a hold was already in place. Export Formats — and Why Chain of Custody Actually Matters Here Once you've searched and scoped what you need, Vault exports it — but the export itself is often the piece that ends up in front of legal or a third party, so how it's packaged matters. Mail exports as MBOX or PST — you choose the format when you set up the export. MBOX is what most investigators pick; it's the more versatile of the two and easier to work with in other tools. Drive exports keep native file formats (Docs, Sheets, etc.) rather than flattening everything to PDF. Chat and Groups export as structured message files, not screenshots. Exports can be limited to the EU or US data region if you have a requirement about where the data physically lands — outside those two, your only option is “no preference.” Exports aren't permanent: you get 15 days to download before Google auto-deletes them, with unlimited re-downloads inside that window but no extension once it closes. Confidential Mode Gmail content is included — Vault can search and export messages sent with Confidential Mode turned on, it isn't a blind spot. The part people skip: an export isn't one file, it's five. .zip — the actual email content (MBOX or PST, whichever you picked), attachments included metadata.csv — per-message summary (IDs, labels, from/to/cc/bcc, sent/received dates) metadata.xml — the same summary data, plus the exact search query used to generate the export results-count.csv — a simple count of messages and accounts in the export checksum file — hashes for the other four files, so you can prove nothing was altered or lost in transit Download all five. Skipping straight to the .zip is the single most common mistake — metadata.xml has your search query on record, and the checksum file backs your chain-of-custody claim later. One more wrinkle: timestamps inside metadata.xml don't reflect the sender's time zone, the recipient's, or the admin's — they reflect whichever mail server (MTA) last touched the message before it landed in the mailbox. Don't assume it's local to anyone in the case. Common Investigator Mistakes Assuming Vault = full visibility. It's access to content, not an audit trail — you still need the Investigation Tool for who/when/how. Placing a hold after the fact and assuming it recovers what's already gone. It doesn't — it only preserves what's still there. Forgetting Calendar (and a few other services) simply aren't in scope, then wondering why a search comes back empty. Treating the export as done without saving the manifest — then having no way to answer "how do we know this wasn't modified" later. Final Thoughts Vault earns its "evidence room" nickname — but a room only holds what someone thought to lock the door on, and only for the wing of the building it was built to cover. Vault preserves content; it doesn't log activity — pair it with the Investigation Tool, not instead of it Holds stop future loss, not past loss — speed matters Search scope has gaps (Calendar is the one people forget) The export isn't done until the manifest is saved with it -------------------------------------------------------Dean-----------------------------------------

Google Workspace is an integral part of many organizations, providing essential tools for communication and collaboration. However, when it comes to forensic investigations, compliance, and eDiscovery, knowing how to extract and analyze data from Google Workspace is crucial. ---------------------------------------------------------------------------------------------------------- Data Extraction in Google Workspace There are three primary ways to extract data from Google Workspace: Admin Console Data Export Available in all paid Google Workspace tiers. Exports data for all users or specific accounts. Covers a wide range of data, including Gmail, Drive, Calendar, Contacts, Chat, Tasks, Voice data, and even Vault-retained items. Data is first archived in cloud storage, from where it can be selectively downloaded. Similar to Google Takeout but allows administrators to manage multiple user exports efficiently. Google Vault (For eDiscovery and Compliance) Included in Business and Enterprise editions or available as an add-on. A powerful tool for data retention, searching, and exporting beyond standard exports. The only method to access Gmail’s “Confidential Mode” messages. Supports retention policies, litigation holds, and compliance-related data archiving. Can search across Gmail, Drive, Shared Drives, Google Groups, Chat messages, Meet recordings, and Google Voice data. Provides search and filtering based on keywords, dates, and user accounts. Gmail API (For Custom Data Collection) Allows programmatic access to Gmail data. Used by third-party email collection tools or for building custom forensic scripts. Grants access to Gmail History Records, which track message additions, deletions, and label changes. Useful for tracking actions like message deletion, marking emails as spam, or email forwarding. ---------------------------------------------------------------------------------------------------------- Google Vault: A Deep Dive into eDiscovery Google Vault is a must-use tool for organizations needing compliance and legal hold capabilities. It goes beyond basic exports, offering: Advanced Search and Filtering: Using search operators similar to Gmail. Comprehensive Export Options: Supports PST and MBOX formats, with additional metadata in XML and CSV formats. Confidential Mode Access: Unlike the Gmail API, Vault retains the full content of confidential messages. Draft Message Versioning: Every version of a draft is saved and available in Vault for 30 days, even if deleted by the user. Retention and Hold Policies: Enforceable for different data types to ensure compliance with organizational policies. Critical Pro-Tip: If a user account is deleted, all associated data is permanently removed from Vault. Instead, suspend user accounts to retain data while restricting access ---------------------------------------------------------------------------------------------------------- Audit Logging and Investigations One of the most powerful aspects of Google Workspace is its audit logs, which help track user activity and identify security incidents. Google provides different types of logs, including: Log Name Purpose Data Retention Admin Log Actions taken by Google Workspace administrators Account, event description, date, IP address 6 months User Log All login activity, including webmail and admin console Account, log-in type, date, IP address 6 months Email Log Search Search emails sent and received by the organization Email headers (no content searches) 30 Days OAuth Log Authorizations by email clients and mobile devices User, Application Name, Scope, IP address, date 6 months User Reports App Usage Consolidated view of user status and account activity Usage of Gmail, Drive, Storage, and External Apps 6 months Log Retention Periods: Most logs are retained for six months, except for Email Log Search, which is available for 30 days. Organizations using Google Workspace Enterprise can store logs indefinitely in Google BigQuery or export them to a SIEM for extended retention. ---------------------------------------------------------------------------------------------------------- Leveraging Open-Source Tools for Google Workspace Investigations ALFA on GitHub: invictus-ir/ALFA Will try to create a article on this tool in coming future(Stay tuned) ---------------------------------------------------------------------------------------------------------- Email Header and Metadata Investigations Google Workspace allows email header searches for messages from the last 30 days. Investigators can extract metadata such as: Sender & recipient email addresses. Subject lines & timestamps. Message ID and client IP address. Mail delivery tracking (e.g., failures, spam filtering). Matched Rules that flag emails for objectionable content, PII, or compliance violations. Key Limitation: Email headers do not contain email message content (only metadata). For full content analysis, investigators must rely on Google Vault or exports. ---------------------------------------------------------------------------------------------------------- Final Thoughts Google Workspace provides robust tools for forensic investigations, data compliance, and eDiscovery. By leveraging Admin Console exports, Google Vault, Gmail API, and audit logs, organizations can effectively extract, search, and preserve critical data. To ensure thorough investigations: Use Google Vault for advanced eDiscovery. Leverage audit logs for security analysis. Export logs to BigQuery or a SIEM for extended analysis. Suspend accounts instead of deleting them to retain forensic evidence. Understanding these mechanisms ensures that organizations can respond effectively to incidents while maintaining compliance with legal and regulatory requirements. --------------------------------------------Dean--------------------------------------

This final article covers the advanced scenarios in AWS incident response — acquiring memory from cloud instances, forensicating containers, understanding the metadata service attack, and making the decision between in-cloud and on-premises analysis. By the end of this series, you have a complete picture of how to investigate any AWS incident from first alert to evidence report. In-Cloud vs. On-Premises Analysis ▸ Recommendation: Always analyse in-cloud first. Download to on-prem only when regulation requires it or tooling demands it. The fundamental question in cloud IR: should you download evidence to an on-prem forensics lab, or do the analysis in the cloud? The answer has shifted dramatically in recent years. Doing analysis in-cloud is almost always faster, cheaper, and more practical. In-cloud analysis: No egress bandwidth costs, no waiting for terabytes to download, spin up forensic EC2 instance in same region in minutes, use cloud-native tooling On-prem analysis: Required when regulation demands it, when your tooling only runs on-prem, or when you need physical custody of evidence 💡 IR Tip: Start in-cloud. Build a DFIR AMI (Amazon Machine Image) with your tools pre-installed — Volatility, Autopsy, YARA, log parsers — and keep it ready to deploy in any region within minutes. -------------------------------------------------------------------------------------------------- Building a DFIR EC2 Image The DFIR AMI is your forensic workstation template in the cloud. Build it once, store it, deploy it in any region within 5 minutes. Build process: (1) Launch a clean EC2 instance (Ubuntu 22.04 LTS recommended), (2) Install all your forensic tools: Volatility 3, AVML, bulk_extractor, log2timeline/plaso, YARA, Autopsy CLI tools, aws cli, python3 libraries, (3) Create an AMI from this instance (EC2 → Actions → Create Image), (4) Copy the AMI to all regions you might respond in (AMIs are regional). -------------------------------------------------------------------------------------------------- Systems Manager (SSM) — Remote Access Without a Bastion AWS Systems Manager Session Manager gives you shell access to EC2 instances without needing SSH, a bastion host, or open port 22. It tunnels the session through AWS's own infrastructure. This is critical for IR because: you don't need to open any network ports on the compromised instance, all session activity is logged to CloudTrail and optionally to S3, you can access instances in private subnets with no internet access. Requirements: SSM Agent must be installed (pre-installed on most AWS AMIs since 2022), the instance must have an IAM role with the AmazonSSMManagedInstanceCore policy. CLI access: aws ssm start-session --target i-0yourinstanceid -------------------------------------------------------------------------------------------------- Linux Memory Acquisition — AVML Memory acquisition from cloud Linux instances requires a different approach than on-premise. The old tool (Margarita Shotgun, which used a kernel module to dump memory) stopped working on modern kernels. https://margaritashotgun.readthedocs.io/en/latest/user_guide.html The current standard for AWS Linux memory acquisition is AVML (Acquire Volatile Memory for Linux), developed by Microsoft and supported by AWS. https://github.com/microsoft/avml ⚡ Update (2024): AVML is the current recommended tool for Linux memory acquisition in AWS. It works as a userspace tool — no kernel module required. It reads /dev/mem or process maps directly. It's included in the AWS Forensics AMI reference implementations. AVML acquisition process via SSM: (1) Connect via SSM: aws ssm start-session --target i-0compromisedinstance, (2) Install AVML if not present: curl -L https://github.com/microsoft/avml/releases/latest/download/avml -o avml && chmod +x avml, (3) Acquire memory: sudo ./avml memory.lime, (4) Copy to S3: aws s3 cp memory.lime s3://your-dfir-bucket/case-2024-047/. The .lime extension is important — Volatility expects LiME format for analysis. 💡 IR Tip: After AVML acquisition, validate the image with Volatility before you do anything else: python3 vol.py -f memory.lime banners. This confirms the memory image is valid and tells you the exact kernel version, which you'll need to find the right Volatility profile. -------------------------------------------------------------------------------------------------- Windows Memory Acquisition in the Cloud Windows EC2 memory acquisition is more complex. Options: (1) WinPMem — open source, runs as a driver, requires admin access. Works on Windows 10/Server 2019 and later. (2) Magnet RAM Capture — free tool from Magnet Forensics, GUI-based, reliable. (3) F-Response — enterprise remote memory acquisition, requires a F-Response service account. For cloud environments, WinPMem via SSM Run Command is the most practical: aws ssm send-command --instance-ids i-0yourwindowsinstance --document-name 'AWS-RunPowerShellScript' --parameters commands=['winpmem_mini_x64_rc2.exe memory.raw'] -------------------------------------------------------------------------------------------------- Container Forensics — ECS and EKS AWS has two managed container services: ECS (Elastic Container Service) — AWS's own container orchestrator, simpler to manage. EKS (Elastic Kubernetes Service) — managed Kubernetes, more complex, industry standard. Container forensics challenge: containers are ephemeral. When a container task is stopped, the container filesystem is gone. Evidence preservation for containers: (1) Don't stop the container — use kubectl exec (EKS) or aws ecs execute-command (ECS) to get a shell into the running container, (2) Dump the container filesystem while running: docker export containerid > container_fs.tar, (3) If running on EC2-backed ECS/EKS (not Fargate), capture the host EC2 memory and disk as well — the container filesystem exists on the host. 💡 IR Tip: For EKS investigations, deploy a forensic pod into the same namespace as the suspicious pod. Use kubectl cp to extract files from the suspect container. This avoids touching the host and minimises your investigation footprint. -------------------------------------------------------------------------------------------------- The IMDS Attack — Stealing Cloud Credentials via the Metadata Service The Instance Metadata Service (IMDS) is an internal AWS service available to every EC2 instance at the fixed IP address 169.254.169.254. It provides the instance with information about itself — instance ID, region, AMI ID, and critically, the temporary credentials for the IAM role attached to the instance. The attack: if a web application running on an EC2 instance is vulnerable to SSRF (Server-Side Request Forgery), an attacker can make the application fetch http://169.254.169.254/latest/meta-data/iam/security-credentials/RoleName — this returns a valid, active ASIA-prefixed temporary credential for the instance's IAM role. The `attacker can then use these credentials from their own machine to make AWS API calls as that instance. Example IMDS credential response: { 'AccessKeyId': 'ASIA...', 'SecretAccessKey': 'xxxxxxx', 'Token': 'IQoJb...', 'Expiration': '2024-03-15T14:00:00Z' }. These credentials expire, but the attacker can refresh them as long as they have SSRF access. In CloudTrail, IMDS credential abuse looks like: API calls with ASIA-prefixed key ID, from an external IP (not an internal AWS IP), for a role that normally only makes internal API calls. The combination of ASIA token + unexpected source IP = stolen IMDS credentials. IMDSv2 — The Fix for IMDS Credential Theft AWS introduced IMDSv2 to mitigate the SSRF-to-IMDS attack. IMDSv2 requires a two-step process: (1) PUT request to get a session token: curl -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' http://169.254.169.254/latest/api/token, (2) Use that token in subsequent GET requests: curl -H 'X-aws-ec2-metadata-token: TOKEN' http://169.254.169.254/latest/meta-data/. SSRF attacks typically can only make GET requests (not PUT with custom headers), so the PUT requirement blocks the attack. ⚡ Update (2024): AWS now enforces IMDSv2 by default on all NEW EC2 instances launched after October 2023. However, existing instances that were launched before this change still use IMDSv1 by default unless explicitly updated. Always check: aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId, MetadataOptions.HttpTokens]' -- if HttpTokens is 'optional', the instance still allows IMDSv1. AWS IR Readiness Checklist CloudTrail trail configured at org level with data events enabled VPC flow logs enabled for all production VPCs S3 server access logs enabled for sensitive buckets Load balancer access logs enabled Route 53 resolver query logging enabled GuardDuty enabled at org level AWS Detective enabled and connected to GuardDuty DFIR AMI built and copied to all regions Evidence S3 bucket configured with WORM and encryption IR Lambda functions and Step Functions workflows deployed and tested IMDSv2 enforced on all EC2 instances (HttpTokens = required) SSM Agent enabled on all instances for agentless remote access -------------------------------------------------------------------------------------------------------- Series Complete From understanding AWS organisations and IAM, through CloudTrail investigation and hunting, networking, S3, automated detection with GuardDuty and Detective, event-driven automation, and in-cloud IR — you now have a complete framework for investigating any AWS incident. This is a complete AWS Incident Response (IR) article. I’ve tried my best to cover the topic based on my knowledge, experience, and learning. There may still be some mistakes or areas that could be improved, but I hope you find it useful and enjoy reading it. If you notice anything incorrect or have suggestions for improvement, please feel free to message me or let me know. I’d really appreciate your feedback! 🙌 ----------------------------------------------------------------------------------------------------

One of the most powerful capabilities AWS gives you for incident response is the ability to automate it. Unlike on-premise environments where IR automation requires complex SOAR platforms, AWS has native services that can trigger forensic actions the moment a threat is detected — no human in the loop required for the initial containment and evidence preservation steps. Lambda — Serverless Functions for DFIR Automation AWS Lambda lets you run code without managing any servers. You write a function, upload it, and AWS executes it in response to triggers — API calls, scheduled events, S3 uploads, GuardDuty findings, whatever you connect to it. For DFIR, Lambda is your execution engine. It runs the Python or Node.js code that does the actual work: isolating an EC2 instance, taking a snapshot, copying evidence to a secure bucket, notifying the security team. Critical constraint: Lambda has a maximum execution time of 15 minutes per invocation. For actions that take longer than 15 minutes — like imaging a large disk — you need to break the work into multiple Lambda functions chained together, or use a different compute option like AWS Fargate for longer-running tasks. Lambda pricing: First 1 million requests per month free, then $0.20 per million requests. Compute time: $0.0000166667 per GB-second. For IR automation, the cost is essentially negligible. Step Functions — Orchestrating Multi-Step DFIR Workflows AWS Step Functions is a workflow orchestration service. You define a state machine — a visual flowchart of steps — and Step Functions manages the execution, error handling, retries, and branching logic between your Lambda functions. For complex DFIR workflows, Step Functions is essential. A typical forensic collection workflow might look like: (1) Receive GuardDuty finding → (2) Identify affected EC2 instance → (3) Check if instance is already isolated → (4a) If yes: take snapshot → (4b) If no: isolate instance, then take snapshot → (5) Copy snapshot to forensic account → (6) Notify IR team → (7) Create case ticket. Each of these steps is a Lambda function. Step Functions ties them together, handles failures, and retries steps that fail transiently. EventBridge — The Routing Layer AWS EventBridge is the event bus and routing service. It receives events from AWS services (GuardDuty findings, CloudTrail events, EC2 state changes) and routes them to targets (Lambda functions, Step Functions, SQS queues, SNS topics) based on rules you define. Example EventBridge rule — trigger isolation on any HIGH or CRITICAL GuardDuty finding of type UnauthorizedAccess: { "source": ["aws.guardduty"], "detail-type": ["GuardDuty Finding"], "detail": { "severity": [{"numeric": [">=", 7]}], "type": [{"prefix": "UnauthorizedAccess"}] } } This rule pattern fires the moment GuardDuty generates any UnauthorizedAccess finding with severity >= 7 (HIGH). EventBridge passes the full finding JSON to your Lambda function or Step Functions workflow as the trigger payload. Lambda Function: Isolating a Compromised EC2 Instance Here's a Python Lambda function that isolates a compromised EC2 instance by replacing its security group with a 'deny all' group: import boto3 def lambda_handler(event, context): ec2 = boto3.client('ec2') instance_id = event['detail']['resource']['instanceDetails']['instanceId'] region = event['detail']['region'] # Create isolation security group vpc_id = ec2.describe_instances( InstanceIds=[instance_id] )['Reservations'][0]['Instances'][0]['VpcId'] sg_response = ec2.create_security_group( Description='IR Isolation - No Inbound/Outbound', GroupName=f'ir-isolation-{instance_id}', VpcId=vpc_id ) isolation_sg_id = sg_response['GroupId'] # Revoke all default outbound (allow all) rule ec2.revoke_security_group_egress( GroupId=isolation_sg_id, IpPermissions=[{'IpProtocol': '-1', 'IpRanges': [{'CidrIp': '0.0.0.0/0'}]}] ) # Replace instance's security groups with isolation group ec2.modify_instance_attribute( InstanceId=instance_id, Groups=[isolation_sg_id] ) return {'status': 'isolated', 'instance': instance_id, 'isolation_sg': isolation_sg_id} 💡 IR Tip: Isolation via security group replacement is the safest containment method in AWS — it doesn't terminate the instance (preserving volatile memory state), it doesn't stop the instance (preserving running processes), and it's reversible by putting the original security groups back. Lambda Function: Enabling VPC Flow Logs Mid-Incident If flow logs weren't enabled before the incident, you can turn them on automatically in response to a GuardDuty finding: import boto3 def lambda_handler(event, context): ec2 = boto3.client('ec2') instance_id = event['detail']['resource']['instanceDetails']['instanceId'] vpc_id = ec2.describe_instances( InstanceIds=[instance_id] )['Reservations'][0]['Instances'][0]['VpcId'] ec2.create_flow_logs( ResourceIds=[vpc_id], ResourceType='VPC', TrafficType='ALL', LogDestinationType='s3', LogDestination='arn:aws:s3:::your-dfir-evidence-bucket/flow-logs/' ) return {'status': 'flow_logs_enabled', 'vpc_id': vpc_id} Full Forensic Disk Collection Automation ▸ EventBridge Rule → Lambda Trigger → Step Functions → isolate + snapshot + copy to forensic bucket Putting it all together — a complete automated forensic collection chain: (1) GuardDuty HIGH finding → (2) EventBridge rule matches → (3) Triggers Step Functions state machine → (4) Step 1: Isolate EC2 (Lambda) → (5) Step 2: Tag instance with case number and timestamp → (6) Step 3: Snapshot all attached EBS volumes → (7) Step 4: Copy snapshots to forensic account (cross-account snapshot sharing) → (8) Step 5: Enable VPC flow logs if not already running → (9) Step 6: Send SNS notification to IR team → (10) Step 7: Create Jira/ServiceNow ticket via webhook. The entire chain from GuardDuty finding to snapshot complete typically takes 3-8 minutes for a standard-sized instance. By the time your IR team picks up the alert, the instance is already isolated and the disk image is already preserved. What's Next The final article in this series — covers in-cloud IR for complex scenarios: Linux memory acquisition with AVML, Windows memory in the cloud, container forensics for ECS and EKS, the IMDS metadata service attack, and how IMDSv2 mitigates it. ---------------------------------------------------------Dean------------------------------------------

So far we've covered the raw evidence sources — CloudTrail, VPC flow logs, S3 logs. Now we look at the tools AWS provides that sit on top of all that telemetry and do the detection work for you. These three services form AWS's native automated detection and investigation stack. Choosing the Right Tool for the Job ▸ CloudTrail Insights (statistical anomaly) → GuardDuty (ML threat detection) → AWS Detective (investigation platform) None of these replace manual CloudTrail analysis. They are the triage layer — they surface the needles so you can investigate. CloudTrail Insights — Statistical Anomaly Detection CloudTrail Insights is an optional feature you enable on a CloudTrail trail. It analyses your management events and builds a statistical model of 'normal' API activity volumes. When activity spikes significantly above or below those baselines, it generates an Insight event. Example: your environment normally sees about 50 IAM API calls per hour. A credential stuffing attack or compromised admin account suddenly generates 2,400 IAM calls in 10 minutes. CloudTrail Insights will flag this as anomalous — even if every individual API call is legitimate in isolation. CloudTrail Insights generates two types of findings: API call rate anomalies — unusual volume of calls to a specific API in a short window. API error rate anomalies — unusual number of access denied or throttled errors, which often indicates reconnaissance or brute forcing. 💡 IR Tip: CloudTrail Insights is particularly good at catching automated attacks — scripts running API enumeration, bulk key creation, or mass resource modification. Human attackers doing things manually are harder for statistical models to catch because their rate is lower. This is where GuardDuty's ML approach adds value. Pricing: $0.35 per 100,000 events analysed. For most mid-sized environments, this is less than $5/month. GuardDuty — Intelligent Threat Detection GuardDuty is AWS's primary threat detection service. It continuously analyses CloudTrail management events, VPC flow logs, and DNS logs across your entire AWS account and organisation, using a combination of machine learning models and threat intelligence feeds to identify attack patterns. Unlike CloudTrail Insights which detects statistical anomalies, GuardDuty knows what attacks look like. Its threat intelligence includes: known malicious IP addresses (Tor nodes, known C2 servers, commercial threat intel feeds), known crypto mining pool addresses, known command-and-control domains. GuardDuty finding categories: UnauthorizedAccess (account takeover, credential abuse), Recon (reconnaissance API calls), Trojan (malware indicators), CryptoCurrency (mining activity), Backdoor (reverse shells, C2), Stealth (log tampering, trail deletion), Discovery (resource enumeration), Exfiltration (data movement), PenTest (common pentesting tools detected). Severity levels: LOW (informational, usually benign), MEDIUM (investigate when you have time), HIGH (investigate immediately — active threat), CRITICAL (rare, immediate action required). Pricing: based on data volume processed. Roughly $1-5/month for small accounts, scales to hundreds of dollars for large enterprise environments. There's a 30-day free trial when you first enable it. ⚡ Update (2024): GuardDuty has significantly expanded its detector types. New in 2023-2024: EC2 Runtime Monitoring (agent-based detection of suspicious OS-level activity on EC2), RDS Protection (anomalous database login detection), Lambda Protection (threat detection in serverless functions), EKS Runtime Monitoring (container escape, suspicious kubectl activity). These require explicit opt-in and some require deploying an agent. AWS Detective — Investigation Platform AWS Detective is a separate service that builds a graph-based analysis layer on top of CloudTrail logs, VPC flow logs, and GuardDuty findings. Where GuardDuty tells you 'this looks suspicious', Detective helps you investigate why it's suspicious and what the full scope is. Detective continuously ingests and normalises your environment data (up to a year of history) and uses graph ML to map relationships between: IAM entities (users, roles, accounts), API calls, IP addresses, resource access patterns, and GuardDuty findings. Key Detective capabilities: Entity profiles (full activity history for any IAM user, role, or IP), Behaviour profiles (what's normal for this entity — detection surface mapped visually), Finding groups (clustering related GuardDuty findings into a single investigation thread), Cross-account investigation (if you have AWS Organizations, Detective can show activity across all accounts in your org). 💡 IR Tip: Start every GuardDuty finding investigation in Detective. The entity profile for the affected IAM user or role will immediately show you the full scope — every API call, every IP, every resource touched — across your retention window. This context is what turns a 2-hour investigation into a 20-minute one. When to Use Which Tool CloudTrail Insights → 'Alert me when API call volumes spike unexpectedly' GuardDuty → 'Detect known attack patterns, malicious IPs, and anomalous behaviours continuously' AWS Detective → 'Investigate a specific alert and understand the full scope and context' Manual CloudTrail querying (Athena) → 'Hunt for something specific that automated tools haven't surfaced' These tools are not mutually exclusive — the ideal AWS security posture uses all of them in combination. --------------------------------------------------------------------------------------------------------- What's Next Next Article covers event-driven DFIR automation — Lambda, Step Functions, and EventBridge. How to build automated response workflows that kick off the moment GuardDuty or CloudTrail Insights triggers an alert. -------------------------------------------------Dean-----------------------------------------

S3 (Simple Storage Service) is the backbone of AWS storage. It's where CloudTrail logs land, where flow logs are stored, where application data lives, and where attackers look for sensitive information. Understanding S3 from an IR perspective means understanding both how attacks happen against S3, and how S3 itself becomes your central evidence repository. S3 Access Controls — The Basics S3 access is controlled at multiple layers. Every S3 bucket is private by default — no public access unless explicitly granted. Access can be granted through: IAM policies (attaches to users/roles, grants access to specific buckets), Bucket policies (attaches to the bucket itself, can grant access to specific IAM accounts or make the bucket public) ACLs (older mechanism, avoid these for anything new). The infamous 'public S3 bucket' misconfiguration happens when an administrator enables public access either through a bucket policy with Principal: '*' (wildcard), or by disabling the Block Public Access setting — which is the AWS-level override that prevents any accidental public exposure. 💡 IR Tip: In any investigation involving data exposure, check the S3 Block Public Access settings first — it's a single boolean setting at the account level and the bucket level. If either is off, public exposure is possible. AWS Console → S3 → Block Public Access settings for this account. The Two Different S3 Log Types CloudTrail data events = API access (who called GetObject via SDK/CLI) S3 server access = HTTP/web access (presigned URLs, browser downloads) ⚠️ During exfiltration investigations — check BOTH log types. API exfiltration → CloudTrail. Presigned URL/web exfiltration → S3 server access logs. This is one of the most commonly misunderstood things in AWS forensics. There are two completely separate logging mechanisms for S3 access, and they capture different things: 1. CloudTrail Data Events for S3 CloudTrail data events record API-level access to S3. This means any access through: AWS CLI (aws s3 cp), AWS SDK (boto3, AWS SDK for Java), Cross-account role access, Lambda functions accessing S3. These logs are in CloudTrail format — you get the full userIdentity, sourceIPAddress, and event details. 2. S3 Server Access Logs S3 server access logs record HTTP-level access to S3. This means any access through: a web browser directly (using presigned URLs), HTTP GET requests to public bucket objects, any access that goes through the S3 HTTPS endpoint without AWS credentials. These logs look more like web server logs — you get the source IP, HTTP method, object key, response code, and bytes transferred. 💡 IR Tip: If you're investigating S3 data exfiltration and only looking at CloudTrail — you may miss exfiltration that happened via presigned URLs or public bucket access. Check both log types. They're stored separately and enabled separately. --------------------------------------------------------------------------------------------------------- Transfer Acceleration S3 Transfer Acceleration is an optional feature that routes S3 uploads and downloads through AWS CloudFront edge locations worldwide, rather than directly to the S3 bucket's region. This dramatically speeds up transfers for users far from the bucket's region. From an IR perspective: if Transfer Acceleration is enabled, the source IPs in your S3 logs will be CloudFront edge node IPs — not the original requester's IP. The X-Forwarded-For header in server access logs will show the real source IP. Using Athena to Search S3 Logs at Scale ▸ S3 Bucket → Glue Crawler → Glue Data Catalog → Athena → SQL Query When you have weeks or months of CloudTrail logs in S3, you need a way to search them efficiently. AWS Athena is a serverless SQL query engine that reads directly from S3 — you pay per query, not per hour of infrastructure. The pipeline: (1) CloudTrail logs land in S3, (2) AWS Glue Crawler scans the S3 bucket and creates a table schema in the Glue Data Catalog, (3) Athena uses that schema to run SQL queries against the raw S3 files. You don't need a database, a server, or ETL pipelines. Example Athena query to find all ConsoleLogin events from a specific IP: SELECT eventtime, useridentity.arn, sourceipaddress, responseelements FROM cloudtrail_logs WHERE eventname = 'ConsoleLogin' AND sourceipaddress = '91.200.14.77' ORDER BY eventtime DESC LIMIT 100; Athena pricing: $5 per TB of data scanned. Partition your CloudTrail bucket by year/month/day to drastically reduce scan costs — a query scoped to one week might only scan a few GB instead of a year's worth. ⚡ Update (2024): CloudTrail Lake now provides a managed alternative to the S3+Glue+Athena pipeline. It includes a built-in query interface with pre-configured schemas. For new deployments, this is often simpler. For existing environments with S3-based trails, Athena remains the standard approach. Setting Up Your Evidence S3 Bucket Your forensic evidence bucket needs specific protections: (1) Enable versioning — so evidence files can't be overwritten without keeping the history. (2) Enable MFA Delete — requires MFA to permanently delete any object. (3) Enable object lock (WORM) — Write Once Read Many, prevents any modification or deletion for a defined retention period. (4) Enable server-side encryption — SSE-KMS with your own KMS key. (5) Enable access logging — so you can see who has accessed your evidence bucket. ----------------------------------------------------------------------------------------------------------- What's Next Next article covers the automated detection stack — CloudTrail Insights, GuardDuty, and AWS Detective. These are the tools that sit on top of everything we've discussed and surface attacks automatically, so you're not manually querying CloudTrail for every threat. --------------------------------------------Dean-----------------------------------------------------