Network forensics in AWS is not like network forensics on-premise. There's no physical switch to mirror traffic from. But AWS gives you powerful tools — VPC flow logs, load balancer logs, DNS resolver logs — that tell you a huge amount about what was happening on the network. VPC — Virtual Private Cloud A VPC is a logical network boundary inside AWS — like a network switch. Everything inside can communicate at Layer 3, nothing gets in or out without explicit routing and firewall rules. Inside a VPC: Subnets (public or private), EC2 instances, Network Security Groups, Internet Gateways, Load Balancers, WAFs. Public subnet = has default route to internet gateway. Private subnet = no route to internet. Key investigative question: was the compromised system in a public subnet, or did the attacker pivot into a private subnet? That pivot is your lateral movement evidence in VPC flow logs. 💡 IR Tip: Check whether the compromised EC2 instance has a public IP attached. If so, check security group rules — what ports are open to 0.0.0.0/0 (the world)? This tells you the initial attack surface. CIDR block and whether each VPC has flow logs enabled (Very Important for monitoring) Load Balancers — The Investigation Blind Spot ▸ ALB terminates connection, creates new one → source IP is ALB, not attacker | Enable X-Forwarded-For header NLB (Network Load Balancer) — Layer 4, preserves original source IP Classic Load Balancer — Layer 4+7, proxies connection, masks source IP ALB (Application Load Balancer) — Layer 7 HTTP/HTTPS, also proxies, masks source IP If your compromised system sits behind an ALB or Classic LB, EC2 and VPC flow logs only show the load balancer's private IP — not the attacker's real IP. You MUST check load balancer access logs (not enabled by default) for the real source IP. 💡 IR Tip: Always check if an ALB or Classic LB is in the path. If yes, load balancer access logs are mandatory evidence — they're the only place that records the original external IP address. For IR or any security Team, Make sure these are enabled Reading VPC Flow Log Fields ▸ version srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status Example entry (modified): srcaddr=178.62.90.41 (attacker), dstaddr=10.0.1.45 (internal EC2), dstport=443, protocol=6 (TCP), bytes=7820, action=ACCEPT. Key fields: srcaddr (who connected), dstaddr (what they hit), dstport (service), bytes (data volume), action (ACCEPT=allowed, REJECT=blocked). 💡 IR Tip: Flow logs record sessions every 10 minutes by default. For bulk data transfers, sum bytes across all entries for that source-destination pair. The protocol + source + destination combination is your grouping key. Flow logs can be delivered to S3 ($0.023/GB) or CloudWatch Logs ($0.05/GB). Enable them at VPC level for broadest coverage. You can turn them on mid-incident with no downtime — AWS collects them out-of-band at the hypervisor level. You can create flow logs here, Sorry I don´t have one to show Route 53 — DNS Logging in AWS Route 53 offers two logging capabilities: (1) DNS Zone Query Logging — logs all DNS queries against your hosted zones. (2) Route 53 Resolver Query Logging — logs every outbound DNS query from any instance within a VPC. The second is the most powerful for IR. Resolver query logs are invaluable for: identifying C2 domain communications from compromised instances, tracking DNS-based exfiltration, and reconstructing what domains an implant communicated with. Even if the EC2 instance has been terminated, DNS logs persist. 💡 IR Tip: If the compromised EC2 was already terminated and there are no endpoint logs, Route 53 resolver query logs may be one of your only remaining evidence sources. Always check if they were enabled. ⚡ Update (2024): Route 53 resolver query logs now include the query response (IP addresses returned) and query type. You can now see not just what domains were queried but what IPs they resolved to — useful for correlating DNS events with VPC flow logs. --------------------------------------------------------------------------------------------------------- What's Next Article 6 covers S3 buckets — the central storage layer for almost everything in AWS, frequent target of attack, and cornerstone of your log collection strategy. ------------------------------------------Dean---------------------------------------------------

When you're investigating a compromised cloud environment, the virtual machine is often where the real action happened. In AWS, virtual machines are EC2 instances, and their storage is handled by EBS volumes. Understanding how these work — and what evidence they leave in CloudTrail — is essential for any cloud IR investigation. ----------------------------------------------------------------------------------------------------- EC2 Instance Types — Why They Matter for IR T-series (t3, t4g) — general purpose, burstable. Most common. The free-tier instance type. M-series — balanced compute/memory for standard workloads. C-series — compute-heavy, more CPUs than RAM. P-series and G-series — GPU-equipped instances. These are the ones crypto miners love. I/D/H-series — storage-optimised, fast or dense storage. 💡 IR Tip: When looking for crypto mining abuse, search CloudTrail for RunInstances events where instanceType contains P or G prefix (p3, p4, g4dn, g5). Attackers use the largest available variant for maximum hash rate. Regions Matter — Don't Miss Evidence in Other Regions IAM is global, but EC2 instances and EBS volumes are regional. A common investigation mistake: querying the wrong region and concluding no evidence exists when it's actually sitting in us-west-2 while you're looking in us-east-1. CloudTrail logs always contain the awsRegion field for the event. When you find a suspicious RunInstances event, note the region — that's where the instance was created and where you need to look. Attackers frequently choose non-default regions specifically because teams overlook them. ----------------------------------------------------------------------------------------------------- What an EC2 Instance Creation Looks Like in CloudTrail RunInstances event (modified values): userIdentity.arn = arn:aws:iam::417823659031:user/devops-deploy, eventSource = ec2.amazonaws.com, awsRegion = eu-west-2, sourceIPAddress = 91.200.14.77, requestParameters.instanceType = g4dn.12xlarge, requestParameters.keyName = temp-access-key What stands out: g4dn.12xlarge is a GPU instance (48 vCPUs, 192GB RAM) — nobody launches this for a web app. Unusual region (eu-west-2 vs org's normal us-east-1). Source IP 91.200.14.77 should be checked for VPN/TOR. Key name 'temp-access-key' is suspicious naming. EBS — Elastic Block Store: AWS Storage Volumes Every EC2 instance has at least one EBS volume — this is the virtual hard drive (like VMDK in VMware, VHD in Hyper-V). EBS volume types: gp3/gp2 (general purpose SSD, gp3 is now default), io2/io1 (provisioned IOPS for databases). ⚡ Update (2024): AWS changed the default EBS volume type from gp2 to gp3 in 2023. gp3 is 20% cheaper and provides 3,000 IOPS baseline without extra cost. This doesn't change anything forensically — both types behave identically from an evidence standpoint. ----------------------------------------------------------------------------------------------------- Snapshots — Your Most Valuable Evidence Preservation Tool EBS snapshots are incremental point-in-time images stored in S3 (managed by AWS). First snapshot = everything. Subsequent snapshots = only changed blocks (like Windows Shadow Copies). You can take a snapshot of a running instance without interrupting it. Create a snapshot via CLI: aws ec2 create-snapshot --volume-id vol-0ab1234567890cdef --description 'DFIR preservation - Case #2026-047' --region eu-west-2 ----------------------------------------------------------------------------------------------------- Coldsnap — Downloading a Snapshot Without Restoring It AWS provides Coldsnap, which uses the DirectBlockAccess API to download a snapshot directly as a disk image: coldsnap --region eu-west-2 download snap-0abc123456def7890 evidence.dd Critical caveat: if this is a differential snapshot (prior snapshots exist), the downloaded image only contains changed blocks and cannot be directly analysed as a full disk image. For full analysis, restore the snapshot to a volume first, then image it. Full two-step restore process: (1) aws ec2 create-volume --availability-zone eu-west-2a --snapshot-id snap-0abc123456def7890 (2) aws ec2 attach-volume --volume-id vol-0newvolumeid --instance-id i-0yourDFIRinstance --device /dev/sdf 💡 IR Tip: The two-step restore-and-mount approach is the most reliable for complete forensic analysis. The Coldsnap approach is faster but only safe when you know the snapshot is a full (first) snapshot, not incremental. ----------------------------------------------------------------------------------------------------- EFS — The Shared Network Storage (Don't Overlook This) EFS (Elastic File Store) is a network-attached NFS share — multiple EC2 instances can read and write simultaneously. Used for shared config, log aggregation, and distributed app data. From an IR standpoint: data exfiltration via EFS shows up in VPC flow logs, not OS logs of individual instances. EFS gets a private IP in the VPC — you need flow logs to see what was accessed. Snapshot Pricing First 1GB free per snapshot Beyond that: $0.05/GB/month Fast Snapshot Restore: $0.75/hour per enabled availability zone Coldsnap DirectBlockAccess reads: $0.003 per 1,000 blocks read ----------------------------------------------------------------------------------------------------- What's Next Next Article moves to the network layer — VPCs, subnets, internet gateways, load balancers, VPC flow logs, and Route 53 DNS logging. -----------------------------------------------Dean--------------------------------------------

Understanding the CloudTrail format is one thing. Actually using it to find attacker activity is another. n this article, we walk through the most common CloudTrail investigation scenarios — tracking who logged into the console, detecting new API keys being created, finding evidence of exposed keys being abused, and running proactive threat hunts. Scenario 1: Tracking Console Logins Console logins leave a specific CloudTrail footprint. The event is called ConsoleLogin from eventSource: signin.amazonaws.com. Key fields: ARN (which account) eventTime (UTC), sourceIPAddress (browser IP), userAgent (browser), responseElements.ConsoleLogin (Success/Failure), additionalEventData.MFAUsed (Yes/No). Example log (modified): userIdentity.arn = arn:aws:iam::417823659031:user/Dean, sourceIPAddress = 82.114.73.19, ConsoleLogin = Success, MFAUsed = No. Red flags: MFA not used, unfamiliar IP, unusual browser. 💡 IR Tip: AWS does not enforce MFA on IAM users by default — unlike Azure. If you land on an incident and find most users don't have MFA enabled, that's a systemic finding worth documenting. Scenario 2: Detecting New API Key Creation When an attacker compromises an account, one of the first things they do is create new API keys for persistence — even if the original password gets changed. The CloudTrail event to look for is CreateAccessKey. Key fields: ARN of who created it, eventTime, userAgent (console vs CLI vs API), sourceIPAddress, requestParameters.userName (which user the key is for), responseElements.accessKey.accessKeyId (the actual key ID created). Example: eventName = CreateAccessKey, sourceIPAddress = 91.200.14.77, requestParameters.userName = svc-billing, responseElements.accessKey.accessKeyId = AKIAXYZ98765LMNOQRST. If the account making the request is NOT svc-billing — that's a pivot. An attacker creating keys on service accounts is classic persistence. Scenario 3: Finding Evidence of Exposed API Keys Exposed API keys trigger a burst of IAM enumeration calls in CloudTrail from a single IP in a short timeframe: ListUsers, ListRoles, GetPolicy, GetPolicyVersion, ListAttachedUserPolicies, SimulatePrincipalPolicy. The attacker is looking for shadow admin roles — overpowered accounts the organisation forgot about. 💡 IR Tip: Filter CloudTrail by the specific AKIA key ID you're tracking. Every API call made with that key shows up — gives you the complete activity timeline from the moment it was first used. API Calls That Return Credentials — The Pivot List These API calls all return new credentials that can be used to impersonate a different role or service: iam:CreateAccessKey — creates a new permanent API key sts:AssumeRole — temporarily assumes another IAM role's permissions sts:AssumeRoleWithSAML — assumes a role via federated authentication sts:GetFederationToken — returns temporary credentials via federation cognito-identity:GetCredentialsForIdentity — returns credentials via Cognito redshift:GetClusterCredentials — returns credentials for a Redshift database --------------------------------------------------------------------------------------------------- CloudTrail Threat Hunting Scenarios Impossible Travel — same account logs in from London at 09:00 and Sydney at 09:45. Physically impossible. New Source IPs — build a 30-60 day baseline of known IPs per account. New IPs on privileged accounts need scrutiny. API Keys Accessing Console — AKIA-prefixed keys generating ConsoleLogin events = attacker using stolen credentials. GPU Instance Creations — crypto mining abuse. Look for RunInstances where instanceType contains g4, g5, p3, p4. New Accounts with Admin Roles — CreateUser followed by AttachUserPolicy for the same new user = persistence. --------------------------------------------------------------------------------------------------- ⚡ Update (2024): GuardDuty now includes many of these as automated detection rules — impossible travel, credential exfiltration patterns, cryptomining detection. Enabling GuardDuty removes the need to write these hunts manually, but understanding the CloudTrail logic makes you a far better investigator when you need to go beyond what GuardDuty surfaces. ------------------------------------------------------------------------------------------------------------- What's Next Next Article covers EC2 instances, EBS volumes, and snapshot-based evidence collection — how to capture a disk image from a running AWS instance without taking it offline. -----------------------------------------------Dean------------------------------------------------

If there's one log source you need to master for AWS investigations, it's CloudTrail. Every API call made in your AWS environment — whether it's someone logging into the console, a Lambda function creating a storage bucket, or an attacker enumerating your IAM roles — gets recorded here. This is your event log for the cloud. ------------------------------------------------------------------------------------------------------------- What CloudTrail Actually Is ▸ Default: 90 days free (management events only) | Trail: unlimited retention + data events CloudTrail records API calls made to AWS services. Think of it like Windows Event Logs, but instead of OS-level events, it's capturing every interaction with the AWS control plane. When someone creates a virtual machine, uploads a file, adds an IAM user, or changes a firewall rule — CloudTrail sees it and logs it. CloudTrail is turned on by default and stores 90 days of management events at no cost. To keep logs beyond 90 days, you need to create a trail routing logs to an S3 bucket. All timestamps in CloudTrail are UTC. CloudTrail does not capture what's happening inside your EC2 instances — only AWS API calls. For in-instance activity, you still need EDR/XDR. 💡 IR Tip: CloudTrail is not a replacement for EDR. It tells you what happened at the cloud control plane level. It won't tell you what commands ran inside a compromised EC2 instance or what files were created on disk. ------------------------------------------------------------------------------------------------------------- Management Events vs. Data Events — This Distinction Matters Management Events = control plane (who did what to the infra) | Data Events = data plane (what touched the data) Management events are control-plane interactions — creating, configuring, or deleting resources. Examples: creating an S3 bucket, launching an EC2 instance, adding an IAM user, modifying a security group. These are enabled by default with 90 days free retention. Data events capture interactions with data inside a service — reading/writing S3 objects, invoking Lambda functions, querying RDS. These are OFF by default. If you're investigating S3 data exfiltration and data events weren't enabled — you have a gap in your evidence. 💡 IR Tip: One of the first questions to ask when you arrive at an AWS incident: 'Do you have a CloudTrail trail? Are data events enabled?' If the answer is no to either, you're working with partial logs. Establish what you have early. ------------------------------------------------------------------------------------------------------------- CloudTrail SLA: How Fast Do Logs Appear? CloudTrail has a maximum SLA of 15 minutes. An API call could take up to 15 minutes to appear in your logs. Compare this to Azure where the SLA is 30 minutes to 24 hours — AWS is significantly faster. But if your attacker is moving quickly, even 15 minutes can be too late, which is exactly why GuardDuty runs continuously against incoming CloudTrail events. Reading CloudTrail Fields Like an Investigator eventTime — When the event was recorded, in UTC. Your timeline anchor. userIdentity — Who triggered this event. Most important field for attribution. eventSource — Which AWS service generated the event (ec2.amazonaws.com, iam.amazonaws.com, s3.amazonaws.com). eventName — The actual API call: ConsoleLogin, RunInstances, CreateAccessKey, GetObject. Learn these names. awsRegion — Which region processed this call. Critical for finding the right resources. sourceIPAddress — Where the call came from. Your geolocation and pivot point. userAgent — How the request was sent: signin.amazonaws.com (console), aws-cli (CLI), lambda.amazonaws.com (Lambda function). requestParameters / responseElements — Full JSON payload of what was requested and returned. Contains bucket names, file keys, instance IDs, created keys. sessionCredentialFromConsole — Boolean: did this API call come from the web console? Example: { "eventVersion": "1.11", "userIdentity": { "type": "AssumedRole", "principalId": "*", "arn": "*", "accountId": "*", "accessKeyId": "*", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "*", "arn": "*", "accountId": "*", "userName": "*" }, "attributes": { "creationDate": "2026-06-25T10:55:09Z", "mfaAuthenticated": "false" }, "ec2RoleDelivery": "*" }, "inScopeOf": { "issuerType": "*", "credentialsIssuedTo": "*" } }, "eventTime": "2*", "eventSource": "*", "eventName": "UpdateInstanceInformation", "awsRegion": "us-east-1", "sourceIPAddress": "*", "userAgent": "*", "requestParameters": { * }, "responseElements": null, "requestID": "*", "eventID": "*", "readOnly": *, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "*", "eventCategory": "Management", "tlsDetails": { "tlsVersion": "*", "cipherSuite": "*", "clientProvidedHostHeader": "*" } } ------------------------------------------------------------------------------------------------------------- ARN — Understanding Amazon Resource Names ▸ Full ARN: arn:aws:iam::417823659031:user/Dean The ARN format: arn:partition:service:region:account-id:resource-type/resource-id Example: arn:aws:iam::417823659031:user/Dean— partition=aws, service=iam (global, no region), account=417823659031, type=user, name=Dean Access Key ID Prefixes — Decoding the Key Type AKIA — permanent access key for an IAM user. High-risk: static, doesn't expire, can be accidentally exposed. ASIA — temporary STS token. These expire. Key indicator of lateral movement via the metadata service. AIDA — IAM user authenticated via the console (not using an API key). AROA — a role made this call, not a user directly. 💡 IR Tip: When you see ASIA tokens appearing for a role that shouldn't be requesting temporary credentials — especially with an unusual source IP — you may be seeing an attacker who accessed the EC2 metadata service and stole STS credentials to impersonate that instance's IAM role. ------------------------------------------------------------------------------------------------------------- Setting Up CloudTrail Trails If no trail is configured, you lose all CloudTrail data older than 90 days permanently. A trail routes logs to an S3 bucket you control. Configure it at the organisation level so every account that joins automatically sends logs to your central S3 bucket. ⚡ Update (2024): AWS introduced CloudTrail Lake — a managed event data store that lets you query CloudTrail events directly using SQL without needing S3, Glue, or Athena setup. For new deployments, CloudTrail Lake is often simpler than the S3+Glue+Athena pipeline. ------------------------------------------------------------------------------------------------------------- What's Next Next Articles goes into the actual investigation scenarios — how to track console logins, detect new API key creation, find evidence of exposed keys being abused, and run CloudTrail threat hunts. ----------------------------------------Dean---------------------------------------------------------

If you've ever walked into an AWS incident and felt like the engineers around you were speaking a completely different language — you're not alone. AWS has its own vocabulary, its own structure, and its own quirks. Before you can even begin to scope an investigation, you need to understand what you're actually dealing with. Think of this article as your field translator. We're going to walk through how AWS is organised, how identity works, and the different ways someone — legitimate or otherwise — can get access. Once you understand these fundamentals, everything else in the AWS forensic world starts making sense. How AWS is Organised: Accounts, Sub-Orgs, and the Root of Trust Diagram: AWS Organizations Hierarchy ▸ IR Role defined at Management Account level = read-only access to ALL accounts below it AWS environments in most enterprises aren't just a single account. They're built around something called AWS Organizations — a structure that lets a company group all their individual AWS accounts under one umbrell Picture it like a company's Active Directory forest. You have: A Management Account — this is the root of trust. It controls everything below it. Think of it as the domain root. Very few people should ever touch this account. Sub-Organizations (Sub-Orgs) — these sit below the management account and typically align to business units or purposes (Production, Development, QA). Roles defined at this level apply to all accounts beneath them but not to adjacent sub-orgs or the management account. Individual AWS Tenants (Accounts) — the actual working environments where resources live. Roles here only affect resources within that one account. For an IR team, the goal is to get a role at the management account level that gives you read-only access across the entire organisation. That's your holy grail — you can see everything without touching anything. The typical way to set this up is with a cross-org IAM role tied to a dedicated security account that sits completely outside the compromised org. IR Tip: Never put your IR account inside the same organisation you're investigating. If the threat actor has organisation-level access, they could tamper with your investigation account. Keep it separate and use a cross-org trust. ----------------------------------------------------------------------------------------------------- IAM — The Backbone of Everything IAM stands for Identity and Access Management. It's the gatekeeper for every single action in AWS. Whether you're creating a virtual machine, downloading from a storage bucket, or running a script — IAM is what decides if you're allowed to do it. For those coming from a Windows background: IAM is your Active Directory. For UNIX people: it's like NIS (Network Information Service). It's the central authority for who is who and what they can do. -------------------------------------------------------------------------------------------------------- Root vs. IAM Accounts — Know the Difference Every single AWS account has a Root account. This is the nuclear option — it has absolute control over everything in that account and cannot be restricted by IAM policies. AWS actively discourages logging into root except when absolutely necessary. Every other user or service runs as an IAM account. These can be authenticated with a password, an API key, or both. Unlike root, IAM accounts have roles and policies that define exactly what they're allowed to do. IR Tip: If you see CloudTrail logs with the root account being used repeatedly, that's a flag. In a properly managed environment, root should barely ever appear. Lots of root activity could mean bad security hygiene or a compromised account. AM Policies and Roles — How Permissions Actually Work ▸ Inline Policy → attached directly to the identity ▸ Managed Policy → reusable across identities Permissions in IAM come in three layers: Policy — the most granular unit. A policy is a JSON document that explicitly lists which API calls are allowed or denied against which AWS resources. Predefined Roles — AWS ships these out of the box for every service. You'll see things like 'S3FullAccess' or 'EC2ReadOnly'. They're meant as examples — most organisations should refine them. Custom Roles — these combine predefined roles with your own policies to create exactly the right level of access for a specific job. The tighter, the better. Here's what a dangerously over-permissive policy looks like — and unfortunately you'll find versions of this in real environments: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" } ] } That's an administrator policy. Action: * means every API call. Resource: * means every AWS resource. If an EC2 instance with this policy attached gets compromised, the threat actor effectively has admin access to your entire AWS account. We see this constantly — a developer can't get something working, slaps on this policy to make the problem go away, and then forgets about it. When you're investigating a compromised IAM account or role, you must expand and review every custom policy attached to it. Predefined roles can include inherited permissions that aren't obvious at first glance. IR Tip: AWS provides the IAM Policy Simulator — use it to test what a given user or role can actually do. It's invaluable for scoping an incident once you've identified a compromised identity. ------------------------------------------------------------------------------------------------------------- Four Ways to Access AWS — and What Each One Means for You Understanding how access is granted is critical because different access methods leave different evidence trails and carry different risks. 1. Username and Password The most familiar method — log in to the AWS web console with your email and password. The limitation here is that it only works for the web console. AWS CLI, SDK scripts, and automation can't use username/password. From an IR perspective, these are the credentials most at risk of phishing and account takeover. Without MFA, a stolen password is game over for that account. 2. API Keys This is how most real-world automation interacts with AWS. API keys come in two parts: an Access Key ID and a Secret Access Key. You can create them per-user, per-service, or per-role. The risk? API keys don't expire by default. They're static credentials. And bug bounty hunters, as well as threat actors, are constantly scanning public GitHub repositories, Pastebin, and Docker image layers looking for accidentally exposed keys. One exposed key in a public repo and your entire AWS environment could be compromised within minutes. 💡 IR Tip: Search for your organisation's API keys on GitHub using search operators like: 'AKIA' filename:.env — AKIA is the prefix that identifies AWS access keys. We cover key prefixes in Article 2. 3. SAML Tokens SAML (Security Assertion Markup Language) tokens are time-limited credentials that expire after a window you define — seconds, minutes, hours. They're generated through the AWS STS (Security Token Service) and are perfect for temporary, one-time access scenarios without managing permanent keys. 4. IAM Roles Assigned to Resources This is the cleanest approach from a security standpoint — attach an IAM role directly to an EC2 instance or other resource. The resource then gets permission to call AWS APIs without any hard-coded credentials anywhere. No credentials to steal in theory. In practice, this becomes a favourite attack path via the metadata service, which we cover in Article 9. ------------------------------------------------------------------------------------------------------------- The AWS Shared Responsibility Model — Know Where Your Boundary Is One thing that catches organisations off guard is thinking AWS is responsible for securing their data. That's not how it works. AWS secures the underlying infrastructure — the physical data centres, the hypervisor layer, the hardware. Everything you build on top of that? That's your responsibility. You configured the S3 bucket — you're responsible for who can access it. You launched the EC2 instance — you're responsible for patching the OS and securing access. You created the firewall rules — you're responsible for what traffic is allowed. The line is simple: if AWS manages the underlying service, they secure it. If you chose to run it, you secure it. 💡 IR Tip: When starting an AWS investigation, always establish the shared responsibility boundary first. Understanding what AWS can help you with and what you have to find yourself will save you time and help you ask the right questions when talking to AWS support. ------------------------------------------------------------------------------------------------------------- What's Next Now that you understand the structure and identity layer of AWS, the next article dives straight into CloudTrail — your primary audit log for every action taken in an AWS environment. We'll walk through what gets logged, what doesn't, and how to read the raw log fields like an investigator. -----------------------------------------------------Dean---------------------------------------

The Attacker Almost Won On a quiet Tuesday morning in June 2026, an employee at a mid-size organization pressed Win+R, pasted a command they found on a website, and hit Enter. By the time SentinelOne surfaced its first alert, Lumma Stealer had already spent over five minutes inside the machine — reading browser memory, querying the Windows credential database, capturing keystrokes, and probing the network for other machines to move to. The user had no idea. They spent the next hour and a half opening documents, reading email, and responding to messages on Slack — completely unaware that an attacker was sitting inside their session. What stopped the attacker from walking away clean was not a single magic detection. It was twelve behavioural indicators, spread across Multiple detection engines, woven together by SentinelOne's Storyline correlation engine. And the only reason we were able to understand the full scope of the compromise was a deep investigation inside Deep Visibility — because the alert alone, taken at face value, told us only a fraction of the story. This article walks through that investigation: what happened, what SentinelOne caught, what it nearly missed, and why looking beyond the alert is not optional — it is the difference between a full containment and a repeat breach. Background: ClickFix and Lumma Stealer ClickFix is a social engineering technique that has surged in 2025 and 2026. The victim is presented with a fake browser error, CAPTCHA, or 'verification' page that instructs them to open the Windows Run dialog (Win+R) and paste a command. The command is already in the clipboard — the victim just has to press Ctrl+V and Enter. No downloads, no suspicious file attachments, no executable to double-click. Lumma Stealer (also tracked as LummaC2) is a commercial infostealer sold as a service on criminal forums. It specialises in extracting browser-stored credentials, cookies, active session tokens, cryptocurrency wallet files, and system information — all in memory, leaving no binary on disk. Version 17 added direct system call capabilities to bypass EDR user-mode hooks, making it one of the more technically capable commodity stealers available in 2026. The combination is extremely effective. The human is the delivery mechanism, the malware never touches disk, and the exfiltration looks like normal HTTPS traffic. Organisations that rely purely on signature-based detection or file-based scanning will see nothing. Incident Timeline — What Happened (All times UTC, anonymised) Phase 1 — Initial Execution (09:31 UTC) The victim, a user on a corporate Windows 11 laptop (referred to as CORP-LT-2847), opened the Windows Run dialog and pasted the following command: This single line initiated a chain of events that would persist for over ninety minutes. The script downloaded Lumma Stealer v17.19 (4.85 MB) entirely into memory and executed it. No file was written to disk. SentinelOne registered the payload download as an outbound connection — status: SUCCESS. ------------------------------------------------------------------------------------------------------- Phase 2 — Evasion (09:31–09:36 UTC) Before Lumma could steal anything, it needed to blind the tools watching it. Two evasion techniques were observed: Windows Defender (AMSI) Bypass — Lumma loaded a legitimate, signed security DLL and used it to disable AMSI's scanning hooks — exploiting trust in signed code to remove the very inspection layer designed to catch malicious scripts. Direct Kernel Calls — EDR Hook Bypass — At 09:36:46, SentinelOne logged a critical evasion event: [09:36:46 UTC] *** KernelCallbackDirectSyscallNonNt *** [Evasion] process : WindowsTerminal.exe storyline: [STORYLINE-ID] Lumma v17 bypasses EDR user-mode hooks entirely by calling the Windows kernel directly — skipping the layer where SentinelOne monitors. The call executes before SentinelOne can inspect or block it. The fact that SentinelOne logged this at all is significant: the evasion was detected even if it could not be blocked in the moment. One second later, SentinelOne's logged another a critical evasion event: [09:36:47 UTC] *** LummaStealerPowerShellStager *** [Execution] process : powershell.exe storyline: [STORYLINE-ID] Both events share the same Storyline ID — SentinelOne's mechanism for grouping causally related events into a single attack chain. Without this, two separate events in Deep visibility with no apparent connection. With it, one coherent kill chain. ------------------------------------------------------------------------------------------------------- Phase 3 — Active Credential Theft (09:33–09:36 UTC) With evasion in place, Lumma began its primary mission. 09:33:51 — SAM Database Queried — The Security Account Manager (SAM) is Windows' local credential store. It holds NTLM password hashes for every local account on the machine. These can be cracked offline or used in pass-the-hash attacks to authenticate to other systems without knowing the actual password. 09:33:52 — Chrome Browser Memory Read (First) — Lumma read the memory of the running Chrome process and extracted saved passwords, session cookies, and active session tokens. This is not a Chrome exploit — Chrome stores credentials in memory while it runs. Lumma reads that memory directly. 09:33:59 — Session Token Captured — A specific SSO authentication token for the organisation's internal workspace was observed in the telemetry. This token allows full workspace access — reading messages, downloading files, accessing private channels — without any username or password. It does not expire until explicitly revoked. 09:34:06 — SentinelOne Registry Tamper Attempted — BLOCKED — Lumma attempted to modify a protected SentinelOne registry key to disable the agent. SentinelOne's self-protection mechanism blocked this attempt. 09:35:04 — ETW (Event Tracing for Windows) Tampered — Windows' native forensic logging framework was partially disrupted. An anti-forensics technique to reduce evidence available to investigators. 09:35:26 — Reconnaissance via svchost.exe — A trusted Windows system process (svchost.exe) showed 771 infostealer indicators and 393 reconnaissance indicators accumulated across its lifetime — signs of Lumma injecting activity into a signed Windows binary to enumerate processes, network shares, and USB devices. Because svchost is a trusted binary, this activity is harder to detect via process-based rules alone. 09:35:58 — Chrome Browser Memory Read (Second) — A second pass to ensure full extraction of session data from browser memory. ------------------------------------------------------------------------------------------------------- Phase 4 — Keylogger + Lateral Movement (09:34+ UTC) A keylogger was installed at 09:34:06, capturing all keystrokes from this point forward — including passwords typed, search terms, messages sent, and any sensitive content entered into forms. At approximately the same time, Lumma invoked WinRM (Windows Remote Management) to attempt lateral movement — probing whether it could authenticate to and execute commands on other machines on the same network segment, using the credentials it had already harvested. ------------------------------------------------------------------------------------------------------- Phase 5 — Shadow Copy Abuse (09:52–10:52 UTC) This phase is the one most often missed by teams that stop at the alert. Three separate accesses to Volume Shadow Copy Service (VSS) snapshots were observed: 09:52:27 — Shadow copy accessed (first) 10:12:45 — Shadow copy accessed (second) 10:52:27 — Shadow copy accessed (third) Why VSS? When Windows locks a file that is in use — like the SAM database or a browser's credential store — you cannot read it normally. VSS snapshots are point-in-time backups of the file system, and files in a snapshot are not 'in use,' so they can be read freely. Lumma used VSS to read locked credential files that would otherwise have been inaccessible. Three accesses across a one-hour window indicates repeated, deliberate attempts to extract different credential stores from backup copies. ------------------------------------------------------------------------------------------------------- Phase 6 — User Works Normally, Unaware (10:02–11:06 UTC) During this window, the user opened documents, received and read Mail messages, and received and read notifications. The keylogger was confirmed still running at 10:03:10 — 32 minutes after infection. The user had no indication anything was wrong. No pop-up, no slowdown, no ransom note. The machine looked and behaved completely normally. ------------------------------------------------------------------------------------------------------- Phase 7 — Chrome Process Injection Detected (11:06 UTC) SentinelOne raised a PreloadInjection behavioural indicator on Chrome — one of the most revealing signals in the entire investigation: indicator.name : PreloadInjection indicator.category: Evasion MITRE : T1055.012 (Process Hollowing) TargetProcess : chrome.exe (renderer) RootProcess : chrome.exe (main browser process) Active Content : UNSIGNED Lumma injected malicious code into Chrome renderer subprocesses during their initialization — before the process's own code had finished loading. The injected renderer ran with Chrome's identity and permissions, meaning its memory reads were attributed to a trusted, signed Google process. The flag activeContent.signedStatus: unsigned on a signed Chrome binary is the fingerprint of injected foreign code. This indicator fired twice in quick succession against two different Chrome child processes — Lumma was still actively operating inside Chrome 90 minutes after initial execution. ------------------------------------------------------------------------------------------------------- Phase 8 — SentinelOne Surfaces 12 Behavioural Indicators (09:36:44 UTC) SentinelOne surfaced all 12 accumulated behavioural indicators together, linked under a single Storyline. This is the 'alert' that an analyst would have seen first. The indicators fired across five detection categories: Execution: LummaStealerPowerShellStager — named signature match Evasion: KernelCallbackDirectSyscallNonNt — direct kernel call bypass Evasion: PreloadInjection — process hollowing in Chrome renderer (T1055.012) Credential Access: SAM database query; Chrome memory read (x2) Defence Evasion: AMSI bypass; ETW tampering; SentinelOne registry tamper attempt (blocked) Twelve signals. One Storyline. Phase 9 — Last Telemetry (11:10 UTC) The last event in available telemetry was logged at 11:10:05 UTC. The attacker remained active — the keylogger and any persistence mechanisms were still installed for the duration of the available window. Before we disconnected device from Internet What SentinelOne Got Right: Multiple Engines, One Signal Lumma Stealer deliberately tried to be invisible. It never wrote a file to disk. It bypassed AMSI. It used direct kernel calls to skip EDR hooks. It injected into trusted processes. It attempted to disable SentinelOne. It tampered with Windows event logging. And yet SentinelOne still detected it — not because of one rule, but because of layered detection architecture: The key lesson: no single engine caught the full picture. Lumma successfully bypassed AMSI. It used direct kernel calls before SentinelOne could block them. But it could not bypass every engine simultaneously. Evading one layer does not mean evading all layers. What the Alert Alone Did NOT Tell Us If an analyst had looked at intial alert and stopped there, they wouldn't have known a known stealer ran, it bypassed AMSI, it read Chrome memory, and it queried SAM. They would NOT have known: The keylogger was still active 32 minutes after the alert fired Three VSS shadow copy accesses occurred over a one-hour window — repeated, deliberate credential extraction from backup snapshots A specific SSO token was confirmed captured in telemetry Chrome PreloadInjection at 11:06 — Lumma still injecting into Chrome renderers 90 minutes after execution WinRM lateral movement was attempted — other machines may have been targeted User was working normally for 90+ minutes with an active keylogger running Every one of these findings came from Deep Visibility. None of them were in the alert. Why Deep Visibility Is Not Optional EDR alerts are designed to be actionable at scale. But for any alert involving a known infostealer, evidence of credential access, a session token captured in telemetry, or a machine not immediately isolated — stopping at the alert is a containment failure. Deep Visibility in SentinelOne is where the investigation begins, not ends. The alert told us the attacker entered the building. Deep Visibility told us which rooms they visited, what they took, and whether they were still inside when we showed up. In this investigation, the DV data changed the response entirely: Chrome memory read → Lumma still injecting into Chrome 90 minutes later SAM queried → SAM accessed three times via VSS snapshots over 60 minutes Session tokens at risk → Specific SSO token confirmed captured Attacker executed and left → Attacker still active at time of log collection Standard credential reset → Full rebuild + revoke all sessions on all platforms SentinelOne Detection Names The following detection names were raised by SentinelOne during this incident: LummaStealerPowerShellStager [Execution] KernelCallbackDirectSyscallNonNt [Evasion] PreloadInjection [Evasion — T1055.012 Process Hollowing] Key Takeaways Final Thought This attacker built a technically sophisticated tool. They bypassed AMSI, evaded user-mode EDR hooks, injected into trusted processes, accessed credentials through backup snapshots, and left the machine looking completely normal for ninety minutes. They nearly got everything they came for. What caught them was not one thing. It was the depth of a platform that kept watching even after individual evasion techniques succeeded — and an investigation that went beyond the alert to read what the telemetry was actually saying. SentinelOne did not prevent every evasion. But it recorded everything. And in endpoint security, the ability to reconstruct what happened with precision is often what separates a contained incident from a breach that goes undetected for months.

Traditional digital forensics has a straightforward playbook for compromised machines: pull the drive, image it, analyze the image. In cloud environments that approach does not work. You cannot physically pull a disk from a data center you do not have physical access to. Downloading a full virtual disk over the internet for a 512GB drive takes hours and costs a significant amount in egress fees. And shutting down the VM disrupts the business and may destroy volatile evidence. Azure's in-cloud IR approach solves all three problems. You can snapshot the VM's disk in seconds — even while the machine is running. You create a forensic copy without modifying the original. You spin up a fresh investigation VM in the same data center and analyze the copy there, avoiding download costs entirely. From snapshot to analysis-ready takes less time than a traditional disk acquisition would take just to image. This article walks through the complete five-step process, covers the VHD download option when you need it, and introduces the tools and resources that can accelerate your Azure investigations. Before You Start: Two Key Concepts Victim VM vs. Forensic VM The in-cloud IR methodology uses two virtual machines: Victim VM — The compromised machine you are investigating. Do not work directly on this machine. Your goal is to preserve its state, image its disk, and never modify the original. Forensic VM — A fresh VM you create specifically for this investigation. This is where forensic tools are installed and where you mount and analyze the copy of the victim's disk. Keeping these two machines distinct is the forensic integrity backbone of the entire process. Why Snapshots Only Take Seconds A disk snapshot in Azure is a full, point-in-time, read-only copy of a virtual disk. Azure uses copy-on-write technology — the snapshot is taken almost instantly, and disk blocks are only actually copied when they are modified after the snapshot is taken. For investigators this means: You can snapshot a running VM's disk without shutting it down The snapshot preserves the exact disk state at the moment of creation Multiple snapshots can be taken at different points in time The snapshot is read-only — you cannot accidentally modify it 💡 Investigator Note: Snapshots have an ongoing cost: Standard HDD snapshots cost approximately $0.05 per GB per month, Premium SSD approximately $0.132 per GB per month. For a 128GB OS disk, that is roughly $6-17 per month. Budget accordingly and confirm snapshot retention requirements with your client before starting. The Five-Step In-Cloud Forensic Acquisition Process Step 1a: Identify the Victim VM's Disk Navigate to the victim VM in the Azure Portal and select the Disks section. For most investigations you will snapshot the OS disk. If the incident involved data stored on a separate data disk — common in database server compromises — you may need to snapshot that as well. Note the disk name before proceeding. Step 1b: Create the Snapshot From the disk's overview page, select 'Create snapshot'. Key configuration choices: Name — Use a descriptive name that immediately identifies this as a forensic artifact. Good convention: VictimVMName-YYYYMMDD-Forensic-Snapshot. Avoid generic names like 'snapshot1' — when you have multiple disks and snapshots in a subscription, clarity is critical. Snapshot Type — Select 'Full'. This creates a complete, standalone point-in-time copy. 'Incremental' snapshots chain to previous snapshots and are used for backup workflows — not for forensic acquisition. Storage Type — Select 'Standard HDD'. There is no performance benefit to paying for Premium SSD for a snapshot since it is read-only and will not be accessed frequently. Step 2: Create a New Disk from the Snapshot The snapshot is read-only and cannot be attached to a VM directly. You need to create a new writable disk based on the snapshot data. This is the disk you will attach to your forensic VM. Navigate to the snapshot you just created Select 'Create disk' Name the disk clearly — good convention: take the snapshot name and append '-disk' For Source type, select 'Snapshot' and choose the snapshot you just created For disk type, select 'Premium SSD' — this disk will be actively used for forensic analysis so performance matters 💡 Investigator Note: At this point if you need to manage costs, you could delete the snapshot after the disk is created. However, keep the snapshot if there is any possibility you will need to re-create the disk. Never compromise forensic integrity to save a few dollars per month. Step 3: Create the Forensic VM Create a fresh VM that will serve as your forensic workstation in the cloud. Key considerations: Specifications — Minimum 4 vCPUs and 16GB RAM is a reasonable baseline. The exact specs depend on which forensic software you will run. Region — Critical: the forensic VM must be in the same Azure region as the snapshot disk. Azure does not allow attaching disks across regions without additional steps. OS Disk — The forensic VM gets its own OS disk where you install forensic tools. Premium SSD recommended. Azure typically provisions 128GB with over 100GB free. Data Disk — During VM creation under the 'Disks' tab, select 'Attach an existing disk' and choose the forensic snapshot disk from Step 2. Warning: Do not skip the data disk step thinking you can add it after. VMs can become unstable if a disk is added or removed while running. Always shut down the forensic VM before attaching or detaching data disks. Step 4: Mount the Snapshot Disk Once the forensic VM is running and you have RDP'd or SSH'd into it, open Disk Management. You will typically see: Disk 0 — The forensic VM's OS disk (C: drive) — your OS and tools Disk 1 — Temporary storage (D: drive if present) — not persistent, do not store evidence here Disk 2 — The snapshot disk containing the victim's OS partition When you first open Disk Management, Disk 2 will show as 'Offline'. Right-click and select 'Online'. Windows will assign drive letters to the partitions. The victim's OS partition will get a new drive letter, for example G:. Important: all mounted disks are writable by default. Be careful not to accidentally write to the victim's disk partition. If you corrupt the analysis disk, you can always re-create it from the snapshot — which is exactly why you keep the snapshot. Step 5: Run Your Forensic Tools With the victim's disk mounted as a readable drive letter, run your standard forensic toolset against it exactly as you would on a physical machine: KAPE — Ideal for targeted artifact collection. Fast and low-overhead. Run KAPE against the victim's drive letter to pull shimcache, amcache, LNK files, browser history, event logs, and more. Autopsy — For full disk analysis and timeline creation. Point it at the mounted volume. Custom scripts — Any Python, PowerShell, or batch script you would normally run against a drive. 💡 Investigator Note: KAPE is the recommended starting point for most Azure VM investigations. A targeted KAPE collection against the mounted victim disk can complete in 30-60 minutes and gives you the artifacts needed for timeline creation without processing the entire disk. Alternative Option: Downloading the VHD The five-step in-cloud process is recommended for most investigations. But there are scenarios where downloading the virtual hard disk to an on-premises forensic workstation makes sense: Your forensic tools cannot be deployed to Azure due to licensing restrictions The investigation requires off-network analysis for security reasons Court proceedings require physical media To download the VHD: Navigate to your snapshot in the Azure Portal Select 'Snapshot export' from the menu Choose how long the download URL should be valid — set this longer than you think you need. If it expires mid-download you have to start over. Generate the URL Under the Networking settings for the snapshot, ensure connectivity is set to 'Public endpoint' Downloading Faster with AzCopy The generated URL works in a browser but for any disk larger than a few gigabytes, browser downloads are impractically slow. Microsoft's AzCopy tool is free, multi-threaded, and significantly faster: azcopy cp "" "C:\Forensics\victim-snapshot.vhd" --check-md5 NoCheck Performance comparison: a 64GB VHD within the same Azure region downloads in under 20 minutes with AzCopy. The same download via browser can take close to an hour. For larger disks the difference compounds significantly. 💡 Investigator Note: Downloading a large VHD out of Azure incurs network egress charges. A 512GB VHD from Azure's primary commercial regions costs roughly $40-50 in egress fees alone. Confirm who absorbs this cost with your client before initiating the download. Building a Reusable Forensic VM Image One of the most time-consuming parts of cloud IR is setting up the forensic VM each time. Azure's VM image gallery solves this permanently. Create a base forensic VM with all your standard tools pre-installed: KAPE, Autopsy, Volatility, Python with forensic libraries, etc. Once configured, capture an image of that VM and store it in the Azure Compute Gallery For any future investigation, deploy a new VM based on that image — fully tooled and ready in minutes Move the forensic VM to different subscriptions or regions as needed This investment pays off quickly. The first time you need to investigate two incidents simultaneously, having a ready-to-deploy forensic VM image means you can spin up two identical fully-tooled environments in the time it would take to manually set up one. Additional Azure Security Resources Microsoft Sentinel Microsoft Sentinel is Azure's native cloud-hosted SIEM and SOAR platform. It ingests logs from Azure and non-Azure sources, provides built-in analytics rules for threat detection, and supports playbooks for automated response. 📌 Old: Previously called Azure Sentinel. ➜ Updated: Microsoft rebranded Azure Sentinel to Microsoft Sentinel in November 2021. Functionally identical but you will see both names in documentation depending on when it was written. Sentinel pricing is per gigabyte of data ingested. Large organizations generating terabytes of logs per day face significant costs. During client engagements, check whether Sentinel is deployed — if it is, it may hold historical log data that is more easily queryable than raw storage account blobs. Microsoft SimuLand SimuLand is an open-source initiative from Microsoft's security research team. It provides pre-built lab environments simulating well-known attack scenarios — phishing campaigns, credential theft, lateral movement, data exfiltration — in a controlled Azure environment. For investigators looking to build and test Azure investigation skills without waiting for a real incident, SimuLand is one of the best free resources available. Run the attack simulations yourself then practice investigating what happened using the techniques in this article series. Microsoft Incident Response Playbooks Microsoft has published four detailed IR playbooks covering the most common Azure attack scenarios: Phishing investigation Password spray investigation App consent grant investigation (OAuth token abuse) Compromised and malicious application investigation These playbooks provide step-by-step investigative guidance, KQL queries, and detection logic. They assume a solid foundation in Azure — which you now have from this article series. Offensive Research Tools Worth Knowing Understanding attacker tooling helps you recognize what evidence to look for. Two notable Azure-focused offensive tools that appear frequently in red team and threat actor activity: PowerZure — A PowerShell-based framework for assessing and exploiting Azure environments. Used by red teams to identify misconfigurations and escalation paths. PowerShell-based Azure enumeration activity during an investigation may look like PowerZure usage. MicroBurst — A collection of scripts from NetSPI for discovering Azure services and auditing configuration for vulnerabilities. Also includes post-exploitation capabilities like credential dumping from Azure AD Connect servers. If your client has an Azure AD Connect server and you see evidence of access to it, this may be how the attacker pivoted between on-premises and cloud. 💡 Investigator Note: ADConnect dump is a tool specifically designed to extract and decrypt credentials stored by Azure AD Connect servers. In hybrid environments where on-premises AD and Azure are synced, and you see evidence of access to the Azure AD Connect server, this is a key pivot technique to investigate. Wrapping Up: The Complete Azure IR Article Series Across these seven articles, we covered the complete Azure forensics and incident response toolkit from the ground up: Article 1: Azure architecture — tenant, subscriptions, resource groups, RBAC, resource IDs Article 2: Access methods — portal, CLI, PowerShell, Graph API, and Cloud Shell forensics Article 3: Compute and networking — VM types, managed disks, VNets, NSGs, network appliances Article 4: Tenant and subscription logs — sign-in logs, audit logs, activity logs, KQL queries Article 5: Storage and NSG flow logs — blob storage, data exfiltration detection, flow log analysis Article 6: VM-level OS logs — WAD, Windows event logs, Linux syslog, IIS logs Article 7: In-cloud VM acquisition — snapshot, forensic VM, disk mount, analysis, VHD download Azure investigations reward preparation. The most significant variable is not the investigator's skill — it is whether the client had logging configured before the incident. Push hard for: Log Analytics workspace deployed, NSG flow logs enabled at Version 2, storage account access logs enabled, VM diagnostic agents installed, and logs streaming to a SIEM or long-term storage. ----------------------------------------------------------------------------------------------------------Special Thanks I would like to extend my heartfelt gratitude to one of my dearest friend, a Microsoft Certified Trainer, for her invaluable assistance in creating these articles. Without her support, this would not have been possible. Thank you so much for your time, expertise, and dedication! https://www.linkedin.com/in/iqrabintishafi/ --------------------------------------------------------------------------------------------------------- Complete Azure series below: https://www.cyberengage.org/courses-1/azure-incident-response

Network logs tell you what traffic hit a machine. Activity logs tell you when it was created and modified. But neither tells you what happened inside the operating system — which processes ran, which accounts authenticated locally, which files were accessed. For that level of detail, you need OS-level logs, and in Azure collecting those requires an agent running on the VM itself. The upside is significant: Azure's diagnostic agent lets you pull Windows event logs and Linux syslog remotely, storing them in a centralized storage account, without needing to RDP or SSH into the compromised machine. For many investigations this means collecting critical OS-level evidence without contaminating the live system or alerting an attacker who may still have access. Azure VM Diagnostic Agents: A Brief History Azure has offered several agents for collecting VM metrics and logs over the years. Which agent is in use in your client's environment determines where logs are stored and how to access them. The Current Agent Lineup Azure Monitor Agent (AMA) — The current, actively developed agent. Generally available since November 2021. Its key feature is Data Collection Rules, which allow fine-grained control over exactly what data is collected from which machines. Current limitation: it cannot write directly to a storage account or event hub — it sends data to a Log Analytics workspace only. Windows Azure Diagnostics Extension (WAD) — The older agent Microsoft calls WAD or the Diagnostics Extension. This is the agent that can write directly to a storage account or event hub, making it the only option when you need OS logs in a storage account. Many environments still use WAD. Log Analytics Agent (MMA — Microsoft Monitoring Agent) — A legacy agent that predates AMA. It sends data to Log Analytics workspaces and is being retired. 📌 Old: The Log Analytics agent (MMA) is being retired by Microsoft. ➜ Updated: Microsoft announced the Log Analytics agent retirement for August 2024. Any environment still using it should have migrated to the Azure Monitor Agent. If you encounter MMA during an investigation, flag it as outdated infrastructure. 💡 Investigator Note: WAD remains relevant for incident response because it is the only agent that can store Windows event logs in a storage account — accessible via Azure Storage Explorer without needing a Log Analytics workspace. Expect to encounter both WAD and AMA in production environments. 📌 Old: The book focuses on configuring WAD via the classic 'Diagnostic settings' menu on the VM. ➜ Updated: In the current Azure Portal, the recommended path is creating a 'Data Collection Rule' under Azure Monitor. The underlying log tables and storage locations remain the same — only the configuration UI changed. Configuring Windows Azure Diagnostics (WAD) To collect Windows event logs from a VM using WAD, configure the diagnostic settings for that VM. Classic Configuration Path Step 1 — In the Azure Portal, navigate to your VM and select 'Diagnostic settings' Step 2 — Under the 'Logs' tab, select Windows event log categories and severity levels to collect: Application, Security, and System logs Step 3 — Under the 'Agent' tab, select the storage account where logs will be written Step 4 — Optionally set a disk quota to prevent logs from consuming unlimited storage space New Azure Monitor Path Step 1 — Search for 'Monitor' in the Azure Portal Step 2 — Navigate to 'Data Collection Rules' and create a new rule Step 3 — Associate the rule with your target VMs and configure the Windows event log sources and severity levels For incident response, collect at minimum: Security (All), System (Error, Warning, Critical), and Application (Error, Warning). Collecting all Information-level events generates substantial volume with limited investigative value. Windows Event Logs: The WADWindowsEventLogsTable Once WAD is configured and collecting, Windows event logs from your VM are written to a NoSQL table in the specified storage account. The table is named WADWindowsEventLogsTable. This table cannot be browsed like a blob container. It is a NoSQL table structure and requires Azure Storage Explorer to access properly. Accessing WADWindowsEventLogsTable with Azure Storage Explorer Step 1 — Open Azure Storage Explorer and connect to the storage account holding VM diagnostic logs Step 2 — In the left panel, expand the storage account and navigate to Tables Step 3 — Locate WADWindowsEventLogsTable Step 4 — Query or export the table data to CSV for analysis in your preferred tool The CSV export is the most practical format for feeding into Timesketch, Excel, a SIEM, or any other analysis platform. The table structure closely mirrors the standard Windows Event Log format. Key Windows Event IDs for Security Investigations 4624 — Successful logon. Look for unusual logon types (Type 3 = network, Type 10 = remote interactive) from unfamiliar IPs or at unusual hours. 4625 — Failed logon. Clusters of failures indicate brute force or credential stuffing. 4648 — Logon using explicit credentials (RunAs). Can indicate lateral movement or privilege escalation. 4720 — A user account was created. Check for backdoor accounts created during the attack. 4732 — A member was added to a security-enabled local group. Attacker adding themselves to Administrators. 7045 — A new service was installed. Common persistence mechanism. 4688 — A new process was created (if process tracking is enabled). Shows what commands were run. 💡 Investigator Note: Process tracking (Event ID 4688) is not enabled by default and must be explicitly turned on via Group Policy. Check whether it is enabled — its presence or absence significantly affects your visibility into what ran on the machine. Linux Logs: The LinuxSyslogVer2v0 Table Linux VMs in Azure work similarly to Windows — a diagnostic agent collects syslog data and writes it to a storage account table. Unlike Linux which has separate log files (auth.log, kern.log, syslog), Azure consolidates all Linux log data into a single table: LinuxSyslogVer2v0. Configuring Linux Syslog Collection Navigate to the VM's Diagnostic Settings in the Azure Portal Under the 'Syslog' tab, select the syslog facilities you want to capture (kern, auth, syslog, etc.) and the minimum severity level Select the storage account destination Under the 'Metrics' tab you can configure collection of system performance data: CPU utilization, memory, network I/O, disk I/O. Metrics can corroborate investigation findings — a crypto miner shows sustained high CPU, and ransomware encrypting files produces distinctive disk activity spikes. Accessing Linux Logs with Azure Storage Explorer Open Azure Storage Explorer Navigate to Tables under the appropriate storage account Locate LinuxSyslogVer2v0 Export to CSV for analysis For security investigations, the auth facility is your highest priority — it captures all authentication events, sudo usage, and PAM activity. Key things to look for in Linux syslog: SSH authentication events — successful and failed logins via sshd sudo usage — any command run with elevated privileges New user creation — useradd or adduser commands in auth.log Cron job additions — new cron entries for persistence Unusual process execution patterns in syslog Application Logs: IIS and Custom Tracing IIS Logs For Windows VMs running Internet Information Services as a web server, WAD can collect IIS access logs. Unlike other log types that go to NoSQL tables, IIS logs are stored as plaintext files in blob storage — making them immediately accessible with standard log analysis tools. IIS logs follow the W3C extended log format with fields including: date and time of the request client IP address (c-ip) HTTP method (cs-method: GET, POST, etc.) URI stem and query string (cs-uri-stem, cs-uri-query) HTTP status code (sc-status) User agent string (cs(User-Agent)) Bytes transferred (sc-bytes, cs-bytes) For web application attack investigations, IIS logs are your primary evidence source. They show SQLi attempts, path traversal attacks, webshell access patterns, POST requests used to upload malicious files, and User-Agent strings that identify automated attack tools. Custom Application Logs and ETW Windows applications can generate diagnostic tracing via the .NET tracing framework. Azure's diagnostic agent can also capture Event Tracing for Windows (ETW) events — a low-level Windows mechanism for capturing kernel and application events. ETW events are primarily useful when investigating an application-layer incident where the developers have instrumented their code with ETW tracing. During a standard intrusion investigation, you are unlikely to prioritize ETW unless the application itself is the specific target. 💡 Investigator Note: If you are investigating a web application compromise, ask the development team whether they have custom application logging configured. Developers often add detailed logging that captures business logic events — which user did what at what time — that never appears in any Azure native log. Log Source Audit Checklist Before closing any Azure investigation, run a log source audit: Tenant logs (Entra ID sign-in + audit) — ON by default. 30-day portal limit — confirm if exported to workspace or storage. Subscription logs (Activity log) — ON by default. 90-day default retention — confirm if exported. NSG flow logs — OFF by default. Confirm Version 2 is enabled for each NSG in each region. Storage account access logs (StorageRead/Write/Delete) — OFF by default. Check each relevant storage account. VM Windows event logs (WADWindowsEventLogsTable) — OFF by default. Confirm agent is installed and configured. VM Linux syslog (LinuxSyslogVer2v0) — OFF by default. Same as above. IIS logs — OFF by default. Enable if web servers are in scope. For any log source that is off: document it, turn it on immediately to capture going forward, and note in your investigation report that evidence for the prior period is unavailable. This is a finding your client needs to remediate. In the final article, we put all of this together and walk through the complete process of acquiring a forensic image of a compromised Azure VM — without downloading it, without disrupting operations, and entirely within the cloud. Special Thanks I would like to extend my heartfelt gratitude to one of my dearest Friends, a Microsoft Certified Trainer, for her invaluable assistance in creating these articles. Without her support, this would not have been possible. Thank you so much for your time, expertise, and dedication! https://www.linkedin.com/in/iqrabintishafi/ Next Article https://www.cyberengage.org/post/azure-logging-part-2-storage-accounts-nsg-flow-logs-and-the-data-exfiltration-trail

When you are called into an Azure incident, there is a good chance a virtual machine is at the center of it. Either the VM itself was compromised, it was used as a launching pad, or the attacker deployed new VMs to run their own workloads. Understanding how Azure's compute and networking layer works — and how it differs from on-premises environments — will save you from costly mistakes and missed evidence. Azure Virtual Machines: Not All VMs Are Equal Azure offers a wide variety of VM types organized into named series. While the investigation process does not change based on which series a VM belongs to, knowing the taxonomy helps you understand what you are looking at when VM names appear in logs and resource IDs — and it helps you spot red flags when unexpected series appear in an environment. VM Series Overview Series A — Entry-level machines suited for development workloads, low-traffic websites, or microservices. If you see Series A VMs in production handling sensitive data, flag it as a potential configuration concern. Examples: A1 v2, A4 v2. Series B — Burstable performance. Low-cost machines that run at reduced CPU baseline but spike when demand rises. Examples: B2S, B4MS, B16MS. Threat actors deploying crypto miners sometimes use burstable instances to reduce costs — the burst behavior can make elevated CPU harder to detect. Series D — General purpose, the workhorse of Azure compute. Covers most production workloads. Examples: D2a v4, D4s v4. This is the series you will most commonly encounter during investigations. Series F — Compute-optimized with a high CPU-to-memory ratio. Examples: F2s v2, F8s v2. If you see these deployed unexpectedly in bulk, it could indicate compute-intensive abuse such as mining. Series E, G, M — Memory-optimized, built for databases and in-memory analytics. Examples: E4s v4, M8ms. Rarely involved in typical attacks unless the database itself is the target. Series NC, NV, ND — GPU-optimized machines for graphics processing, machine learning, and predictive analytics. These are expensive. Their unexpected presence in an environment is a significant red flag. Series H — High-performance computing for scientific and financial modeling. Also expensive and unusual in typical enterprise environments. 💡 Investigator Note: Regardless of VM series, log collection and forensic analysis works the same way. The series affects cost, performance, and potential attacker motivation — not the investigation methodology. Managed Disks: The Foundation of VM Storage Every Azure VM has at least one managed disk — its operating system disk. Unlike the physical hard drive in a laptop, Azure managed disks are cloud-hosted storage volumes billed on a recurring basis. Understanding managed disks is critical because disk snapshots are the primary method for forensically acquiring a running VM in Azure without disrupting it. Four Types of Managed Disks Standard HDD — Slowest and cheapest. Used for infrequent access workloads or backups. Rarely seen on production systems handling sensitive data. Standard SSD — The standard for production workloads. Balanced performance and cost. Premium SSD — High-performance storage for demanding applications. Common on database servers and high-traffic web servers. Ultra Disk — Extreme performance for the most demanding workloads such as real-time analytics and high-transaction databases. What VMs Typically Have OS Disk — Selected when the VM is created. Contains the operating system. Temporary Disk — Short-term storage used for page files or swap files. Not all VM types have this. Data here is NOT persistent and will be lost when the VM is stopped or deallocated. Data Disks — Optional additional disks added by the user. These persist independently of the VM. The Cost Reality That Shapes Investigations Azure managed disk costs are ongoing — you pay for disk type, size, snapshots you create, data transfers out of Azure, and per-transaction charges. When you create a forensic snapshot of a disk, that snapshot incurs charges for as long as it exists. Coordinate with your client on retention and who absorbs that cost. 💡 Investigator Note: Downloading a disk image out of Azure to an external forensic workstation will incur network egress charges. For a 500GB disk, that cost adds up fast. Consider performing forensic analysis in-cloud instead. Azure Virtual Networks: The Plumbing That Connects Everything An Azure Virtual Network (VNet) is the software-defined private network that connects Azure resources to each other and, optionally, to the internet and on-premises environments. Understanding the network topology of an environment is foundational — it tells you how resources communicate, what is exposed to the internet, and what paths an attacker could have used to move laterally. IP Address Allocation When you create a VNet, Azure assigns a private IP address range from RFC 1918 space — typically starting with 10.0.0.0/24 for the first VNet. Resources within the same VNet communicate using these private addresses. VMs do not have public internet connectivity unless a public IP address is explicitly assigned to them, which carries an additional recurring cost. Connecting to On-Premises Environments Many organizations extend their Azure environments to connect back to on-premises infrastructure. Three ways this is done: Point-to-site VPN — A connection between the VNet and a single remote computer. Common for individual admin access. Site-to-site VPN — A persistent connection between an on-premises VPN device and an Azure VPN gateway. Common in hybrid environments. Azure ExpressRoute — A private dedicated circuit between an on-premises location and Azure through a network partner. No internet traffic — higher cost, lower latency, higher security. From an investigative standpoint, knowing what connectivity exists between Azure and on-premises is crucial. Lateral movement between cloud and on-premises environments is a common attacker technique, and understanding the network paths helps you trace that movement. Network Security Groups: Azure's Built-In Basic Firewall When you deploy a VM in Azure, a Network Security Group (NSG) is automatically created alongside it. The NSG controls which traffic is allowed to reach your VM — both inbound and outbound. Think of it as a basic stateful firewall at the network interface level. How NSG Rules Work NSG rules evaluate traffic based on five parameters: source IP, source port, destination IP, destination port, and protocol. Rules are applied in priority order, from lowest number to highest — 100 is the highest priority, 4096 is the lowest. Azure automatically creates default rules in the 65000 priority range as catch-all fallbacks. A critical investigative scenario: imagine you are told a firewall rule was in place to block malicious inbound traffic — but your investigation reveals that rule was assigned priority 4000, while another rule explicitly allowing that traffic was assigned priority 200. The lower-numbered rule wins, and the blocking rule never actually blocked anything. Priority misconfigurations like this are surprisingly common and can dramatically change the incident narrative. NSG Flow Logs: Your Network Traffic Record NSGs can be configured to capture flow logs — records of the network traffic hitting the NSG. These are one of the most valuable data sources in an Azure investigation. Key characteristics: Layer 4 visibility — You see IPs, ports, and protocols. Not application-layer content. One-minute intervals — Fixed, cannot be changed. JSON format — Consistent with all other Azure logs. One-year default retention — Longer than most Azure log sources. 5-tuple per record — Source IP, source port, destination IP, destination port, protocol — plus the traffic decision (Allow or Deny) and in Version 2, throughput data. This data is essential for tracing data exfiltration, lateral movement, and C2 communications. The catch: flow logs are NOT enabled by default. If your client did not turn them on before the incident, you are working without this crucial data source. 💡 Investigator Note: NSG flow logs cost $0.50 per GB beyond the first 5GB per month, plus storage account charges. Some organizations skip them to save money. When conducting a pre-incident readiness review, always check that flow logs are enabled — and make sure your client understands what they are giving up if they are not. Beyond NSGs: Other Network Infrastructure You Might Encounter NSGs are just one layer of network control in Azure. Depending on the environment, you may encounter additional components that affect traffic flow and generate their own logs. Azure Load Balancer Distributes incoming traffic across multiple VM instances. If a compromised application server sits behind a load balancer, you may need to check load balancer logs to understand which backend VM actually handled specific requests. Azure Firewall A managed, cloud-based firewall that provides both stateful network filtering and application-layer (Layer 7) inspection. Unlike NSGs, Azure Firewall can filter traffic based on fully qualified domain names, detect and block known malicious traffic, and provide centralized logging. If an environment has Azure Firewall deployed, its logs are a rich data source. Application Gateway A web application firewall (WAF) that sits in front of web applications to filter malicious HTTP/HTTPS traffic. If the incident involves a compromised web application, Application Gateway logs may show what attack traffic looked like before it reached the application. VPN Gateway Manages the connection between on-premises networks and Azure. Log any authentication events and connection records — these can reveal unauthorized remote access attempts. Third-Party Network Virtual Appliances Many organizations deploy third-party firewall, VPN, or WAN optimization appliances from the Azure Marketplace — vendors like Palo Alto, Check Point, Fortinet. These have their own logging capabilities separate from native Azure logs. Always ask your client what third-party network appliances are deployed. They may hold critical evidence that does not appear anywhere in the native Azure log stack. When you first access an environment during an investigation, build a network topology picture before diving into logs. Know what is filtering traffic, where traffic flows, and what is exposed to the internet. This context makes every log entry more meaningful. In the next article, we move into the core of what makes or breaks an Azure investigation: the logging system. We will cover the five types of Azure logs and how to configure, access, and query each one. Special Thanks I would like to extend my heartfelt gratitude to one of my dearest freinds, a Microsoft Certified Trainer, for her invaluable assistance in creating these articles. Without her support, this would not have been possible. Thank you so much for your time, expertise, and dedication! https://www.linkedin.com/in/iqrabintishafi/ ----------------------------------------------------------Dean-------------------------------------- Next Article https://www.cyberengage.org/post/azure-logging-part-1-tenant-and-subscription-logs-the-starting-point-for-every-azure-investigatio

If the previous article covered the logs you are likely to find turned on when you arrive at a scene, this one covers the logs you need but probably will not find. NSG flow logs, storage account access logs, and the forensic trails for tracking data exfiltration — all off by default. That means you either find them already configured or you turn them on immediately and accept that prior activity may be gone forever. The good news: when these logs were configured, they hold some of the most specific, actionable evidence in Azure. Network flow data can reconstruct attacker movement. Storage access logs can confirm exactly which files were read, downloaded, or deleted. Understanding how to find, enable, and interpret these logs is what separates a complete Azure investigation from one that goes cold. Azure Storage: The Context You Need Before the Logs Storage accounts appear everywhere in Azure investigations — not just as a log destination but as actual evidence sources and frequently as the exfiltration target itself. Before jumping into storage logs, it is worth understanding how Azure storage works at a level that makes the forensic implications clear. Four Types of Azure Storage Blob — Binary Large Object storage. Used for unstructured data: images, videos, documents, and especially logs. Most relevant type for investigations. File — A distributed file share accessible via SMB protocol, similar to a Windows file share. Queue — A message queue service for storing and retrieving messages between application components. Table — NoSQL table storage, now part of Azure Cosmos DB. This is where VM operating system logs (WAD) are stored. Blob Storage: How It Works Blob stands for Binary Large Object. Any file type can be stored in a blob container — making it ideal for large volumes of log data and, unfortunately, for staging stolen data before exfiltration. Three types of blobs: Block Blobs — Used for text and binary data. Can store up to 4.75TB (preview supports up to 190.7TB). This is what most logs are stored as. Append Blobs — Optimized for append-only operations. Ideal for logging data since entries are always added to the end. Page Blobs — Used for random-access files up to 8TB. This is the format for virtual hard disk (VHD) files — so when you download a VM disk snapshot, it comes as a page blob. Every storage account has a globally unique URL: https://accountname.blob.core.windows.net. This means any blob can be accessed directly over the internet using that URL and the correct credential. That is not just a technical detail — it is a data exfiltration vector. Storage Account Access: Keys, SAS, and the Public Access Problem Storage accounts come with two access keys by default. These keys grant full access to everything in the storage account. The reason there are two is operational — you can rotate one while the other stays active. A better approach is Shared Access Signatures (SAS). An SAS is a time-limited credential granting specific, restricted access to specific storage resources. It can be scoped as narrowly as a single blob file with read-only access for the next 24 hours. SAS tokens expire automatically. The worst option — and surprisingly common — is enabling public access on a blob container. This allows anyone on the internet to read data without any credential at all. This has led to numerous breach incidents where organizations accidentally exposed sensitive data in publicly accessible Azure blob containers. 💡 Investigator Note: During any Azure investigation, run a check for publicly accessible blob containers. Navigate to each storage account and check the 'Public access level' setting on each container. This is a quick win that frequently reveals unintentional data exposure. Checking for Key Enumeration When investigating a storage account that may have been accessed by an attacker, one of the first things to check is whether any keys were recently listed. In the Activity Log, look for this specific operation: AzureActivity | where OperationNameValue == "MICROSOFT.STORAGE/STORAGEACCOUNTS/LISTKEYS/ACTION" | project TimeGenerated, CallerIpAddress, Caller, ResourceId | order by TimeGenerated desc LISTKEYS events from an unfamiliar IP address or account strongly indicate an attacker obtained access credentials and could have subsequently exfiltrated data from that storage account. Storage Account Logs: Finding the Exfiltration Evidence Knowing that access keys were listed does not prove data was actually taken. To confirm exfiltration, you need storage account access logs — specifically the StorageRead log that records every read operation against blob data. Critical problem: StorageRead logs are disabled by default. If your client did not enable them before the incident, there is no record of data being read. An attacker could have downloaded gigabytes of data and left absolutely no trace in Azure native logs. Enabling Storage Account Logs Storage account logging is configured through Diagnostic Settings. Two options: Diagnostic Settings (preview) — The newer, recommended approach. Allows granular configuration for each storage type (blob, queue, table, file) separately. Can send logs to a Log Analytics workspace, a different storage account, an event hub, or a partner solution. Diagnostic Settings (classic) — The older approach. Still functional but being phased out. For each storage type, you can enable logging for Read, Write, or Delete operations individually. The relevant one for data exfiltration is Read. Be selective — in high-traffic environments, enabling Read logging for all blob containers generates enormous log volume and significant storage costs. 💡 Investigator Note: Microsoft's Threat Matrix for Storage Services maps specific MITRE ATT&CK techniques applicable to Azure storage including data exfiltration. It is a useful reference for building detection logic around storage access patterns. Tracing Data Exfiltration: The GetBlob Operation Once StorageRead logs are enabled, every file downloaded from a blob container generates a log entry with the operation type GetBlob. To search for exfiltration evidence: StorageBlobLogs | where OperationName == "GetBlob" | where TimeGenerated > ago(7d) | project TimeGenerated, CallerIpAddress, Uri, StatusCode | order by TimeGenerated desc Look for bulk GetBlob operations from an unfamiliar IP, particularly if they occur shortly after a LISTKEYS event. The Uri field tells you exactly which files were accessed. Enforcing Storage Logging at Scale with Policies Manually enabling storage logging on every storage account is impractical for large organizations. A better approach is Azure Policy to enforce logging on all storage accounts — existing and future — at the management group or subscription level. There is no out-of-the-box predefined policy for forcing storage account logging, but custom policies can be written. The policy can send storage logs to an event hub. Note: the event hub must be in the same region as the storage account, so you may need one per region. If you are doing a proactive engagement or post-incident hardening review, recommend storage logging policies as a priority item. It is one of the highest-value configurations for future investigations. NSG Flow Logs: Your Network-Level Investigation Record let us cover how to set them up, what they contain, and how to use them for investigation. What NSG Flow Logs Capture Layer 4 visibility — IPs, ports, and protocols. Not application-layer content. One-minute intervals — Fixed, cannot be changed. JSON format — Consistent with all other Azure logs. One-year default retention — Longer than most other Azure log sources. 5-tuple per record — Source IP, source port, destination IP, destination port, protocol — plus the traffic decision (Allow or Deny) and in Version 2, throughput data (bytes and packets). Setting Up NSG Flow Logs: Three Steps Step 1: Enable Network Watcher — Must be enabled per region. If VMs are deployed in East US, West Europe, and Southeast Asia, you need Network Watcher in all three. Do not miss a region. Step 2: Register the Microsoft.Insights provider — Must be registered in each subscription that will use flow logs. Step 3: Enable NSG flow logging — Navigate to the specific NSG, configure flow log Version 2 (strongly recommended over Version 1 — Version 2 adds throughput data), and specify the storage account destination. 💡 Investigator Note: Version 2 is the critical choice here. Version 1 only tells you whether traffic was allowed or denied. Version 2 also tells you how much data was transferred. For exfiltration investigations, knowing throughput is often the difference between confirming and merely suspecting data was moved. Traffic Analytics: Visualizing the Flow Data Raw NSG flow logs in JSON format are useful but not the fastest way to spot anomalies. Traffic Analytics is a feature built on top of Log Analytics that processes flow log data and provides visual insights. Traffic Analytics can: Visualize network activity across all Azure subscriptions on a world map Identify security threats such as known malicious IPs and unusual traffic patterns Show traffic flow patterns between resources Highlight network misconfigurations like unused NSG rules or overly permissive rules From an investigation standpoint, Traffic Analytics is particularly useful for quickly identifying unexpected external connections — C2 communications, exfiltration destinations, and lateral movement paths — without writing raw KQL queries against JSON flow data. 📌 Old: Traffic Analytics was previously a premium feature with additional cost. ➜ Updated: As of 2023, Traffic Analytics pricing changed to be based on data processed, billed per GB. The first 1GB per month is free. Factor the cost into logging strategy recommendations for high-traffic environments. Azure Storage Explorer: The Investigator's Download Tool Azure Storage Explorer is a free graphical desktop application from Microsoft that lets you browse and download blobs, tables, and file shares from Azure storage accounts. It is the simplest way to get logs out of Azure without writing code. During an investigation, you will use Storage Explorer to: Navigate to the storage account holding archived logs Download log blobs (PT1H.json files) for offline analysis Access Windows Azure Diagnostics tables for VM event logs Download Cloud Shell Linux container contents such as .bash_history Download VM disk snapshots (VHD files) for forensic analysis Quick reference of the log blob names you will encounter most frequently: insights-logs-auditlogs — Tenant audit logs (directory changes) insights-logs-signinlogs — User sign-in logs insights-logs-noninteractiveusersigninlogs — Non-interactive sign-in logs insights-activity-logs — Subscription activity logs insights-logs-networksecuritygroupevent — NSG events insights-logs-networksecuritygroupflowevent — NSG flow logs (your network traffic records) The PT1H.json naming convention means one file per hour. For a seven-day investigation window, you will be dealing with 168 hourly files per log type. Tools in various DFIR toolkits can merge these into a single JSON file for easier analysis. In the next article, we go inside the virtual machines themselves — covering Windows event logs, Linux syslog, and application logs collected through Azure's diagnostic agent. ----------------------------------------------------------------------------------------------------------Special Thanks I would like to extend my heartfelt gratitude to one of my dearest friends, a Microsoft Certified Trainer, for her invaluable assistance in creating these articles. Without her support, this would not have been possible. Thank you so much for your time, expertise, and dedication! https://www.linkedin.com/in/iqrabintishafi/ -------------------------------------------------------------------------------------------------------------

A complete, no-fluff technical walkthrough — from zero infrastructure to a live, multi-region phishing drill hitting larger set of employees across multiple countries. -------------------------------------------------------------------------------------------------------- 1. What Is This and Why Are We Building It? Alright, let's kick things off. What exactly is a phishing simulation drill? In plain English: it's a controlled, authorized test where your own security team sends fake phishing emails to employees — just to see who clicks, who submits credentials, and (hopefully) who reports it. I know it sounds a bit sneaky, but honestly it's one of the most effective things you can do for security awareness training. Why? Because it measures real behaviour under real conditions — not just whether someone sat through a training video. For this, we used GoPhish — a free, open-source phishing framework written in Go. It gives you a slick web dashboard where you can create email templates, build landing pages, manage target groups, launch campaigns, and watch results come in live. Oh, and it has a clean REST API. It's pretty great. For our project specifically, we needed something that could: Send around 5000 phishing simulation emails across multiple countries Host a convincing landing page where employees land after clicking the link Track who opened, who clicked, and who actually submitted data Redirect to an awareness/education page after someone interacts with it Run on proper enterprise-grade infrastructure (Azure) — not a home server Pass SPF, DKIM, and DMARC checks so the emails actually land in inboxes One thing before we go any further — and this is non-negotiable: phishing drills MUST have written legal sign-off before you start. That means HR, Legal, and the relevant councils or DPOs depending on the country. Never run a phishing simulation without proper authorization — it is genuinely illegal in many jurisdictions without it. -------------------------------------------------------------------------------------------------------- 2. What You Need Before You Start 2.1 Legal and Organizational Prerequisites Seriously, sort this out first. Before any server gets touched: Written authorization from management, HR, and Legal — for every country employee you're targeting A clear scope document spelling out who's being tested, when, and what scenario Employee CSV files from HR (First Name, Last Name, Email, Position) — one per region, handled securely 2.2 Infrastructure Requirements Microsoft Azure subscription with quota for: 2 Linux VMs, 1 Windows Server VM, 1 Application Gateway (Standard_v2 or WAF_v2), 1 Standard SKU Load Balancer, and Azure Bastion Domain registrar account — we used Hostinger (more on this shortly) Two domain names: one for the sender (email FROM address), one for the landing page link Basic Linux command-line knowledge (SSH, systemctl, apt, editing files) IT Admin coordination — get them to whitelist your sender IP and domain before campaigns go live 2.3 The Software Stack GoPhish — the phishing simulation engine (free, open source) Postfix — Mail Transfer Agent for actually sending emails OpenDKIM — dedicated DKIM signing service wired into Postfix Nginx — reverse proxy sitting in front of GoPhish Certbot / Let's Encrypt — free SSL certificates Ubuntu 24.04 LTS — OS for both the mail and web VMs Windows Server 2022 — for the Jumpbox VM Azure Application Gateway — public HTTPS endpoint with TLS termination Azure Bastion — secure RDP/SSH access without any public IPs on VMs Azure Standard Load Balancer — public static IP for the Postfix VM -------------------------------------------------------------------------------------------------------- 3. Architecture Overview Before we touched a single server, we mapped out exactly how everything connects. This step saves you so much headache later — trust me. Here's what we built: Traffic Flows at a Glance Phishing emails: GoPhish (VM1) → Postfix relay (VM2) → Target mail server → Employee inbox Employee clicks phishing link: Internet → Application Gateway (HTTPS) → Nginx (VM1 port 80) → GoPhish (port 8080) → Landing page Admin access to GoPhish: Azure Bastion → Jumpbox RDP → Browser → VM1 private IP:3333 VM Roles VM1 — GoPhish server + Landing page | No public IP (goes through Application Gateway) | GoPhish, Nginx, Certbot VM2 — Postfix mail relay | Public IP via Load Balancer | Postfix, OpenDKIM Jumpbox — Admin access via RDP | No public IP (via Azure Bastion) | Windows Server 2022, Browser Why two VMs instead of one? Great question. GoPhish can send email directly — but using it as its own mail server is unreliable for deliverability and harder to lock down. Separating the concerns means VM1 handles web/landing pages, and VM2 is a dedicated Postfix relay that only VM1 can talk to. If the sending IP ever gets blacklisted, you just replace VM2 without touching the GoPhish server. Clean separation, easy recovery. -------------------------------------------------------------------------------------------------------- 4. Domain Setup on Hostinger 4.1 Why Hostinger? We chose Hostinger as our domain registrar because it's cheap, has a clean DNS interface, and supported the domain ( Because of Region constraints). We registered two domains — one for the sender (email FROM address) that points to VM2's Load Balancer IP, and one for the landing page links that points to the Application Gateway IP. Fun fact: we originally started with GoDaddy and abandoned it. Their registration process was slower and more expensive. Hostinger had both domains live in under 15 minutes. 4.2 DNS Records We Added For the sender domain — the one that goes in the email FROM address: A record → @ → [POSTFIX-LB-PUBLIC-IP] — root domain points to your mail server IP A record → mail → [POSTFIX-LB-PUBLIC-IP] — mail subdomain points to same IP MX record → @ → mail.[SENDER-DOMAIN] (priority 10) — mail routing TXT record → @ → v=spf1 ip4:[POSTFIX-LB-PUBLIC-IP] -all — SPF authorizes your IP to send TXT record → _dmarc → v=DMARC1; p=quarantine; rua=mailto:dmarc@[SENDER-DOMAIN] — DMARC policy TXT record → mail._domainkey → v=DKIM1; k=rsa; p=[DKIM PUBLIC KEY] — DKIM email signing For the landing domain — the one in the phishing link: A record → @ → [APPLICATION-GATEWAY-PUBLIC-IP] — points to your Azure Application Gateway DNS propagated in about 15-30 minutes on Hostinger. You can verify with: nslookup [YOUR-DOMAIN] # or dig [YOUR-DOMAIN] A dig mail._domainkey.[SENDING-DOMAIN] TXT dig _dmarc.[SENDING-DOMAIN] TXT -------------------------------------------------------------------------------------------------------- 5. Azure Infrastructure Setup 5.1 Resource Group and Virtual Network Everything lives in one Resource Group. Create it in your target Azure region. Then spin up a Virtual Network (VNet) with one subnet. All three VMs go into the same VNet so they can chat privately without any internet hop. VNet address space: 172.*.*.*/* Subnet: 172.*.*.*/* All three VMs must be in this subnet for private IP communication 5.2 Public IPs — Here's Where People Get Confused This trips up a lot of people with Azure, so let's be clear: VM1 (the GoPhish/landing page server) has NO public IP. It's completely private. The Application Gateway gets the public IP, terminates TLS (HTTPS), and forwards plain HTTP to the VM on its private IP. Way more secure — GoPhish is never directly exposed to the internet. If you need to test GoPhish before the Gateway is set up, you can temporarily assign a public IP to VM1, then remove it once the Gateway is ready. VM2 (the Postfix mail relay) DOES have a public IP — but through an Azure Standard Load Balancer, not directly on the NIC. Why a Load Balancer and not a direct public IP? Two reasons: if you need to replace the VM later, the public IP stays (you just re-add the new VM to the backend pool). And the Standard Load Balancer lets you set a DNS name label on the public IP — which you need for PTR/RDNS, which mail deliverability depends on. Important Azure gotcha: Standard Load Balancers do NOT allow outbound internet by default. You'll need to add an outbound rule to let VM2 reach the internet on port 25 for sending mail. 5.3 Creating the VMs VM1 — GoPhish + Landing Page Server: OS: Ubuntu 24.04 LTS Size: Standard B2s (2 vCPUs, 4 GB RAM) Private IP: * Public IP: None — access via Application Gateway (public) or Bastion (admin) NSG: Allow port 22 from Jumpbox IP only; allow port 80 from Application Gateway subnet only; allow port 3333 from VNet only VM2 — Postfix Mail Relay: OS: Ubuntu 24.04 LTS Size: Standard B2als v2 (2 vCPUs, 4 GB RAM) Private IP: * Public IP: via Azure Standard Load Balancer (static) — do NOT assign a public IP directly to the NIC NSG: Allow port 25 inbound from VM1 private IP only; allow port 25 outbound to internet; allow port 22 from VNet for SSH 5.4 Setting PTR / RDNS on the Load Balancer Public IP To set the PTR record, go to: Azure Portal → Load Balancer public IP → Configuration → DNS name label → enter a label (e.g. your-mail-label). Azure sets the PTR to: [YOUR-IP] → your-mail-label.[region].cloudapp.azure.com. Make your Postfix myhostname match this FQDN exactly (see Section 10). 5.5 SSH Between VMs # On VM1, generate SSH key and copy public key to VM2 ssh-keygen -t ed25519 -f ~/.ssh/vm2_key ssh-copy-id -i ~/.ssh/vm2_key.pub user@[VM2-PRIVATE-IP] # Test SSH access from VM1 to VM2 ssh -i ~/.ssh/vm2_key user@[VM2-PRIVATE-IP] -------------------------------------------------------------------------------------------------------- 6. The Jumpbox VM — Why It Exists and How to Use It 6.1 The Problem: GoPhish Admin Panel is Localhost-Only Here's the deal: by design, GoPhish binds its admin panel to 127.0.0.1:3333 — you can only reach it from the VM itself. VM1 has no public IP. So how do you actually use the GoPhish UI? You've got two options: Option A: SSH Tunnel (fine if you're comfortable on the command line) ssh -L 3333:127.0.0.1:3333 user@[VM1-PRIVATE-IP] # Then open browser on your machine: https://localhost:3333 This tunnels port 3333 from VM1 through your SSH connection to your local machine. Works for solo use, but it's awkward for a team — you'd need to distribute SSH keys everywhere. Option B: Jumpbox VM (what we built — much better for teams). A Jumpbox is a VM inside the same VNet as your other VMs. It has no public IP, but you access it via Azure Bastion — Microsoft's managed RDP/SSH gateway that runs inside Azure without exposing any port to the internet. 6.2 Setting Up the Jumpbox Step 1: Enable Azure Bastion. In the Azure portal, go to your VNet → Bastion. Click Enable. Azure creates a dedicated AzureBastionSubnet (/27 minimum) and a Bastion host with its own public IP. Takes about 5-10 minutes. Step 2: Create the Windows Server Jumpbox VM: OS: Windows Server 2022 Datacenter Size: Standard_B2s (2 vCPUs, 4 GB RAM) Public IP: None — that's the whole point VNet/Subnet: Same VNet as VM1 and VM2 NSG: No inbound internet rules needed — Bastion handles access Step 3: RDP in via Azure Bastion. In the Azure portal, go to your Jumpbox VM → Connect → Bastion → enter your Windows admin credentials. Azure opens a browser-based RDP session. No RDP client needed. Step 4: Once inside the Jumpbox, open Edge or Chrome and go to: https://[VM1-PRIVATE-IP]:3333 Accept the self-signed certificate warning (GoPhish uses a self-signed cert on the admin panel by default). You're in. Why Windows for the Jumpbox? A full GUI, Edge browser, and other admin tools all ready to go. For a pure SSH jumpbox to manage Linux VMs, Ubuntu works fine — but for accessing GoPhish's web UI, Windows is just easier. -------------------------------------------------------------------------------------------------------- 7. VM1: Installing and Running GoPhish 7.1 Download and Install GoPhish SSH into VM1 and run: cd /opt sudo wget https://github.com/gophish/gophish/releases/download/v0.12.1/gophish-v0.12.1-linux-64bit.zip sudo apt install unzip -y sudo unzip gophish-v0.12.1-linux-64bit.zip -d gophish sudo chmod +x /opt/gophish/gophish 7.2 Configure GoPhish Edit the config file at /opt/gophish/config.json: { "admin_server": { "listen_url": "localhost:3333", "use_tls": true, "cert_path": "certs/admin.crt", "key_path": "certs/admin.key" }, "phish_server": { "listen_url": "0.0.0.0:8080", "use_tls": false, "cert_path": "example.crt", "key_path": "example.key" }, "db_name": "sqlite3", "db_path": "gophish.db", "migrations_prefix": "db/db_", "contact_address": "", "logging": { "filename": "", "level": "" } } Key points to note: the admin_server listens on localhost only — never expose this to the internet. The phish_server listens on 0.0.0.0:8080 — Nginx will proxy to this. The use_tls is false on the phish server because Nginx and the Application Gateway handle HTTPS termination. Double-TLS would break the proxy chain. And heads up: after editing config.json, always do a full restart with sudo systemctl restart gophish. Reloading does not apply all config changes. 7.3 Generate Self-Signed Cert for Admin Panel sudo mkdir -p /opt/gophish/certs sudo openssl req -newkey rsa:4096 -nodes \ -keyout /opt/gophish/certs/admin.key \ -x509 -days 365 -out /opt/gophish/certs/admin.crt \ -subj "/CN=gophish-admin" 7.4 Run GoPhish as a systemd Service Create /etc/systemd/system/gophish.service: [Unit] Description=GoPhish Phishing Simulation After=network.target [Service] Type=simple User=root WorkingDirectory=/opt/gophish ExecStart=/opt/gophish/gophish Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target sudo systemctl daemon-reload sudo systemctl enable gophish sudo systemctl start gophish sudo systemctl status gophish 7.5 First Login On first run, GoPhish prints a temporary password in the logs. Grab it with: sudo journalctl -u gophish -f | grep "Please login" Log in via the Jumpbox browser, accept the cert warning, use the temp credentials, and set a strong password immediately. Don't skip that step. -------------------------------------------------------------------------------------------------------- 8. Nginx as a Reverse Proxy GoPhish runs on port 8080. Nginx sits in front on port 80 and proxies traffic to it. This also lets you serve static pages (like your awareness page) directly from Nginx without going through GoPhish. sudo apt install nginx -y Create /etc/nginx/sites-available/gophish: server { listen 80; server_name [LANDING-DOMAIN]; location / { proxy_pass http://localhost:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Serve awareness page for any path GoPhish doesn't recognize error_page 404 =200 @awareness; } # Named location fix — must use this pattern location @awareness { root /var/www/html; try_files /awareness.html =200; } location /awareness { root /var/www/html; try_files /awareness.html =404; } } sudo ln -s /etc/nginx/sites-available/gophish /etc/nginx/sites-enabled/ sudo nginx -t sudo systemctl reload nginx Two critical things to know here: First — do NOT add an HTTP-to-HTTPS redirect in Nginx when you're using an Azure Application Gateway. The Application Gateway terminates TLS and sends plain HTTP to port 80 on your VM. If Nginx redirects port 80 to HTTPS, you get an infinite redirect loop: Gateway → port 80 → Nginx 301 → HTTPS → Gateway → port 80 → repeat forever. It will drive you nuts. Second — the error_page 404 =200 @awareness must use a named location (@awareness), not a direct file path. Using a file path causes Nginx's internal try_files logic to override the 200 status and return 404 anyway. -------------------------------------------------------------------------------------------------------- 9. SSL Certificate with Let's Encrypt A trusted SSL cert on the landing domain is essential. If the browser shows a certificate warning, employees will immediately know something is off and the drill is blown. sudo apt install certbot python3-certbot-nginx -y Method 1 — Nginx plugin (use this if VM1 is temporarily accessible on port 80 before the Application Gateway is in place): sudo certbot --nginx -d [LANDING-DOMAIN] Method 2 — DNS challenge (use this when VM1 has no public IP, which is our standard setup): sudo certbot certonly --manual --preferred-challenges dns \ -d [LANDING-DOMAIN] Certbot will ask you to add a _acme-challenge TXT record to your domain in Hostinger. Add it, wait 30-60 seconds for propagation, then press Enter. The cert goes to /etc/letsencrypt/live/[LANDING-DOMAIN]/. Set up auto-renewal: sudo systemctl enable certbot.timer sudo systemctl start certbot.timer systemctl list-timers | grep certbot -------------------------------------------------------------------------------------------------------- 10. VM2: Setting Up Postfix as a Mail Relay Postfix on VM2 is a relay-only MTA. GoPhish submits mail to Postfix on port 25, and Postfix delivers it to the recipient's mail server. Postfix never accepts mail from the internet — only from VM1's private IP. That's the whole security model here. 10.1 Install Postfix sudo apt update sudo apt install postfix -y # When prompted: choose "Internet Site", enter your sender domain as the mail name 10.2 Harden Postfix Edit /etc/postfix/main.cf: myhostname = mail.[SENDER-DOMAIN] mydomain = [SENDER-DOMAIN] myorigin = $mydomain inet_interfaces = all inet_protocols = ipv4 # Only accept mail from VM1's private IP — this is critical mynetworks = 127.0.0.0/8, [VM1-PRIVATE-IP]/32 # No local delivery — relay only mydestination = local_transport = error:local delivery disabled # Strip the server banner — hides Postfix version from mail headers smtpd_banner = $myhostname ESMTP # TLS settings smtp_tls_security_level = may smtpd_tls_security_level = may # Recipient restrictions — only permit from mynetworks, reject all else smtpd_recipient_restrictions = permit_mynetworks, reject # Rate limiting — prevent abuse if VM is compromised anvil_rate_time_unit = 60s smtpd_client_message_rate_limit = 200 smtpd_client_connection_rate_limit = 50 relayhost = smtp_sasl_auth_enable = no sudo systemctl restart postfix sudo systemctl enable postfix # Test from VM1 that it can reach VM2 on port 25 telnet [VM2-PRIVATE-IP] 25 Azure port 25 heads-up: Azure blocks outbound port 25 by default on new subscriptions to prevent spam. You need to submit a request to Microsoft to enable it. Go to the Azure portal → New support request → Subscription and billing → Subscription management → Enable outbound SMTP emails. In our experience this took 24-48 hours to get approved. 10.3 DKIM Signing with OpenDKIM Rather than relying on GoPhish's built-in DKIM, we use OpenDKIM as a dedicated Milter service wired into Postfix. This is the right production approach and gives you proper control over key management. sudo apt install opendkim opendkim-tools -y # Generate key pair opendkim-genkey -s mail -d [SENDER-DOMAIN] # Creates mail.private and mail.txt # Store private key sudo mkdir -p /etc/opendkim/keys/[SENDER-DOMAIN] sudo mv mail.private /etc/opendkim/keys/[SENDER-DOMAIN]/mail.private sudo chown opendkim:opendkim /etc/opendkim/keys/[SENDER-DOMAIN]/mail.private sudo chmod 600 /etc/opendkim/keys/[SENDER-DOMAIN]/mail.private Take the public key from mail.txt and add it as a DNS TXT record in Hostinger: Name = mail._domainkey.[SENDER-DOMAIN], Value = v=DKIM1; k=rsa; p=[YOUR-PUBLIC-KEY-HERE] Wire OpenDKIM to Postfix by adding to main.cf: smtpd_milters = inet:localhost:port non_smtpd_milters = inet:127.0.0.1:port milter_default_action = accept sudo systemctl enable opendkim sudo systemctl start opendkim sudo systemctl restart postfix 10.4 Start VM2 Before Each Campaign To save Azure costs, we deallocate VM2 between sessions. Just remember to start it in the Azure portal before any campaign send — it takes about 2-3 minutes to boot. Postfix and OpenDKIM auto-start via systemd. -------------------------------------------------------------------------------------------------------- 11. Azure Application Gateway Configuration The Application Gateway is the public HTTPS endpoint for the landing page. It receives internet traffic, terminates TLS, and forwards plain HTTP to VM1 on port 80. Here are the components you need to configure: Frontend IP: Static public IP — this is your APPLICATION-GATEWAY-PUBLIC-IP Backend Pool: VM1 private NIC IP Backend HTTP Setting: Protocol HTTP, Port 80 — the AG sends plain HTTP to VM1 Health Probe: HTTP, path /, match status 200-404 — GoPhish returns 404 for unknown paths, so you MUST include 404 or the probe marks the backend unhealthy and stops forwarding HTTP Listener: Port 80 — needed for plain HTTP and Certbot HTTP challenges HTTPS Listener: Port 443 with SSL certificate — the AG handles TLS termination Routing Rules: Both HTTP and HTTPS forward to the backend pool 11.2 Upload the SSL Certificate Export the Let's Encrypt cert as a PFX file: sudo openssl pkcs12 -export \ -in /etc/letsencrypt/live/[LANDING-DOMAIN]/fullchain.pem \ -inkey /etc/letsencrypt/live/[LANDING-DOMAIN]/privkey.pem \ -out /tmp/landing-cert.pfx \ -passout pass:[YOUR-PFX-PASSWORD] Upload this PFX in the Azure portal under Application Gateway → Listeners → HTTPS Listener → Certificate. -------------------------------------------------------------------------------------------------------- 12. DNS and Email Authentication (SPF, DKIM, DMARC, PTR) This is the most commonly skipped step — and it's exactly why most phishing simulation emails end up in spam. All four of these have to be configured correctly. Let's go through each one. SPF (Sender Policy Framework) SPF tells receiving mail servers which IPs are allowed to send email for your domain. Add this TXT record: v=spf1 ip4:[POSTFIX-LB-PUBLIC-IP] -all Use -all (hard fail), NOT ~all (soft fail). The hard fail is required for inbox delivery to most corporate mail servers. DKIM (DomainKeys Identified Mail) DKIM adds a cryptographic signature to every outgoing email. The receiving server verifies it against the public key in DNS — confirming the email was sent by an authorized sender and wasn't modified in transit. If you're using OpenDKIM (recommended), add this DNS TXT record: Name: mail._domainkey.[SENDER-DOMAIN] Value: v=DKIM1; k=rsa; p=[YOUR-PUBLIC-KEY-HERE] DMARC DMARC ties SPF and DKIM together and tells receivers what to do with failures: Name: _dmarc.[SENDER-DOMAIN] Value: v=DMARC1; p=quarantine; rua=mailto:dmarc@[SENDER-DOMAIN]; adkim=r; aspf=r PTR / Reverse DNS Many mail servers (especially corporate ones) check that the sending IP's PTR record matches the hostname in the SMTP banner. Set the DNS name label on the Load Balancer's public IP, then set your Postfix myhostname to match that FQDN exactly. Verify All Records # SPF nslookup -type=TXT [SENDER-DOMAIN] # DKIM nslookup -type=TXT mail._domainkey.[SENDER-DOMAIN] # DMARC nslookup -type=TXT _dmarc.[SENDER-DOMAIN] # PTR nslookup [POSTFIX-LB-PUBLIC-IP] Use mxtoolbox.com for a visual check — it tells you exactly what passes and what fails. Also register your sender domain in Google Postmaster Tools to monitor sending reputation over time. -------------------------------------------------------------------------------------------------------- 13. Configuring GoPhish — Sending Profile, Templates, Landing Pages, Campaigns 13.1 Sending Profile The Sending Profile tells GoPhish how to relay email. Go to Sending Profiles → New Profile: Name: Anything descriptive SMTP From: name@[SENDER-DOMAIN] — bare email address, NO display name here. Postfix rejects display names in MAIL FROM (e.g. "CERT " causes Postfix to produce invalid SMTP syntax). The display name goes in the email template's From field instead. Host: [VM2-PRIVATE-IP]:25 — your Postfix relay on its private IP Username / Password: Leave blank — Postfix accepts without auth from VM1's IP Ignore Certificate Errors: Yes Email Headers: Add a custom header to help IT identify drill emails — e.g. X-Company-Drill:1999 . This header is critical: it enables the SCL = -1 whitelist rule and lets IT set up auto-responses when employees report the email. 13.2 Email Templates Go to Email Templates → New Template. We created a realistic Microsoft-branded template. Effective scenarios include: Microsoft Unusual Sign-in Activity, IT helpdesk password expiry, HR payroll update, DocuSign signature request. Key principles: Subject: Creates urgency without being obvious — e.g. "Unusual sign-in activity on your Microsoft account" From (display name): This is where the display name goes — e.g. "Microsoft Account Team " HTML Body: Include {{.URL}} as the phishing link — GoPhish replaces this with a unique tracking URL per recipient at send time. Include {{.Tracker}} for open tracking (1x1 pixel). Use {{.FirstName}} for a higher click rate. Add Tracking Image: Check this box. Create separate templates for each language — a Japan employee getting an email in perfect japanese is far more convincing than a translated English one. 13.3 Landing Pages Go to Landing Pages → New Page: Capture Submitted Data: Yes (to track who interacted) Capture Passwords: NO — never capture real passwords in a phishing drill. Legally risky and unnecessary — you only need to know they would have submitted, not what password they would have used. Redirect To: Your awareness page URL — e.g. https://[LANDING-DOMAIN]/awareness HTML: Import from a site (GoPhish can clone any public webpage) or write custom HTML 13.4 Users and Groups (Target Lists) Go to Users and Groups → New Group. Import a CSV with headers: First Name,Last Name,Email,Position John,Smith,j.smith@[TARGET-DOMAIN],Engineer Create separate groups per region so you can run region-specific campaigns with language-appropriate templates. Always test with a small trusted group (1-5 people) before uploading the full employee list. Handle CSV files securely — delete from disk after import into GoPhish. 13.5 Campaigns Go to Campaigns → New Campaign: Name: Descriptive Wave 1 Email Template: Select the template for this region Landing Page: Select your awareness landing page URL: https://[LANDING-DOMAIN] — GoPhish automatically appends a unique per-recipient tracking token. The {{.URL}} variable in your template gets replaced with each recipient's unique link at send time. Sending Profile: Your Postfix relay profile Groups: Select the target group for this region Launch Date: Schedule it — stagger by region 24-48 hours apart to manage helpdesk load Send Emails By: Set a campaign window (e.g. 9 AM to 5 PM on working days) — GoPhish distributes sends evenly across this window automatically 13.6 Campaign Results Dashboard GoPhish shows you real-time stats: Sent — email delivered to the mail server Opened — tracking pixel loaded (email was opened) Clicked — recipient clicked the phishing link Submitted Data — recipient filled in and submitted the form Reported — recipient reported the email via your IT reporting button You can export all results as CSV for analysis. -------------------------------------------------------------------------------------------------------- 14. Getting Whitelisted — The IT Admin Process Even with perfect SPF/DKIM/DMARC, emails to corporate inboxes (Microsoft 365 / Exchange Online) will likely land in Junk unless you get whitelisted. This is actually correct behaviour — your phishing domain is new, unknown, and looks suspicious by design. Here's exactly what IT needs to do. 14.1 Two Steps — Both Are Required Step 1 — EOP Connection Filter: IP Allow List: Microsoft 365 Admin Center → Security → Email & Collaboration → Policies & Rules → Threat Policies → Anti-spam → Connection filter policy (Default) → IP Allow List → Add [POSTFIX-LB-PUBLIC-IP]. This skips connection-level blocking — but Exchange still runs content and spam analysis after this. That's why Step 2 is also mandatory. Step 2 — Mail Flow Rule: Set SCL = -1: Exchange Admin Center → Mail flow → Rules → New rule. Condition: Sender IP is in range [VM2-LB-PUBLIC-IP] OR sender domain is [SENDING-DOMAIN] OR message header X-[COMPANY]-PhishDrill = [YEAR] Action: Set the spam confidence level (SCL) to -1 SCL = -1 bypasses ALL spam filtering and delivers directly to inbox. Set this rule at higher priority than any existing anti-phishing or anti-spam rules. Why both steps? The IP Allow List alone prevents connection-level rejection but does NOT bypass EOP content filtering or Microsoft Defender anti-phishing policies. The SCL = -1 mail flow rule is the only reliable way to guarantee inbox delivery. 14.2 For Corporate Mail Gateways (Non-Microsoft) If your org uses a third-party gateway (Cisco IronPort, Proofpoint, Mimecast, etc.) in addition to or instead of EOP, that gateway also needs to whitelist your IP and domain. Raise a separate ticket with whichever team manages it, including the sender IP, sender domain, the custom X-header, and a reference to your security awareness programme authorization number. 14.3 Whitelist Ticket Template When raising a ticket with IT, provide all of this in one message to avoid back-and-forth: Subject: Whitelist Request — Security Awareness Phishing Drill Please whitelist the following for our authorized phishing drill: Sender IP: [POSTFIX-LB-PUBLIC-IP] Sender Domain: [SENDER-DOMAIN] Phishing Links: https://[LANDING-DOMAIN] Email Header: X-Company-Drill: 1999 Required Actions: 1. Add [IP] to EOP Connection Filter → IP Allow List 2. Create mail flow rule: Condition: header X-Company-PhishDrill = 2026 Action: set SCL = -1 -------------------------------------------------------------------------------------------------------- 15. End-to-End Testing Before you send a single email to a real employee, validate everything with a small trusted test group. Here's how to do it right. 15.1 Send a Test Email via GoPhish API Here's a quirk worth knowing: the /api/util/send_test_email endpoint ignores {"template": {"id": 2}} — it does NOT look up the template from the database. You have to pass the full template JSON inline: # Step 1: Fetch the full template object TEMPLATE=$(curl -sk https://127.0.0.1:3333/api/templates/[TEMPLATE-ID] \ -H "Authorization: Bearer [YOUR-API-KEY]") # Step 2: Send test email with full template inline curl -sk -X POST https://localhost:port/api/util/send_test_email \ -H "Authorization: Bearer [YOUR-API-KEY]" \ -H "Content-Type: application/json" \ -d "{ \"template\": $TEMPLATE, \"smtp\": { \"id\": 1, \"host\": \"[VM2-PRIVATE-IP]:25\", \"from_address\": \"name@[SENDER-DOMAIN]\", \"ignore_cert_errors\": true, \"headers\": [{\"key\": \"X-Company-PhishDrill\", \"value\": \"2026\"}] }, \"email\": \"[YOUR-TEST-EMAIL]\", \"url\": \"https://[LANDING-DOMAIN]\" }" 15.2 Full Test Checklist Email arrives in your test inbox (check both inbox and spam) Email headers show SPF=pass, DKIM=pass, DMARC=pass Custom header X-Company-PhishDrill: 2026 is present in raw source Clicking the link in the email opens the landing page at https://[LANDING-DOMAIN] Landing page loads over HTTPS with a valid certificate (padlock visible, correct domain) The awareness page loads correctly after click Back in GoPhish campaigns dashboard, the click event is recorded GoPhish records Email Sent, Email Opened, Clicked Link, Submitted Data for the test recipient PTR/RDNS verified: dig -x [VM2-LB-PUBLIC-IP] resolves to expected hostname Test delivery to a corporate email address once IT whitelisting is confirmed active 15.3 Check Email Headers In Gmail: click the three-dot menu on the email → "Show original". Look for: Received-SPF: pass Authentication-Results: dkim=pass; spf=pass; dmarc=pass X-Company-Drill: 1999← your custom header confirming it's being added -------------------------------------------------------------------------------------------------------- 16. Common Issues and Fixes We Encountered Let me save you some pain. Here are every problem we actually ran into and exactly how we got past each one. Issue 1: Landing page shows ERR_TOO_MANY_REDIRECTS Cause: Nginx was redirecting port 80 → HTTPS. The Application Gateway sends HTTP to port 80 after terminating TLS. The redirect creates an infinite loop. Fix: Remove the HTTP→HTTPS redirect from Nginx. Both port 80 and 443 server blocks should serve content directly. The Application Gateway handles TLS before traffic reaches Nginx. Issue 4: Direct test URL returns 404 even though nginx error_page is set Cause: Using error_page 404 =200 /awareness.html triggers an internal redirect that hits the try_files fallback, which overrides the =200 status code. Fix: Use a named location in Nginx: error_page 404 =200 @awareness; location @awareness { root /var/www/html; try_files /awareness.html =200; } Issue 2: GoPhish config changes don't take effect Cause: After editing config.json, a SIGHUP or reload does not apply all changes. Fix: Always do a full restart: sudo systemctl restart gophish. Issue 3: Email goes to spam on Gmail (new domain) Cause: Brand-new domain and IP with no sending history = zero reputation. This is expected and not a configuration error. SPF/DKIM/DMARC can all pass but Gmail's reputation filter still soft-blocks new senders. Fix: Set PTR/RDNS for the sending IP (Azure DNS label on the Load Balancer public IP) Change SPF from ~all to -all Register your sender domain in Google Postmaster Tools to monitor reputation For corporate targets: it doesn't matter — the IT whitelist rule (SCL = -1) overrides spam scoring entirely -------------------------------------------------------------------------------------------------------- 17. Sending Strategy for Large Campaigns Sending 3,emails all at once is a genuinely bad idea. It triggers spam filters, looks unnatural, can overwhelm a small Postfix server, and employees warn each other in Slack. Here's what we actually did. 17.1 Batch by Region, Not Blast Separate campaign per region — allows localised templates and independent per-region reporting Around 1,000 recipients per region GoPhish has a built-in scheduler — set a campaign window (e.g. 9 AM to 5 PM on working days) and it automatically spaces the sends Target approximately 200 emails/day per region — well within deliverability limits for a new domain 5-day campaign window per region Stagger region launches by 24-48 hours — prevents helpdesk overload from simultaneous report spikes 17.2 Language-Specific Templates Use native-language templates for each region. A employee receiving an email in perfect German is far more convincing than a translated English template. Build separate templates for countries 17.3 Scenario Realism The best phishing scenarios mimic tools the target actually uses daily: Microsoft 365 login alerts, IT helpdesk notifications, HR payslip portals, shipping notifications. The more generic the scenario, the lower the click rate — which defeats the whole point of the drill. 17.4 Brief the IT Helpdesk in Advance Give the helpdesk the auto-response text to send back to employees who report the suspicious email: 17.5 Key Metrics to Report After the Drill These four numbers tell you exactly where additional training investment is needed and give you a measurable baseline for next time: Open rate — % of employees who opened the email Click-through rate — % who clicked the phishing link Submission rate — % who interacted with the landing page Reporting rate — % who reported the suspicious email to SOC/Incident Team -------------------------------------------------------------------------------------------------------- Key Takeaways The two-VM architecture (GoPhish + Postfix separated) is way more robust and maintainable than cramming everything onto one server. Azure Application Gateway is the right tool when your VM has no public IP — it handles TLS cleanly and keeps the GoPhish server off the internet entirely. The Jumpbox is essential for accessing GoPhish's localhost-only admin panel without exposing it. The custom X-header is what makes the IT whitelist rule reliable and gives you a unique signal to auto-respond to employee reports. And legal sign-off is non-negotiable — get it in writing before you touch a single config. Got questions or hit a different issue? Drop a comment below — happy to help troubleshoot. ----------------------------------------------Dean-------------------------------------------------