In-Cloud IR — Memory, Containers, and the Metadata Service Attack
- 2 days ago
- 5 min read
This final article covers the advanced scenarios in AWS incident response — acquiring memory from cloud instances, forensicating containers, understanding the metadata service attack, and making the decision between in-cloud and on-premises analysis. By the end of this series, you have a complete picture of how to investigate any AWS incident from first alert to evidence report.
In-Cloud vs. On-Premises Analysis
▸ Recommendation: Always analyse in-cloud first. Download to on-prem only when regulation requires it or tooling demands it.
The fundamental question in cloud IR: should you download evidence to an on-prem forensics lab, or do the analysis in the cloud?
The answer has shifted dramatically in recent years. Doing analysis in-cloud is almost always faster, cheaper, and more practical.
In-cloud analysis: No egress bandwidth costs, no waiting for terabytes to download, spin up forensic EC2 instance in same region in minutes, use cloud-native tooling
On-prem analysis: Required when regulation demands it, when your tooling only runs on-prem, or when you need physical custody of evidence
💡 IR Tip: Start in-cloud. Build a DFIR AMI (Amazon Machine Image) with your tools pre-installed — Volatility, Autopsy, YARA, log parsers — and keep it ready to deploy in any region within minutes.
--------------------------------------------------------------------------------------------------
Building a DFIR EC2 Image
The DFIR AMI is your forensic workstation template in the cloud. Build it once, store it, deploy it in any region within 5 minutes.
Build process:
(1) Launch a clean EC2 instance (Ubuntu 22.04 LTS recommended),
(2) Install all your forensic tools: Volatility 3, AVML, bulk_extractor, log2timeline/plaso, YARA, Autopsy CLI tools, aws cli, python3 libraries,
(3) Create an AMI from this instance (EC2 → Actions → Create Image),
(4) Copy the AMI to all regions you might respond in (AMIs are regional).
--------------------------------------------------------------------------------------------------
Systems Manager (SSM) — Remote Access Without a Bastion
AWS Systems Manager Session Manager gives you shell access to EC2 instances without needing SSH, a bastion host, or open port 22. It tunnels the session through AWS's own infrastructure.
This is critical for IR because:
you don't need to open any network ports on the compromised instance, all session activity is logged to CloudTrail and optionally to S3, you can access instances in private subnets with no internet access.
Requirements: SSM Agent must be installed (pre-installed on most AWS AMIs since 2022), the instance must have an IAM role with the AmazonSSMManagedInstanceCore policy. CLI access: aws ssm start-session --target i-0yourinstanceid
--------------------------------------------------------------------------------------------------
Linux Memory Acquisition — AVML
Memory acquisition from cloud Linux instances requires a different approach than on-premise.
The old tool (Margarita Shotgun, which used a kernel module to dump memory) stopped working on modern kernels.
The current standard for AWS Linux memory acquisition is AVML (Acquire Volatile Memory for Linux), developed by Microsoft and supported by AWS.
⚡ Update (2024): AVML is the current recommended tool for Linux memory acquisition in AWS. It works as a userspace tool — no kernel module required. It reads /dev/mem or process maps directly. It's included in the AWS Forensics AMI reference implementations.
AVML acquisition process via SSM:
(1) Connect via SSM: aws ssm start-session --target i-0compromisedinstance,
(2) Install AVML if not present: curl -L https://github.com/microsoft/avml/releases/latest/download/avml -o avml && chmod +x avml, (3) Acquire memory: sudo ./avml memory.lime,
(4) Copy to S3: aws s3 cp memory.lime s3://your-dfir-bucket/case-2024-047/. The .lime extension is important — Volatility expects LiME format for analysis.
💡 IR Tip: After AVML acquisition, validate the image with Volatility before you do anything else: python3 vol.py -f memory.lime banners. This confirms the memory image is valid and tells you the exact kernel version, which you'll need to find the right Volatility profile.
--------------------------------------------------------------------------------------------------
Windows Memory Acquisition in the Cloud
Windows EC2 memory acquisition is more complex.
Options:
(1) WinPMem — open source, runs as a driver, requires admin access. Works on Windows 10/Server 2019 and later.
(2) Magnet RAM Capture — free tool from Magnet Forensics, GUI-based, reliable.
(3) F-Response — enterprise remote memory acquisition, requires a F-Response service account.
For cloud environments, WinPMem via SSM Run Command is the most practical: aws ssm send-command --instance-ids i-0yourwindowsinstance --document-name 'AWS-RunPowerShellScript' --parameters commands=['winpmem_mini_x64_rc2.exe memory.raw']
--------------------------------------------------------------------------------------------------
Container Forensics — ECS and EKS
AWS has two managed container services: ECS (Elastic Container Service) — AWS's own container orchestrator, simpler to manage. EKS (Elastic Kubernetes Service) — managed Kubernetes, more complex, industry standard.
Container forensics challenge: containers are ephemeral. When a container task is stopped, the container filesystem is gone.
Evidence preservation for containers:
(1) Don't stop the container — use kubectl exec (EKS) or aws ecs execute-command (ECS) to get a shell into the running container,
(2) Dump the container filesystem while running: docker export containerid > container_fs.tar, (3) If running on EC2-backed ECS/EKS (not Fargate), capture the host EC2 memory and disk as well — the container filesystem exists on the host.
💡 IR Tip: For EKS investigations, deploy a forensic pod into the same namespace as the suspicious pod. Use kubectl cp to extract files from the suspect container. This avoids touching the host and minimises your investigation footprint.
--------------------------------------------------------------------------------------------------
The IMDS Attack — Stealing Cloud Credentials via the Metadata Service
The Instance Metadata Service (IMDS) is an internal AWS service available to every EC2 instance at the fixed IP address 169.254.169.254. It provides the instance with information about itself — instance ID, region, AMI ID, and critically, the temporary credentials for the IAM role attached to the instance.
The attack:
if a web application running on an EC2 instance is vulnerable to SSRF (Server-Side Request Forgery), an attacker can make the application fetch http://169.254.169.254/latest/meta-data/iam/security-credentials/RoleName — this returns a valid, active ASIA-prefixed temporary credential for the instance's IAM role. The `attacker can then use these credentials from their own machine to make AWS API calls as that instance.
Example
IMDS credential response: { 'AccessKeyId': 'ASIA...', 'SecretAccessKey': 'xxxxxxx', 'Token': 'IQoJb...', 'Expiration': '2024-03-15T14:00:00Z' }.
These credentials expire, but the attacker can refresh them as long as they have SSRF access.
In CloudTrail, IMDS credential abuse looks like: API calls with ASIA-prefixed key ID, from an external IP (not an internal AWS IP), for a role that normally only makes internal API calls. The combination of ASIA token + unexpected source IP = stolen IMDS credentials.
IMDSv2 — The Fix for IMDS Credential Theft
AWS introduced IMDSv2 to mitigate the SSRF-to-IMDS attack. IMDSv2 requires a two-step process:
(1) PUT request to get a session token: curl -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' http://169.254.169.254/latest/api/token,
(2) Use that token in subsequent GET requests: curl -H 'X-aws-ec2-metadata-token: TOKEN' http://169.254.169.254/latest/meta-data/. SSRF attacks typically can only make GET requests (not PUT with custom headers), so the PUT requirement blocks the attack.
⚡ Update (2024): AWS now enforces IMDSv2 by default on all NEW EC2 instances launched after October 2023. However, existing instances that were launched before this change still use IMDSv1 by default unless explicitly updated. Always check: aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId, MetadataOptions.HttpTokens]' -- if HttpTokens is 'optional', the instance still allows IMDSv1.
AWS IR Readiness Checklist
CloudTrail trail configured at org level with data events enabled
VPC flow logs enabled for all production VPCs
S3 server access logs enabled for sensitive buckets
Load balancer access logs enabled
Route 53 resolver query logging enabled
GuardDuty enabled at org level
AWS Detective enabled and connected to GuardDuty
DFIR AMI built and copied to all regions
Evidence S3 bucket configured with WORM and encryption
IR Lambda functions and Step Functions workflows deployed and tested
IMDSv2 enforced on all EC2 instances (HttpTokens = required)
SSM Agent enabled on all instances for agentless remote access
--------------------------------------------------------------------------------------------------------
Series Complete
From understanding AWS organisations and IAM, through CloudTrail investigation and hunting, networking, S3, automated detection with GuardDuty and Detective, event-driven automation, and in-cloud IR — you now have a complete framework for investigating any AWS incident.
This is a complete AWS Incident Response (IR) article.
I’ve tried my best to cover the topic based on my knowledge, experience, and learning. There may still be some mistakes or areas that could be improved, but I hope you find it useful and enjoy reading it.
If you notice anything incorrect or have suggestions for improvement, please feel free to message me or let me know. I’d really appreciate your feedback! 🙌
----------------------------------------------------------------------------------------------------
Comments