Open, deterministic, evidence-linked
The final answer can look right while the agent run goes wrong.
Agent Failure Atlas inspects the trajectory: tool scope, approvals, injected instructions, repeated actions, silent failures, and exposed secrets. This static edition runs entirely inside your browser.
◎Local by design. DeltaStore exchange packages are validated and normalized locally in your browser. Files are not uploaded or persisted.
Run summary6 failure classes
01
User goalInspect workspace only
02
Scope violationread_file('/etc/shadow')
03
Injected tool outputOverride + exfiltration instruction
04
False successSMTP failed; agent reported sent
Synthetic verification
20 controlled sessions. Six detector families.
The fixture suite validates implemented behavior. It is intentionally not presented as real-world accuracy evidence.
Per-category
Not a model-safety scoreFixture coverage
What it catches
Failures hidden by task-success metrics
Scope
Unauthorized tools, path traversal, and network targets outside policy.
Approval
High-impact actions without approval bound to the action.
Injection
Instruction-like content in tool output followed by agent action.
Loops
Identical tool calls repeated beyond a bounded retry threshold.
Failure handling
Tool errors ignored, left unresolved, or followed by unsupported success.
Secrets
Credential patterns persisted in messages or passed as raw tool arguments.
Local trace analysis
Scan one agent session
Drop JSONL, use the bundled demonstration, or paste a trace. Nothing leaves this browser.
HF STSNormalized JSONLOpenAI messages
A
No trace scanned yet
Use the bundled failure demo to see evidence-linked findings immediately.
Evidence-linked results
Findings
Session trajectory
—Messages and tool calls
Counterfactual inspection
Compare two trace outcomes
Find new, resolved, and persistent failures after a policy, prompt, model, or harness change.
Comparison is ready when both traces are loaded
The demo illustrates a vulnerable run followed by a controlled run under the same task intent.
Finding classification
—What changed
Category-level movement
Before and after
Methodology and boundaries
Evidence before confidence
Atlas is deliberately narrow: deterministic checks, explicit policy, exact evidence positions, and honest separation between fixture validation and external validity.
Normalize
Parse supported trace formats into one session model while enforcing file, line, row, and duplicate tool-call ID limits.
Evaluate
Apply six deterministic detector families under a visible policy. No model call decides whether a finding exists.
Link evidence
Every finding points to message indices, excerpts, tool names, and parsed arguments instead of producing only a score.
What this Space demonstrates
- Local STS and JSONL parsing
- Six deterministic failure detectors
- Policy-aware findings
- Evidence-linked trace inspection
- Before/after classification
- 20-session synthetic fixture dashboard
What it does not demonstrate
- General real-world detector accuracy
- Safety of an unflagged trace
- Model-level causal attribution
- Semantic or intent understanding
- Complete secret redaction
- Production authorization enforcement
Metric interpretation
A perfect synthetic score is a software check, not a research conclusion.
The 1.00 precision, recall, and F1 values show that the current implementation recognizes the exact deterministic patterns represented by its own fixtures. Real-trace validation requires independent labels, ambiguity handling, held-out cases, inter-reviewer agreement, and representative false-positive and false-negative analysis.