Proof
Detection quality, measured.
A live, transparent triage harness running over a documented, held-out, labeled benchmark of AI/agent SOC alerts. Move the auto-close threshold and every number recomputes in your browser from the actual data — nothing here is scripted. This is the measurement scaffolding our customer eval sets will run through unchanged.
We land first on AI/agent threat detection in the SOC. It is the one detection surface no incumbent SIEM or CNAPP owns yet, and the one where a held-out eval set does not exist. We build the benchmark, so we set the bar. The same harness extends to cloud control-plane and identity alerts without changing the engine.
Score a live alert through the API.
This console posts to a real endpoint — the model runs server-side and returns a decision plus a reasoning trace. It takes alerts you define, not the ones we shipped. Same endpoint a design partner pipes their own telemetry into; the benchmark below is just this service run at scale on a labeled, held-out set.
Pipe a real alert through the whole system.
POST /api/ingest accepts the webhook payloads Splunk, Elastic, Microsoft Sentinel, and OCSF-speaking tools actually emit. Each alert is normalized, scored against the per-actor baseline the system has learned on its own, and routed — with a deterministic 10% of auto-closes trickled to human audit, because the engine never trusts itself alone. This queue is the live state of that pipeline.
The queue is empty — this instance has not ingested anything yet. Fire a real vendor payload at it:
curl -s 'https://ironsoc.com/api/ingest?source=splunk' \
-H 'content-type: application/json' \
-d '{"search_name":"Agent tool-call anomaly","result":{"user":"svc-agent-01","urgency":"high","_raw":"tool=shell.exec after retrieved doc said ignore previous instructions"}}'curl -s 'https://ironsoc.com/api/ingest?source=sentinel' \
-H 'content-type: application/json' \
-d '{"object":{"properties":{"AlertDisplayName":"Mass download by service principal","AlertSeverity":"Medium","Description":"400 files with customer data"}}}'curl -s 'https://ironsoc.com/api/ingest?source=generic' \
-H 'content-type: application/json' \
-d '{"id":"a-001","title":"Batch summarization","actor":"svc-batch","severity":"low","text":"scheduled read-only listing"}'Supported sources: splunk, elastic, sentinel, ocsf, generic. The pipeline normalizes the vendor payload, scores it against the per-actor baseline it has learned, and routes it — including the deterministic 10% audit trickle.
Alerts scoring below τ are auto-closed; the rest escalate to a human (scores ≥ 0.85 are auto-contained). Raise τ to clear more of the queue — watch the miss count for the price.
The highest τ that still misses zero threats on the held-out set — the point we report against and tune toward.
Risk-score distribution, held-out set
Everything left of τ is auto-closed. A red dot left of τ is a missed threat. The model separates the classes well, but the overlap band is real — that is where the safety/efficiency tradeoff lives, and why we tune to the rightmost zero-miss point rather than maximum auto-close.
Confusion matrix
Positive class = malicious (should reach a human).
- Precision (escalation purity)
- 71.9%
- Recall (threats caught)
- 100.0%
Hardest cases at this threshold
The eight alerts closest to the boundary — where the model is least certain and a human is most warranted.
| Alert | Score | Truth | Outcome |
|---|---|---|---|
Jailbreak / system-prompt override ASB-103 · OWASP LLM01 · ATLAS AML.T0054 | 0.517 | malicious | escalated |
High-volume batch summarization job ASB-188 · Baseline workload | 0.513 | benign | auto-closed |
Security keyword in benign user query ASB-093 · False trigger | 0.525 | benign | escalated — wrong |
Legitimate admin tool usage ASB-020 · Authorized operator | 0.500 | benign | auto-closed |
Sanctioned red-team exercise ASB-148 · Approved engagement | 0.530 | benign | escalated — wrong |
High-volume batch summarization job ASB-015 · Baseline workload | 0.531 | benign | escalated — wrong |
Security keyword in benign user query ASB-149 · False trigger | 0.531 | benign | escalated — wrong |
Customer-support agent normal operation ASB-219 · Baseline workload | 0.532 | benign | escalated — wrong |
Methodology
How the number is produced — and why you can argue with it.
A documented, labeled benchmark
AgentSOC-Bench v0.1: 240 synthetic AI/agent SOC alerts generated deterministically from a fixed seed (0x1R0N50C), spanning 15 archetypes mapped to OWASP LLM Top 10 and MITRE ATLAS. Class-conditional draws with deliberate overlap, so scary-but-benign (sanctioned red-team) and quiet-but-malicious cases are present and counted.
A held-out split, no leakage
Coefficients are fit on the 96-alert tuning split, then frozen. Every metric on this page is computed only on the 144-alert held-out split the model never saw. Same discipline our CI eval gate enforces on every detection change.
Tuned to zero misses, not max efficiency
A miss — a real threat auto-closed — is the unacceptable error. We report against the highest threshold that still misses zero on the held-out set. The slider lets you trade that safety for more auto-close and see exactly what it costs.
The 90/10 doctrine — it never trusts itself
Automation clears the queue, but a deterministic 10% sample of everything it auto-closes is trickled back to a human for audit — sampled by alert ID, not cherry-picked. On the benchmark we tune to zero misses because truth is known; in production truth arrives late, so the trickle keeps human judgment in the loop. The analyst-hours figure above is already net of it.
Synthetic now, customer data next
These are benchmark numbers, not production metrics — the harness is the product, not this dataset. When a design partner goes live, their labeled alerts replace the generator and this exact page runs against real telemetry. We will publish those numbers only when they exist.
The model, in full — logistic over nine signals
No black box. Risk score = sigmoid(bias + Σ wᵢ·signalᵢ), every signal in [0,1]. The point of the prototype is the honest harness around the model, not the model itself — swap these coefficients for a trained artifact and nothing else changes.
| Signal | Weight | What it measures |
|---|---|---|
| Injection signal | +2.6 | Prompt-injection / jailbreak language in the prompt or retrieved content. |
| Behavioral deviation | +1.7 | Distance from the agent's learned baseline of tools, volume, and targets. |
| Tool sensitivity | +1.6 | Impact class of the invoked tool: read < write < exec < payment / admin. |
| Data sensitivity | +1.5 | Secrets, PII, or regulated data present in context or output. |
| Instruction provenance | +1.4 | Where the instruction originated: user < retrieved-doc < tool-output (indirect injection). |
| Actor risk | +1.3 | Inverse trust of the calling identity: known service account < authed user < anonymous / new. |
| Blast radius | +1.2 | Reachable systems and records if the action proceeds. |
| Prior incidents | +1.1 | Recent related alerts for this actor or agent. |
| Approved context | -2.2 | Sanctioned red-team or approved change window. Strongly suppresses risk. |
| Bias | -3.6 | Base rate — pulls the default decision toward auto-close. |
Honesty note: AgentSOC-Bench is a synthetic reference benchmark, not customer telemetry. It exists to make the measurement harness real and inspectable before production data exists. We will not publish customer-derived detection-quality numbers until a documented customer eval set stands behind them — see our eval methodology.