Every Frontier Model Fails the Majority of SRE Incidents. ITBench-AA Finally Has Numbers.

The first controlled, independent agentic enterprise IT benchmark is live. Artificial Analysis and IBM tested 59 Kubernetes incident scenarios against every major frontier model — all of them failed more than half. The cost-vs-accuracy table that comes out of it is the one model-selection reference SRE architects have been waiting for.

The Benchmark No Vendor Designed

Every major observability vendor ships an AI SRE agent now. Datadog, AWS, PagerDuty, incident.io — all of them report 40–70% MTTR reduction in their own testing.

ITBench-AA is what happens when neither IBM nor Artificial Analysis is the vendor selling you the agent.

The framework runs 59 SRE tasks. Forty are public — drawn from real-world Kubernetes incident templates — and 19 are held-out, not previewed before evaluation. Each task hands the agent a Kubernetes snapshot at the moment of failure: active alerts, event streams, distributed traces, metrics, logs, and application topology. The agent’s job is to identify root cause. No scaffolding beyond that, no curated “this one demos well” scenario selection.

The evaluation infrastructure is open-source. IBM built it on CrewAI, deployed to Kubernetes, and published the framework at github.com/itbench-hub/ITBench. Any team can submit to the Artificial Analysis leaderboard. That matters: this isn’t a press-release number from a vendor — it’s a submittable, reproducible benchmark with public methodology.

The benchmark currently covers SRE tasks only. IBM has announced FinOps and CISO as follow-on domains, but neither is live. Don’t generalize the SRE results across all enterprise IT automation yet.

What makes the SRE scope architecturally interesting is the incident format. A Kubernetes snapshot at point-of-failure is exactly what lands in a real on-call queue — the same log-plus-topology blob your pipeline already ingests. That’s the scenario your model actually has to reason over at 2 AM.

All Below 50% — and Then the Cost Table Hits

The headline number: zero frontier models cracked 50% on ITBench-AA’s 59 SRE tasks. Claude Opus 4.7 leads at 47%. GPT-5.5 comes in at 46%. Then the drop: Gemini 3.1 Pro Preview at 30%, nearly 17 points behind second place.

The absolute ceiling matters architecturally. A 47% leader means you’re building around a pipeline that fails the majority of incidents by default. Whatever instrumentation you have today for human escalation, you need it for AI-ops too.

But the score column is only half the story. The cost column is where the benchmark gets disruptive.

Gemma 4 31B, an open-weights model running with reasoning enabled, scores 37% at $0.14 per task. Gemini 3.1 Pro Preview — a closed frontier model from Google — scores 30% at $2.23 per task. Gemma 4 31B wins on both dimensions simultaneously: higher accuracy, 16x lower cost. That’s not a tradeoff. That’s a Pareto improvement.

The open-weights picture extends further. GLM-5.1, running with reasoning, hits 40% at $1.23 per task — also ahead of Gemini 3.1 Pro Preview on score at roughly half the cost. The data is telling architects that the open-weights tier has crossed some kind of competency threshold for this task class.

Claude Opus 4.7 remains the accuracy leader at $5.38 per task. Whether 47% versus 37% is worth a 38x cost premium over Gemma 4 31B is a pipeline design question, not a capabilities question. ITBench-AA has turned that into an answerable calculation.

The Turns Trap

The benchmark tracks not just accuracy but how many reasoning turns each model uses. Gemini 3.1 Pro Preview averages 83 turns per task. GPT-5.5 averages 31 turns per task. Gemini 3.1 Pro scores 30%; GPT-5.5 scores 46%.

More investigation made things worse, not better.

The mechanism isn’t hard to reconstruct. In a Kubernetes incident snapshot, the signal is dense: alerts fire across multiple layers simultaneously, events propagate through dependent services, traces catch co-occurring anomalies that aren’t the root cause. An agent that keeps investigating past the point of commitment starts surfacing upstream fault-injection artifacts and correlated symptoms instead of locking an answer. It’s not finding the cause — it’s accumulating noise.

From a harness design perspective, this is the counterintuitive result that changes the loop architecture. Longer agentic loops don’t converge; they diverge. If your SRE pipeline is set to “keep trying if uncertain,” you may be burning token budget to generate false positives at scale.

The practical implication: reasoning loop length is a tunable variable in your harness, not a fixed property of the model. If the benchmark says a 46%-accuracy model resolves incidents in ~31 turns, treating 83-turn runs as a sign of diligence is a misconfiguration. A budget-cap on your agentic loop might be the cheapest architecture decision you make this week.

This also explains why the cost table reads the way it does. Gemini 3.1 Pro Preview costs $2.23 per task partly because it’s running 2.7x as many turns as GPT-5.5 — and getting a worse outcome for it. You’re paying for the wrong behavior.

Building on a Majority-Failure Floor

The 47% ceiling doesn’t mean AI-ops pipelines don’t work. It means they work the way scaffolded systems work: failure handling is a first-class primitive, not an afterthought.

Vendor benchmarks and independent benchmarks measure different things. Komodor’s Klaudia reports 95% accuracy on real-world Kubernetes incident resolution. ITBench-AA’s frontier model leaders land at 47%. The gap isn’t evidence that one number is wrong — it’s evidence of methodological difference. Vendor accuracy figures are typically measured on curated, pre-labeled scenarios the vendor’s agent was tuned against. ITBench-AA uses blind held-out tasks, no vendor involvement, no pre-tuning window. Both measurements are technically accurate. Only one predicts how your pipeline will behave the first time it sees an incident type it wasn’t designed around.

The architecture implication: treat sub-50% accuracy as the correct production prior until your harness has domain-specific telemetry proving otherwise. Gartner’s 2025 report put over 40% of agentic AI projects on track for cancellation by 2027, primarily from cost overruns and unclear business value. That’s partly a story about teams that built in optimistic mode and ran out of runway when the incident queue kept needing humans anyway.

What does a majority-failure-floor architecture look like? Confidence thresholds on agent outputs, not just outputs themselves. Parallel human escalation paths that activate on low-confidence runs rather than waiting for agent timeout. Rollback logic scoped per-task, not per-pipeline. None of these are novel patterns — they’re what you’d build for any system with a 47% success rate. The difference is that “AI SRE agent” somehow convinced teams they were exempt from designing for failure.

The one genuinely useful thing about the 47% ceiling: it’s measurable. Six months ago, the honest answer to “how reliable is your AI-ops layer?” was vendor-dependent and empirically thin. Now there’s a leaderboard, a cost table, and a reproducible methodology. You can’t fix a failure rate you can’t see.

Resources

• ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — IBM Research and Artificial Analysis explain the benchmark design, evaluation methodology, and full leaderboard results. — huggingface.co

• ITBench-AA Benchmark Leaderboard — Live leaderboard with per-model accuracy scores and cost-per-task data, updated as new submissions arrive. — artificialanalysis.ai

• ITBench: Open-Source Benchmarking Framework — IBM Research’s CrewAI-based framework for submitting custom agents to the benchmark; includes task definitions and evaluation harness. — github.com

• ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks — Academic paper presenting the full task suite design, agent architectures, and detailed methodology. — openreview.net