Is SWE-bench the Right Benchmark for CPU Performance in Agentic AI?

Agentic AI has done something no previous AI workload managed: it made the host CPU interesting again.

In single-shot inference the CPU tokenizes a prompt, hands off to the accelerator, and waits. In an agent loop it runs everything between model calls — orchestration and control flow, tool dispatch, sandbox spawn and teardown, file I/O, test execution, retrieval, JSON parsing, retries. Recent characterization work has put numbers on this. In tool-dominated agentic pipelines, CPU-side tool processing can account for the large majority of end-to-end latency, and the imbalance gets worse as accelerators get faster. Systems-level measurement suites like AgentSysBench report that non-LLM components dominate latency in half the applications they profile, that per-task latencies across components can diverge by more than an order of magnitude, and that the bottleneck moves depending on the request, the model, and the deployment.

So the question is a live one. If you're evaluating a server CPU, a platform, or a CPU-to-accelerator ratio for agentic workloads, what do you measure?

The answer that comes back most often is SWE-bench. It's the benchmark everyone quotes for coding agents. It runs real repositories in real containers, executes real test suites, and looks like a workload rather than a synthetic loop. It has face validity in a way STREAM never will.

It is also, as it ships, the wrong instrument for this job. Worth being precise about why, because a variant of it, used differently, is close to right.

What SWE-bench actually measures

SWE-bench (Princeton, 2023) is a set of real GitHub issues paired with the commits that fixed them, drawn from a dozen popular Python repositories. The agent gets the repo at the pre-fix commit plus the issue text, and has to produce a patch that makes the hidden tests pass. SWE-bench Verified is the 500-instance human-reviewed subset built with OpenAI in 2024; Lite is 300; Pro, Multilingual, Live, and Multimodal variants followed.

The metric is resolve rate. Percentage of instances where the patch makes the tests pass.

That's it. There is no time term in the metric. No throughput, no latency, no concurrency, no power. A benchmark whose score is invariant to how fast the machine ran it cannot, by construction, be a performance benchmark. Two platforms with a 2x difference in CPU capability will produce the same SWE-bench number — that's a feature of the benchmark, not a bug, but it means the thing you care about is not in the output.

The honest case for using it anyway

Before the objections, the steelman, because it's stronger than the dismissive version suggests.

A SWE-bench harness run is a genuinely mixed, genuinely realistic system workload. It exercises container lifecycle, dependency resolution, compilation, filesystem metadata pressure, test-suite execution, and concurrent sandbox memory footprint — all at the same time, under contention. It surfaces bottlenecks that no microbenchmark predicts. Work on agentic serving systems has found exactly this class of problem in coding-agent harnesses: environment preparation (image pull, package install, repo build) dominating the pre-trajectory phase, and container images never being reclaimed so disk consumption grows linearly until the node falls over. STREAM does not find that. A tuned pip benchmark does not find that.

Concurrency scaling is the other real contribution. How many agent sandboxes can one socket hold before p95 step latency blows up? Sandbox working sets in agentic applications have been measured peaking in the tens of gigabytes per session. That density number — agents per socket at a fixed SLO — is arguably the headline figure a datacenter CPU vendor should be quoting for this market, and you can only get it from something SWE-bench-shaped.

Keep both of those. The problem isn't the workload. It's the benchmark wrapped around it.

Where it breaks as a CPU benchmark

The variance swamps the signal. Agent trajectories branch. An agent that goes down a rabbit hole runs forty tool calls where another ran eight, and the wall-clock difference between those two runs is far larger than the 15–25% you're trying to attribute to a CPU. LLM sampling is nondeterministic even at temperature zero on many stacks. Retries, timeouts, and API jitter add more. You would need many repeats per configuration to resolve a platform delta, and the confidence intervals would still overlap.

The critical path is usually somewhere else. If inference is remote — the normal case — your run is gated by time-to-first-token and decode speed on someone else's cluster, plus the internet. You are measuring a hyperscaler's GPUs and your egress path, and calling it a CPU result. Move inference on-box and it gets better, but now the number is entangled with the serving stack: batching policy, KV cache management, quantization, tokenizer implementation. Useful, but it's a serving benchmark, not a CPU one.

You can't decompose the delta. Platform A finishes in 41 minutes, platform B in 47. Was that cores, memory bandwidth, NVMe, NIC, kernel version, container runtime, Python build flags, or trajectory luck? A benchmark that can't be attributed can't guide a design decision. This is the fundamental difference between a capability benchmark and an engineering benchmark, and it's the reason microbenchmark suites still exist.

The CPU profile is narrow, and mostly setup. SWE-bench's original form is Python-only across twelve repos. The compute mix is CPython interpreter work, some compiled numeric kernels, dependency installation, and pytest. A large share of that is disk and network bound, not core-count or bandwidth bound. If your customer's agentic workload is document extraction, browser automation, SQL analytics, or C++ builds, this profile tells them very little.

The dataset itself is no longer sound. Verified is saturated — as of late 2026 the leading models cluster within a few points of each other in the mid-nineties, having started from under 2% in 2023. Worse, OpenAI's own audit of problems its models failed inconsistently found that a majority contained material defects in test design or problem statement: narrow tests enforcing implementation details, tests rejecting functionally correct patches. Independent work has reported roughly a third of "successful" patches involving solution text leaked into the issue thread, and a similar share passing on inadequate test coverage; correcting for both dropped one agent configuration's reported rate from about 12.5% to about 4%. Adding stronger generated tests reshuffled leaderboard rankings in a quarter to two-fifths of cases. OpenAI has stopped reporting it for frontier launches. You do not want a measurement fixture with those properties sitting in a procurement decision.

It's expensive per bit of information. Published leaderboard runs on a 50-instance subset have cost anywhere from a few hundred to nearly two thousand dollars in API spend per evaluation, before you count engineer hours and machine time. For a number that contains no CPU information, that's a bad trade.