On July 9, 2026, DevRev released Enterprise-Bench: an open-source benchmark designed to evaluate AI agents under the conditions that define real enterprise work — fragmented data, siloed systems, and permission boundaries. The headline result is a 94.3% vs 63.6% accuracy gap between DevRev’s own Computer agent and Claude Code, using the same Opus 4.8 model family, with Computer achieving 4.4× better token efficiency.
The technical claim is straightforward: the performance difference is not the model — it’s the data retrieval architecture underneath it.
This is a research-based summary; we reviewed the benchmark announcement, third-party coverage, and DevRev’s published methodology, but did not run the benchmark ourselves.
What Enterprise-Bench Tests
Most current AI agent benchmarks test consumer-grade tasks: write this code, answer this question, summarize this document. The data is clean. Outcomes are binary. The task is isolated.
That’s not what enterprise AI actually encounters. Enterprise systems have:
- Customer records split across CRM, ticketing, email, Slack, and legacy databases
- Permission layers where one role can read deals but not contracts
- Data that changes while the agent is reasoning about it
- Questions whose correct answer requires cross-referencing five source systems, none of which share a schema
Enterprise-Bench measures organizational complexity — how well an agent performs as data volume, fragmentation, and permission complexity scale up.
Ahmed Bashir, CTO at DevRev, described the distinction: “What Enterprise-Bench adds is a way to measure organizational complexity, not just task complexity.” UC Berkeley professor Alexandros Dimakis, who validated the benchmark design, noted: “Most agent benchmarks test consumer tasks where data is clean and outcomes binary. What’s different here is difficulty scales with the data itself.”
Evaluation Framework
Enterprise-Bench uses an autonomy ladder modeled on autonomous driving levels:
| Level | Task Type | Status |
|---|---|---|
| L1 | Factual retrieval from enterprise data | Released July 9 |
| L2 | Complex multi-source queries requiring synthesis | Released July 9 |
| L3 | Strategic orchestration (multi-step workflows) | Planned later 2026 |
| L4 | Autonomous operation (full task ownership) | Planned beyond 2026 |
The current benchmark scores agents on three axes:
- Precision: Correct answer, verifiable source, auditable action path
- Efficiency: Does token cost scale with question complexity, or with data volume (bad)?
- Safety: Permission boundaries respected; every action logged and traceable
Evaluation uses an independent LLM judge against published criteria, with mandatory execution trace submission alongside scores. All results are reproducible: the full dataset, queries, judging criteria, and Harbor evaluation harness are available on Harbor Hub.
The Results
On identical L1-L2 tasks, both agents using Opus 4.8:
| Metric | DevRev Computer | Claude Code |
|---|---|---|
| Accuracy (XL dataset) | 94.3% | 63.6% |
| Tokens per correct response | ~5,598 | ~24,461 |
| Total tokens (largest runs) | ~54M | ~92M |
Computer achieved 48% higher accuracy and used 4.4× fewer tokens per correct response. As dataset size grew, Computer’s total token usage stayed roughly flat. Claude Code’s climbed 29% — ~54M vs ~92M tokens on the largest runs.
Jeff Smith, Office of the CTO at DevRev, on the benchmark design: “Benchmarks that are not public are not benchmarks. They are marketing. We are publishing everything.”
Why the Same Model Produces Different Results
This is the technically interesting part. DevRev used Opus 4.8 as the underlying model for both Computer and Claude Code. The 4.4× token efficiency gap cannot be attributed to model capability. It comes from data retrieval architecture.
Claude Code is a general-purpose agent that reasons over code and files by loading relevant content into its context window. Applied to enterprise data retrieval tasks — pulling the correct record from across CRM, tickets, and emails for a specific query — it must either:
- Search broadly and pull large chunks into context (high token cost, potentially low precision), or
- Search narrowly and risk missing relevant records (lower token cost, lower recall)
Enterprise-Bench’s XL dataset appears to amplify this tradeoff. As data volume increases, a context-stuffing approach costs more tokens for the same or worse answers.
DevRev’s Computer uses what it calls “Native Shared Memory” (GBrain) — a persistent, permission-aware knowledge graph that unifies enterprise data sources with real-time sync (AirSync). Instead of pulling raw records into the model’s context window, Computer queries a structured graph layer and retrieves only the precise data the model needs. The model sees fewer tokens, answers more precisely, and the token cost doesn’t grow with dataset size the way context stuffing does.
The architectural contrast:
Context-stuffing approach (Claude Code on enterprise tasks):
Query → Search enterprise data → Pull matching chunks into context → Model reasons over 24K tokens → Answer
Graph-retrieval approach (Computer):
Query → Query structured knowledge graph → Retrieve targeted records → Model reasons over 5.6K tokens → Answer
This is not a critique of Claude Code as a coding agent — it was never designed for enterprise data retrieval. The comparison is intentionally cross-category, and that’s where the benchmark’s conflict-of-interest concern lives.
The Vendor-Benchmark Problem
DevRev made Enterprise-Bench and DevRev tops Enterprise-Bench. That’s the obvious concern, and it’s legitimate.
Three factors partially address it:
-
Open-source methodology: The dataset, harness, evaluation criteria, and traces are all published. Any vendor or researcher can reproduce the results and submit to the leaderboard.
-
Academic validation: The benchmark design was validated by Professor Alexandros Dimakis of UC Berkeley, affiliated with the Harbor evaluation project run by researchers from UC Berkeley and Stanford.
-
The comparison is arguably category-inappropriate: Claude Code is an excellent coding agent. Enterprise-Bench tests enterprise data retrieval under organizational complexity. Comparing them is like running a database query benchmark against a text editor — the “loser” wasn’t designed for those conditions. That DevRev Computer wins on a benchmark that maps to DevRev Computer’s exact design purpose is not surprising.
What would make the benchmark credible: comparable vendors — Salesforce Agentforce, ServiceNow AI Agents, Microsoft Copilot Studio, AWS AgentCore — submitting their own results to Harbor Hub. The published methodology makes this possible. Whether they do is what will determine if Enterprise-Bench becomes a real standard.
Until that happens, treat the accuracy numbers as a lower bound on what specialized enterprise agents can do, and as a rough upper bound on what general-purpose agents do when applied to tasks they weren’t optimized for.
What This Means for Builders
If you’re building enterprise AI workflows, the token efficiency finding is the most actionable result, independent of whether DevRev’s accuracy numbers are fully trustworthy.
The core insight: context stuffing doesn’t scale for enterprise data tasks. As your data volume grows — more customers, more tickets, more emails, more CRM records — the cost of naive retrieval grows with it. Architectural decisions made now compound as data grows.
Builders evaluating their enterprise AI stack:
When general-purpose agents (Claude Code, Codex, Cursor) are the right choice:
- Tasks centered on code: writing, reviewing, debugging, test generation
- Repository-level work where the “data” is files you control
- Workflows where context window management is straightforward
When purpose-built enterprise agents earn consideration:
- Data retrieval across multiple live enterprise systems (CRM + ticketing + email + Slack)
- Queries where permission awareness is a hard requirement (not a policy layer bolted on)
- High-volume workflows where token costs at scale are a real budget constraint
For builders using Claude Code on enterprise data tasks: the 24K tokens-per-response finding points to retrieval strategy as the lever. Chunking enterprise data more aggressively, adding a graph retrieval layer (e.g. a knowledge graph prebuilt from your enterprise sources), or using MCP servers with structured query interfaces can reduce context window bloat without switching to a different agent platform.
How to Run Enterprise-Bench Yourself
The benchmark is available on Harbor Hub. To run it against your own agent:
- Access Harbor Hub and download the Enterprise-Bench L1-L2 dataset and harness
- Configure your agent to connect to the provided enterprise data environment
- Run the Harbor evaluation harness against your agent
- Submit results (scores + execution traces) to the public leaderboard
The Laude Institute researchers affiliated with the Harbor project built the harness to be agent-agnostic. If you run it, the leaderboard gets more useful — and the benchmark’s vendor-bias concern gets proportionally weaker.
What to Watch
- L3/L4 releases: Strategic orchestration and autonomous operation benchmarks are planned for later 2026. Those levels are where the gap between purpose-built and general-purpose agents may narrow or widen significantly.
- Competing vendor submissions: Whether Salesforce, ServiceNow, Microsoft, or Google submit results to Harbor Hub is the real test of Enterprise-Bench’s independence.
- Reproducibility: Third-party researchers replicating DevRev’s 94.3% accuracy claim on Computer — or finding it lower in independent runs — will be the benchmark’s credibility moment.
The token efficiency finding (same model, 4.4× gap) is the result most likely to hold up under independent scrutiny, because it’s architecturally principled. The accuracy numbers will need more external validation before they can be fully trusted.