Answering questions isn’t
getting the work done.
Measurable lift on APEX-Agents.
Hundreds of files, threaded emails, and a one-line, ambiguous prompt — that’s the real job. We build process-level data in the same distribution as the work, trainable and evaluable.
A new kind of agent eval is becoming the standard.
Agent evaluation is moving from isolated Q&A toward real, long-horizon, professional-workflow tasks — and Mercor’s APEX-Agents is the leading example: no model passes half of it. We don’t build data to game this board; we build it to train the general capability real work demands — third-party-verified to transfer to unseen tasks.
Gemini 3.5 Flash
Fable 5Opus 4.8
GPT 5.5
GLM-5.2Axis 0–100 · dashed line = 50%. Every model tops out below it.
Official dataset: 33 Worlds · 480 Tasks, spanning investment banking, consulting and law.
More real
Authored by 256 senior practitioners (avg. 12.9 yrs — BCG, McKinsey, Morgan Stanley, Citi) around real workflows, across banking / consulting / law.
Becoming standard
Frontier labs keep investing in evals like this; Artificial Analysis independently reproduced the results.
Trains general skill
Third-party (Applied Compute): models trained on this data gained +7.7pp on GDPVal and +8.0pp on Toolathalon — transfer to unseen tasks, not board-fitting.
One World, and a whole set of Tasks around it.
A World is the simulated work computer of an investment-banking manager — a file system of 150+ attachments, an inbox, and chat threads. Each Task is a one-line prompt with the judging rubric and a gold response behind it.
The manager’s work computer.
- File system · 150+ attachments
- PDF & spreadsheet-heavy
- Email inbox · Chat threads
One line — with the answer key behind it.
- Task Prompt (the one-liner)
- Rubrics · LLM-as-a-Judge
- Gold Response · expert reference
What the agent actually did.
- Score against the rubric
- Full Trajectory
- Every tool call & file read
Sampled statistics track the official distribution: 150–200 attachments per World, difficulty validated across 90 model runs.
Even the best models reach only half — and that’s the RL headroom.
Even the strongest frontier models pass only ~34% on the first try — and the failures cluster on a handful of real capability gaps, not random noise.
RL turns “right after several tries” into “right on the first” — the ~20pp Pass@1→Pass@3 gap is its room.
One task ≈ 149 files and ~88 tool calls. The slower the climb, the harder the task.
Difficulty spreads evenly; green (solved) grows from Pass@1 to Pass@3.
Models can’t reliably identify the authoritative data source, reconcile conventions across files, trace the full calculation chain, or converge on a single final answer among candidates.
Buildable — and built right.
Public boards forbid training on them, and few vendors can synthesize new items in the same distribution. We can — with system synthesis, a three-tier expert bench, and a loop that keeps compounding.
Tasks feed the World back. The data asset compounds.
A one-line prompt; a whole data room of work.
Real tasks, gold responses and rubrics — straight from the APEX-Agents dataset (Investment Banking).
Models keep evolving — so bring your own tasks and run a round.
Tell us what you’re training; we’ll match it with same-distribution, process-level data and evaluation.