FrontierCUA methodology
Environment
The benchmark contains 40 workflows across 11 enterprise domains, from support, CRM, and site reliability to finance, HR, and legal. Each runs on functional clones of the commercial apps it uses, such as Zendesk, HubSpot, Jira, Workday, and Google Workspace.
- Find related records across apps
- Compare evidence
- Save the requested result
Agents work only from the screen: each step reads a screenshot and takes a mouse or keyboard action. They cannot see the clones’ source code, query databases, call app APIs, or use dev tools, export URLs, or code.
Attempts
Each model runs at high reasoning, with one graded attempt per workflow. Each workflow counts once.
Each attempt starts from the workflow’s seeded state and runs until the agent stops or a wall-clock cap of two to three hours runs out. The agent receives the task brief and tool instructions, never the grading answer.
Anthropic models use the native computer-use tool, OpenAI models use the Codex agent, and open models use the OSWorld pyautogui agent, through OpenRouter or the model’s own harness.
Scoring
Each workflow has a rubric of atomic yes-or-no checks on the final state, weighted by importance, with weights that sum to 1. When the run ends, the environment records each app’s final state. An AI judge reads that state and scores each check independently, using the agent’s trajectory only to resolve intent.
- Partial credit
- The weighted share of checks met.
- Pass
- A graded attempt scoring at least 0.95.
pass@1 is the share of workflows whose graded attempt passes.
| Workflow A | Workflow B | Outcome |
|---|---|---|
| 0.70, partial credit | 0.96, pass | pass@1 0.50, avg@1 0.83 |
- avg@1
- The mean workflow score across all workflows. avg@1 breaks ties in the ranking.
Task quality
Every workflow ships with a reference solution that scores 1.0 in the same apps, and a run that does nothing scores 0. Before a workflow ships, each of its parts is checked.
- Instruction
- A reviewer must be able to finish the task from the brief and the apps alone.
- Rubric
- Every check is atomic and decidable from the saved state.
- Seeded world
- Records, documents, and metadata are swept for leaks, and shortcuts such as a sortable column, an export, or a dashboard total are closed.
- Rollouts
- A hinted run must solve the task in full. Rollouts are scanned for reward hacking: empty outputs that pass a check, unintended side effects, and GUI bypasses.
Cost and time
Cost. Cost per rollout is the mean model inference cost across counted attempts.
Time. The median run takes about 374 steps and 36 minutes.
Reporting details
- Timeouts
- An attempt that reaches its time limit is graded on the work it saved, and scores 0 if nothing is gradeable.
- Missing grades
- An attempt with no numeric grade is omitted. It is not entered as zero. The timeout rule above is the exception.
- Coverage
- Every model is scored on all 40 workflows.
- Cost coverage
- Cost is the mean model inference spend over counted attempts for the full set of workflows. It excludes EC2 and grading. Missing cost is unknown, not zero.
- Time coverage
- Run time is a median across all runs. Per-model time is not part of this snapshot.
- Cost provenance
- Figures mix provider-reported and adapter-reported model inference. None are invoices.
This page includes a canary string to help detect its appearance in model training data.