FrontierCUA

FrontierCUA methodology

Environment

The benchmark contains 40 workflows across 11 enterprise domains, from support, CRM, and site reliability to finance, HR, and legal. Each runs on functional clones of the commercial apps it uses, such as Zendesk, HubSpot, Jira, Workday, and Google Workspace.

  1. Find related records across apps
  2. Compare evidence
  3. Save the requested result

Agents work only from the screen: each step reads a screenshot and takes a mouse or keyboard action. They cannot see the clones’ source code, query databases, call app APIs, or use dev tools, export URLs, or code.

Attempts

Each model runs at high reasoning, with one graded attempt per workflow. Each workflow counts once.

Each attempt starts from the workflow’s seeded state and runs until the agent stops or a wall-clock cap of two to three hours runs out. The agent receives the task brief and tool instructions, never the grading answer.

Anthropic models use the native computer-use tool, OpenAI models use the Codex agent, and open models use the OSWorld pyautogui agent, through OpenRouter or the model’s own harness.

Scoring

Each workflow has a rubric of atomic yes-or-no checks on the final state, weighted by importance, with weights that sum to 1. When the run ends, the environment records each app’s final state. An AI judge reads that state and scores each check independently, using the agent’s trajectory only to resolve intent.

Partial credit
The weighted share of checks met.
Pass
A graded attempt scoring at least 0.95.

pass@1 is the share of workflows whose graded attempt passes.

Example: two workflows
Workflow AWorkflow BOutcome
0.70, partial credit0.96, passpass@1 0.50, avg@1 0.83
avg@1
The mean workflow score across all workflows. avg@1 breaks ties in the ranking.

Task quality

Every workflow ships with a reference solution that scores 1.0 in the same apps, and a run that does nothing scores 0. Before a workflow ships, each of its parts is checked.

Instruction
A reviewer must be able to finish the task from the brief and the apps alone.
Rubric
Every check is atomic and decidable from the saved state.
Seeded world
Records, documents, and metadata are swept for leaks, and shortcuts such as a sortable column, an export, or a dashboard total are closed.
Rollouts
A hinted run must solve the task in full. Rollouts are scanned for reward hacking: empty outputs that pass a check, unintended side effects, and GUI bypasses.

Cost and time

Cost. Cost per rollout is the mean model inference cost across counted attempts.

Time. The median run takes about 374 steps and 36 minutes.

Reporting details

Timeouts
An attempt that reaches its time limit is graded on the work it saved, and scores 0 if nothing is gradeable.
Missing grades
An attempt with no numeric grade is omitted. It is not entered as zero. The timeout rule above is the exception.
Coverage
Every model is scored on all 40 workflows.
Cost coverage
Cost is the mean model inference spend over counted attempts for the full set of workflows. It excludes EC2 and grading. Missing cost is unknown, not zero.
Time coverage
Run time is a median across all runs. Per-model time is not part of this snapshot.
Cost provenance
Figures mix provider-reported and adapter-reported model inference. None are invoices.

This page includes a canary string to help detect its appearance in model training data.