Autonomous Science Bench0.1

Autonomous Science Bench evaluates AI agents for autonomous scientific discovery on real workflows in scientific laboratories.

Written by Yibo Wen on

Seedmeasured designs given123456ABCDEDesignthe agent picks a batch123456ABCDEBuild & Testthe lab measures it123456ABCDELearnthe agent refits its modelafterbeforedesignsvaluestartorderresultsnext batch, repeated for R roundsledger and modelGradeafter R rounds, both must passDiscovery gategood designs it measuredLearning gatemodel on unseen designs
  • Untested well
  • Measured, low to high value
  • Picked by the agent
  • Model before and after the new results

Autonomous Science Bench is a benchmark for AI systems that run autonomous laboratories. Its tasks come from the working experiments of laboratories across the United States. It scores the whole system, from model and agent to tools and harness, on how well it plans experiments, reads what comes back, and improves its decisions over time. This first release sets a common evaluation framework and six pilot tasks, two in each domain.

Autonomous Science Bench 0.1 leaderboard

Autonomous ScienceBench 0.1
  1. Opus 5.5Claude Code
  2. GPT-5.6 TerraCodex
  3. GPT-5.6 LunaCodex
  4. GLM-5.3 Flashmini-SWE-agent
  5. Sonnet 5.5Claude Code
  6. MiniMax M3mini-SWE-agent
  7. Qwen3.8 27Bmini-SWE-agent
  8. Haiku 4.5Claude Code
Resolution rates across 6 pilot tasks on Autonomous Science Bench 0.1.

Overview

An autonomous laboratory runs planning, automation, and measurement as one continuous loop. Being useful in that loop takes more than accurate prediction. The agent has to:

  • Choose the evidence: decide which experiments are worth running.
  • Explore or exploit: know when to search and when to commit.
  • Adapt: respond when outcomes defy expectations.
  • Know the limits: recognize when the data cannot support a conclusion.
  • Respect the lab: stay within its operations, costs, and measurement limits.

Every task specifies

The rules of the campaign. How the agent plays within them is up to it.

  • Available actions and feedback
  • Rounds, batch sizes, and budgets
  • Success criteria

Final evaluation

After the last round, two gates grade the campaign, each against a calibrated threshold. It is resolved only if both pass.

  • Discovery: good designs it chose to measure
  • Learning: ranking of designs never measured

Scientific Tasks

Tasks come from the labs’ own research and are built with their scientists. They test optimization, model identification, experimental design, and trade-offs between objectives. Each one ties the agent’s decisions to an outcome measured independently of it.

  • Biology

    Protein variants and cell-free expression.

    PilotNuclease active learning, Biosensor active learning

  • Chemistry

    Reaction conditions, catalysts, yield, and selectivity.

    PilotPropylene active learning, Suzuki condition screen

  • Materials

    Compositions and processing that reach target properties.

    PilotOER composition screen, Coercivity peak search

Results

Eight configurations ran all six tasks, three trials each. Opus 5.5 with Claude Code leads at 67%, ahead of GPT-5.6 Terra with Codex at 44%, for $173 and $52 across the six. Both pass every trial of the OER composition screen, and Opus 5.5 alone passes every nuclease trial. GPT-5.6 Luna follows at 28%, for $7. The rest pass at most two trials, and Haiku 4.5 none. With three trials per task, small gaps between configurations are not yet meaningful.

Autonomous Science Bench 0.1 cost vs. resolution rate

0%20%40%60%80%$5$10$20$50$100$200Total cost (USD, log scale)Resolution rateGPT-5.6 Terra (max), Codex: 44% (8/18) at $51.57Haiku 4.5, Claude Code: 0% (0/18) at $7.62Sonnet 5.5 (high), Claude Code: 11% (2/18) at $7.84GPT-5.6 Luna (max), Codex: 28% (5/18) at $6.91MiniMax M3, mini-SWE-agent: 6% (1/18) at $28.37Qwen3.8 27B, mini-SWE-agent: 6% (1/18) at $21.89GLM-5.3 Flash, mini-SWE-agent: 11% (2/18) at $25.68Opus 5.5 (max), Claude Code: 67% (12/18) at $172.91GPT-5.6 TerraCodexHaiku 4.5Claude CodeSonnet 5.5Claude CodeGPT-5.6 LunaCodexMiniMax M3mini-SWE-agentQwen3.8 27Bmini-SWE-agentGLM-5.3 Flashmini-SWE-agentOpus 5.5Claude Code
Each square is one configuration, pooled over every pilot trial; cost is its total across them, on a log scale. The line traces the cost frontier, the best resolution rate reached at each budget.

How results are scored

Each task scores two things: progress during the campaign, and the quality of what the agent delivers at the end. Both are measured against simple baselines under the same starting conditions and budget. A result always names the full configuration: model, agent, and harness.

During the campaign

  • Improvement over the seed
  • Experiments or cost to reach a target

Final outcome

  • Ranking quality or predictive accuracy
  • Constraints met across objectives

Compared against

  • Random selection
  • Bayesian optimization
  • Lab reference workflows

Roadmap

Each task can run on three backends. The agent’s interface never changes; only what answers its experiments gets closer to the real lab.

  1. v0
    Replay oracleResults come from recorded experiments, replayed exactly each time.
  2. v1
    Digital twinResults come from a simulator or model, with noise, limits, and failures.
  3. v2
    Live labResults come from real instruments, with real queues, costs, and faults.
Same tasks · same agent APIand more physical reality at each step

Built to stay trustworthy

  • Against contamination: private final evaluation, controlled data access, and fresh experiments.
  • For fair comparisons: versioned tasks, documented protocols, and expert review.

The goal: a shared foundation that shows where autonomous labs work, where they fail, and which advances lead to measurable scientific progress.

Citation

If you find this work useful, please cite it.

BibTeX
@article{wen2026autonomous,
  title={Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery},
  author={Yibo Wen and Han Liu},
  journal={arXiv preprint arXiv:2610.18449},
  year={2026},
  url={https://arxiv.org/abs/2610.18449}
}

Acknowledgements

Autonomous Science Bench is developed with researchers and engineers across the scientific community. We thank the participating labs for their scientific expertise and experimental infrastructure, and acknowledge the U.S. National Science Foundation’s support for this community.