Tasks and scoring

Task format

<task>/
  task.toml       # identity, resources and budgets
  instruction.md  # what the agent sees
  environment/    # solver image and inputs
  tests/          # verifier and reference resources
  solution/       # reference solution, where available

Task inputs and hidden references are separate datasets. Reference solutions, target answers and hidden fixtures must stay out of the solver environment.

Capability scores

Scientific Computing reports reward_repair = max(0, reward − floor) / (1 − floor). Follow the task’s instruction and verifier for exact checks and tolerances.

Aggregation

Official scores follow the website leaderboard. All six assessment scores included in the composite use a 0–100 scale:

The composite is the equal-weight average of six assessments, with weight 1/6 for each: Hypothesis Discovery, Protocol Planning, Scientific Computing, Scientific Software Operation, Experiment Observation and End-to-end research. End-to-end research scores 0 for all currently listed models. Cost and time equally average the per-task means for Protocol Planning, Scientific Computing, Scientific Software Operation and Experiment Observation. Wet-lab Execution is not included in the composite.

A complete standard evaluation covers Hypothesis Discovery (5), Protocol Planning (40), Scientific Computing (23), Scientific Software Operation (14), Experiment Observation (30) and End-to-end research (1): 113 tasks, with valid scores throughout. Infrastructure failures remain unscored; do not substitute partial coverage for the full composite.

The runner retains task rewards, repair gain, mean scores and full-credit rates for diagnosis. In summary.json.by_capability, use mean_score × 100 for Protocol Planning and full_credit_rate × 100 for Scientific Computing, Scientific Software Operation and Experiment Observation. The existing capability_balanced_mean averages raw task-score means and is a diagnostic, not the official composite. Hypothesis Discovery is scored outside this runner, as described below.

The currently published website figures were derived from rounded source chart values; their display precision may differ from a fresh aggregation of unrounded trial results.

Hypothesis Discovery

Hypothesis Discovery evaluates complete research runs. Agents run in the DiscoveryRSI research harness. Each run starts from a research database and a research goal from domain experts; the agent poses research questions, then develops and revises hypotheses, and every hypothesis version is scored.

The leaderboard reports the Inspire rate: the percentage of a model’s hypothesis versions that anticipate research published after the model’s knowledge cutoff, identified by a literature audit against journal-indexed publications. Each result is reported with its count of positives over hypothesis versions.

The evaluation uses six research databases. They are obtained from their official providers under each provider’s access terms; SLE does not redistribute them in its datasets.

To run it, select selected_tasks/hypothesis-discovery.txt in the experiment file and use the usual prepare, check and run commands; prepare installs the harness and downloads the data it can. See Execution environments and the Hypothesis Discovery guide.

End-to-end research (To be expanded)

The current end-to-end rubric awards 0.1 for valid simulation evidence, 0.2 for replayable matched runs and 0.3 for controlled comparisons. It does not yet establish the scientific validity of conclusions. The official composite includes the End-to-end research score, currently 0 for all listed models; the progress rubric remains a separate diagnostic output.

Source of truth

Use the website leaderboard for official reporting metrics, the repository’s task and scoring specification for per-task checks, and the matching release notes for evaluation scope.