Scientific workflow
At the laboratory bench, systems must observe an experiment and reason about its dynamics. At the computer, they must compute correctly and operate specialist tools. In research, they must develop hypotheses that lead back to new experiments.
Tasks are evaluated separately. SLE does not require an agent to carry one experiment through all five stages. The capability profile is intended to locate specific strengths and failures.
Read the five benchmark definitions →Execution and evaluation
The model receives the instruction, inputs, and working environment. Reference answers and checks are kept separate. Once execution ends, the delivered output is frozen for evaluation.
Prepare
Fix the task, input materials, environment, and allowed tools.
Execute
Run SLE-Agent and record responses, actions, artifacts, and resource use.
Evaluate
Freeze the output, then apply independent scientific checks.
Execution may be a direct response, an interaction with a container or virtual desktop, or a research process that develops a hypothesis tree. All use the same agent loop and run-record format, with the tool interface appropriate to the task.
Scoring by capability
Match choice answers to references and open responses to observable facts in first-person experimental video.
Assess next-action accuracy and the operations, order, and parameters of structured plans.
Execute the delivered code and check scientific outputs using task-specific scientific checks and specified tolerances. A program that runs can still be wrong.
Inspect application state, saved artifacts, or visual evidence against the task’s scientific requirements.
Assess every generated hypothesis version for actionable correspondence with research published after the model’s knowledge cutoff.
Evaluator calibration and expert-agreement results are pending.
Conditional rate = actionable versions ÷ filtered versions
Retrieval coverage and the conditional judgment rate remain visible alongside the overall rate. Revisions in a hypothesis tree are related observations, not independent experimental replications. Correspondence with later work does not imply that researchers adopted an agent’s hypothesis.
Inspect the discovery results →Task construction
Domain experts source meaningful scenarios and fix scientific observables, constraints, and acceptance criteria. Agents implement executable task packages. Reference execution and expert review determine which tasks are admitted.
Source a scientific scenario
Domain experts identify work with meaningful scientific outcomes.
Specify the task
Fix observables, constraints, inputs, and acceptance criteria.
Implement the environment
Build the executable task package and prepare its tools and inputs.
Check the reference execution
Verify that the intended output can be produced and evaluated.
Review for admission
Experts examine scientific validity and the task’s evaluation requirements.
Prepare the release
Assign related instances together and document the public and private collections.
Construction design follows the project manuscript. Counts and review coverage remain provisional.
Public and private collections
The planned public set contains 198 tasks: 154 Representative and 44 Last-Exam. The remaining 8,965 tasks and all six Hypothesis Discovery databases form the private collection. Instances sharing a source video, protocol, environment, or database remain on the same side.
Public subset scores average four capabilities. Hypothesis Discovery is reported separately through Inspire Rate. Rotation from the private collection is part of the proposed release design; no rotating service is live in this preview.