Run an experiment

1. Install

Requirements

• Linux x86-64 and Python 3.12.
• Docker Engine and Compose, with permission to run containers.
• Code and task-input downloads open on 6 November 2026. Reference solutions require manual approval on Hugging Face. See release and data access.

Set up the checkout

git clone https://github.com/Gen-Verse/Science-Last-Exam.git
cd Science-Last-Exam
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp secret/.env.example secret/.env
cp example_exp.yaml my_experiment.yaml

2. Configure

Credentials

Put your model API key and HF_TOKEN in secret/.env.

Agent

Copy a profile from env/native/profiles/. Set the model and compatible API endpoint; retain the exact CLI version. Codex is installed in the solver image. Claude Code requires the matching CLI on the host.

Experiment

name: sle-smoke
agent: env/native/profiles/codex.example.json
environment: configs/environments/local.yaml
tasks: selected_tasks/smoke.txt
concurrency: 1
output:
  root: runs

Paths are repository-relative or absolute. See environment setup before selecting GUI, end-to-end or Hypothesis Discovery tasks.

3. Choose tasks

Change tasks: in the experiment, then run the same commands. For a custom selection, create a text file containing IDs from tasks/index.jsonl, one per line. full.txt covers Protocol Planning, Scientific Computing, Scientific Software Operation and Experiment Observation; leaderboard.txt and all.txt both contain all 113 tasks, including Hypothesis Discovery and End-to-end research (To be expanded). For the entire benchmark, select all.txt and set end_to_end_profile to a profile under env/closeloop/profiles/. Hypothesis Discovery tasks need their research databases; see Hypothesis Discovery setup.

4. Run

Preview

python -m sle_run run my_experiment.yaml --dry-run

Inspect the selected tasks, models and runtimes. This makes no downloads or model calls.

Prepare and check

python -m sle_run prepare my_experiment.yaml
python -m sle_run check my_experiment.yaml

Fetch selected data and prepare required runtimes, then check dependencies, credentials and services. The check does not test model endpoint connectivity.

Launch

python -m sle_run run my_experiment.yaml

Start with the smoke list. The launcher repeats checks and records the experiment; it does not automatically retry or resume.

5. Inspect results

Open the new directory under runs/:

• plan.json: selected tasks, models and budgets.
• summary.json: stage status and output locations.
• capabilities/: task scores, native traces and verifier output.
• end-to-end/: the separate research result, if selected.

Understand scores → · Package and submit results →