Environments and agents

Docker tasks

Planning, observation and computing tasks use the local Docker environment. Solver images contain the task inputs and scientific tools; grading runs separately with withheld references. Computing images may require several GB per task.

The network allowlist permits the model API. Web search and web fetch are disabled. If you enable docker_compat, the solver network is open; disclose that setting with your results.

GUI tasks

GUI tasks use one QEMU desktop per episode, controlled through screenshots, mouse and keyboard actions. Prepare the desktop locally; no prebuilt SLE desktop is distributed.

cp env/gui/local.env.example env/gui/local.env
mkdir -p .local/gui/config .local/gui/data/materials
cp env/gui/vm/config/linux.example.json .local/gui/config/linux.json

1. Set the controller and bridge tokens in env/gui/local.env.
2. Add the installer locations and hashes to the desktop configuration. Follow the image build guide.
3. Check configs/environments/local.yaml. Use KVM with /dev/kvm; select the slower tcg accelerator explicitly otherwise.
4. For Codex, provide a native model catalog containing the chosen model or select a valid model template.
5. Run prepare, check and run.

The controller stays running after evaluation. Stop it when finished:

docker compose --env-file env/gui/local.env -f env/gui/compose.yaml stop controller

End-to-end research (To be expanded)

This assessment has a separate hosted-Codex runtime and model configuration. Set end_to_end_profile in the experiment to a profile under env/closeloop/profiles/.

Node/npm, Docker Buildx and Compose are required. prepare installs the pinned runtime and builds its images. Its official End-to-end research score is one of the six composite assessments; in the currently published results it is 0 for every listed model. The task’s progress rubric is a separate diagnostic and does not replace that score.

Hypothesis Discovery

Hypothesis Discovery uses the same experiment file and commands as the other assessments. Set tasks to selected_tasks/hypothesis-discovery.txt for all five research databases, or to selected_tasks/hypothesis-discovery-public-data.txt for a first run on public GEO studies that needs no data-access application. It needs no Docker.

python -m sle_run prepare my_experiment.yaml   # pinned DiscoveryRSI harness + data
python -m sle_run check my_experiment.yaml     # harness, data per database, model CLI and key
python -m sle_run run my_experiment.yaml

prepare installs the DiscoveryRSI research harness at a pinned commit and places the data under data/hypothesis-discovery/<database>/. The public GEO starting studies are downloaded automatically; MIMIC-IV and eICU are downloaded from PhysioNet with the PHYSIONET_USERNAME and PHYSIONET_PASSWORD of an account that has access; UK Biobank and the HCC cohort are placed there by you. The research agent follows the agent profile (Claude Code or Codex on the host), or an OpenAI-compatible endpoint set under hypothesis_discovery. Each run writes every hypothesis version with its parent links to runs/<experiment>/hypothesis-discovery/. The Hypothesis Discovery guide gives the data layout, adapters and outputs; database sources and the Inspire rate are described in Tasks and scoring.

Agent profiles

Use a copy of a profile under env/native/profiles/ for custom settings. Other agents require a Harbor integration; the unified launcher currently documents Codex and Claude Code. Hypothesis Discovery runs use the DiscoveryRSI harness instead. See the complete running guide.