FOR RESEARCHERS
Evaluate. Document. Submit.
Prepare an evaluation and share the evidence behind your results. Start with the task catalog, record your setup, and check your submission before sending it to the team.
The catalog and submission tools are available now. Runnable code, task inputs, and environment downloads are still being prepared. Researchers who already have the materials can send results for preliminary review; new participants should request access first.
Request evaluation access ↗Release preparationOffline checks · 24 October 2026
A private copy of the 198 public tasks has been prepared independently of the research checkout. The checks below passed on that copy; a public runnable download is still being prepared.
- 198 / 198
- Task configurations loaded in Harbor
- 198 · 154 · 44
- Full, Representative and Last-Exam plans checked
- 96 / 96
- QA reference and empty-answer checks passed
5,872 task and runner files have checksums (about 3.03 GB, excluding desktop images). These checks do not build environments, run models, or establish a clean-machine reproduction.
Before a public runnable release
- Confirm material redistribution and whether reference answers and solutions are public.
- Package the GUI controller and Linux desktop materials, then verify installation.
- Build and test the environments on a fresh machine, including the 27 Coding environment groups.
- Freeze comparison settings and complete the separate Inspire Rate evaluator and database access.
No new model results are implied. Download this status as JSON ↗
01 / SCOPE
The draft public catalog
198 task identities across the first four capabilities: 154 Representative and 44 Last-Exam. This metadata catalog includes no inputs, reference answers, or executable environments. Hypothesis Discovery is a separate evaluation setting.
Loading catalog…
| Task ID | Capability | Discipline | Subset |
|---|
Snapshot: 24 October 2026. Task identities are provisional; the selection checksum does not pin task content or establish runtime readiness. View illustrated task examples →
02 / REPRODUCE
Run with an explicit configuration
Get the materials first
The current repository depends on task data and desktop images that are not included in a bare checkout. Ask the team for an agreed release, its task-content checksum, and the environment materials. The commands below apply once those materials are available.
Use a dedicated evaluation machine with Docker. The trusted runner uses the host Docker service; task solvers must remain isolated from grader data and credentials. GUI additionally requires the Linux desktop image and controller described in the supplied environment guide.
Record what actually ran
Keep the exact model identifier and version, harness commit, task-content revision, environment versions, command, network settings, budgets, and retry/reuse policy. Never guess missing model suffixes or settings.
Current launcher inspection: science-tool-agent/2.3.1. Historical website runs used other configurations. A newer runner does not reproduce an older score by itself.
Inspect the public plan — no model calls
export SCIENCEBENCH_WORKSPACE="$PWD"
docker compose -f env/compose.yaml build runner
docker compose -f env/compose.yaml run --rm \
--entrypoint python runner build.py --verify
docker compose -f env/compose.yaml run --rm runner \
--split public --planThe plan command was checked on the existing project environment and selected 198 tasks. A clean-machine reproduction has not yet been validated. Building the runner needs Docker and dependency downloads.
Run a small Scientific Coding evaluation
# Set SLE_BASE_URL and SLE_API_KEY in your environment.
# This command starts a real evaluation and model requests.
docker compose -f env/compose.yaml run --rm runner \
--split public --ability scientific_coding \
--limit 1 --model YOUR_MODEL --workers 1This is a pilot, not a complete benchmark result. Record it as custom scope. For a full public run, remove the ability and limit filters after preparing all required environments. Use --subset representative or --subset last_exam to select a complete subset. Agree budgets and network policy with the team before comparison.
Keep evaluation evidence
Preserve the run manifest, per-task results, summary, logs, delivered outputs and evaluator evidence. Hash archives with sha256sum archive.zip (Linux) or shasum -a 256 archive.zip (macOS). Preserve failed and unfinished trials; do not select only successful attempts.
Hypothesis Discovery uses Inspire Rate
Send generated, filtered and actionable version counts, together with the database snapshot, literature cutoff, judge configuration and version-level evidence. The executable Inspire Rate release and database access are still being finalized. Legacy five-level evaluations do not reproduce this metric.
03 / SUBMIT
Check your result packet
Loading validation tools…
Opening a draft does not send your submission or attach files. Attach both downloads and press Send in your email app. You can also send them directly to junyangwang@stu.pku.edu.cn. Review is coordinated by email; there is no automatic receipt or tracking dashboard.
04 / REVIEW
From submission to a published result
Check the evidence
The team checks scope, versions, hashes, coverage and execution settings, and requests anything missing.
Verify the evaluation
Review or reproduction establishes what the scores support. Partial runs and protocol differences remain visible.
Approve publication
Only approved results enter the public tables. Contact information and restricted evidence stay out of the public export.
The current intake is preliminary and manually reviewed. A valid packet does not grant a rank or launch a hosted evaluation. Public PR submission will become available when a dedicated public repository is announced.
Contribute a task or environment
Email a task description, scientific motivation, input provenance and redistribution permissions, required software, expected deliverables, and a proposed verifier with positive and negative examples. State any license or data-access restrictions. The team will review the proposal before adding it to a release.