Trajectories and results
Keep scores, settings and evidence together.
Read a run
runs/<name>-<time>-<id>/
plan.json
summary.json
capabilities/<agent-run>/
summary.json
results.jsonl
jobs/
hypothesis-discovery/<run>/
end-to-end/plan.json records task IDs, models and budgets. The top-level summary records stage status and output locations. Capability results contain one normalized record per task; jobs/ holds native traces and verifier output. Hypothesis Discovery runs record every hypothesis version; End-to-end research results remain separate.
Handle infrastructure errors
Inspect the original error before rerunning. The existing capability command can select tasks with infrastructure errors:
python evaluate.py --profile runs/<experiment>/agent.json \
--rerun-errors runs/<experiment>/capabilities/<agent-run>Preserve the original run. Report task scope, model settings and any protocol changes with the new result.
Package results
Use the downloadable result template. Place these items at the archive root:
sle-results.zip
metadata.json
eval.json
runs/- Metadata: exact model, harness, versions, task selection, budgets and reproduction command.
- Evaluation: per-task scores, status and aggregate metrics.
- Evidence: native traces, verifier outputs and required artifacts.
Send for review
Email the archive or a reviewer-accessible download link to ly1988@princeton.edu and wuyc@stanford.edu. The result-submission page opens a draft; it does not upload or send automatically.
Report the official leaderboard metrics on a 0–100 scale: the Hypothesis Discovery Inspire rate, the Protocol Planning mean reward, the pass rates of Scientific Computing, Scientific Software Operation and Experiment Observation, and the End-to-end research score. The official composite averages these six assessments with weight 1/6 each. Wet-lab Execution is in construction and is not scored in v1. In the currently published results, End-to-end research (To be expanded) is 0 for every listed model. For Hypothesis Discovery, include the hypothesis versions and the audit outputs behind the Inspire rate. The runner’s raw capability-balanced mean and the end-to-end progress rubric are diagnostics, not official scores. Subset results and alternative settings can be submitted, but remain separate from a complete standard evaluation.