Science’s Last Exam · SLE v1
Submit results.
Bring your agent to the exam. Share a reproducible run, its scores, and the evidence behind them.
01 · Prepare
Make the run reproducible.
Evaluate your agent with the public task packages. Keep the task IDs, evaluation settings, and original run records together so the team can reproduce your result.
- Model & harness
- The exact system you evaluatedModel ID, agent framework and version, code commit, and the command used to run the evaluation.
- Scope & settings
- A clearly defined evaluationBenchmark revision, capabilities and task IDs, reasoning effort, sampling settings, time and token budgets, and attempts per task.
- Run evidence
- The records behind every scoreAgent trajectories, tool calls, final artifacts, and verifier outputs. Include failed runs, timeouts, and errors, with their treatment in the reported metrics.
02 · Format
One archive. Everything to review.
Use .zip, .tar.gz, or .tgz. Place metadata.json, eval.json, and runs/ at the archive root, without an extra enclosing folder.
- metadata.json
- Who ran what, and howRequired:
harness,model, andbenchmark_version. Add the exact commits, settings, reproduction command, and contact details from the template. - eval.json
- Scores and evaluation coverageInclude
overall_score,task_count,completed_count, and per-capability measurements. Completed means a run has a grader result, including graded failures; it does not mean passed. - runs/
- Original per-task evidenceCopy your runner’s output directories with their task and trial IDs. Preserve the native layout and all attempts, along with logs, artifacts, and verifier results.
metadata.json · template
Replace the placeholders with your run configuration. Use null only for settings that do not apply.
{
"harness": "YOUR_HARNESS",
"harness_version": "VERSION_OR_COMMIT",
"model": "EXACT_MODEL_ID",
"benchmark_version": "SLE-v1",
"benchmark_commit": "BENCHMARK_COMMIT_SHA",
"capabilities": ["hd", "pp", "sc", "so", "wl", "eo", "e2e"],
"run_config": {
"reasoning_effort": "YOUR_SETTING",
"temperature": null,
"seed": null,
"timeout_seconds": null,
"token_budget": null,
"attempts_per_task": 1
},
"reproduction_command": "YOUR_EVALUATION_COMMAND",
"code_url": "https://github.com/YOUR_ORG/YOUR_AGENT",
"contact": {
"name": "YOUR_NAME",
"email": "YOUR_EMAIL",
"organization": "YOUR_ORGANIZATION"
},
"notes": "Describe any changes to the standard evaluation settings."
}eval.json · template
Fill in measured scores and counts. Remove capabilities you did not evaluate; leave unmeasured cost or time as null. For Hypothesis Discovery, report the Inspire rate with its positives and hypothesis versions.
{
"overall_score": null,
"task_count": 0,
"completed_count": 0,
"capabilities": {
"hd": {
"metric": "inspire_rate",
"score": null,
"positive_count": 0,
"hypothesis_version_count": 0
},
"pp": {
"metric": "mean_score",
"score": null,
"task_count": 0,
"completed_count": 0,
"cost_usd_per_task": null,
"agent_time_seconds_per_task": null
},
"sc": {
"metric": "pass_rate",
"score": null,
"task_count": 0,
"completed_count": 0,
"cost_usd_per_task": null,
"agent_time_seconds_per_task": null
},
"so": {
"metric": "pass_rate",
"score": null,
"task_count": 0,
"completed_count": 0,
"cost_usd_per_task": null,
"agent_time_seconds_per_task": null
},
"wl": {
"metric": "not_scored",
"score": null,
"task_count": 0,
"completed_count": 0,
"note": "Wet-lab Execution is in construction and is not part of the v1 composite."
},
"eo": {
"metric": "pass_rate",
"score": null,
"task_count": 0,
"completed_count": 0,
"cost_usd_per_task": null,
"agent_time_seconds_per_task": null
},
"e2e": {
"metric": "end_to_end_research_score",
"score": null,
"task_count": 1,
"completed_count": 0,
"note": "End-to-end research (To be expanded) is included in the composite on a 0–100 scale. Its progress rubric is a separate diagnostic; do not substitute it for the official score."
}
}
}- Hypothesis Discovery
hd - Inspire rate · 0–100%
- Protocol Planning
pp - Mean score · 0–100
- Scientific Computing
sc - Pass rate · 0–100%
- Scientific Software Operation
so - Pass rate · 0–100%
- Wet-lab Execution
wl - In construction · not scored in v1
- Experiment Observation
eo - Pass rate · 0–100%
- End-to-end research (To be expanded)
e2e - Score on a 0–100 scale · included in All
overall_score to null and state the evaluated subset. Never replace missing measurements with zero. Report cost in USD per task and agent execution time in seconds per task. Describe the averaging rule, pricing, and repeated trials in your notes.# Run inside your completed template directory
zip -r sle-results.zip metadata.json eval.json runs/The downloaded template is unfilled. Replace its example values and add your real runs before submitting.
03 · Submit
Send it to the review team.
Email your completed archive to ly1988@princeton.edu and wuyc@stanford.edu. For larger archives, include a download link and make sure both reviewers can access it.
Use the subject [SLE Results] <harness> / <model> / SLE-v1. Include your name, organization, evaluated capabilities, and reproduction command.
Opens a draft in your email app. Attach the archive or paste its link, review the details, then send. Nothing is uploaded by this page.
What happens after I send my results?
The team checks the package, follows up if information is missing, and reproduces the evaluation under the same settings before publishing a leaderboard entry.
Can I submit a subset or a different evaluation setup?
Yes. List the exact tasks, capabilities, budgets, and any changes to the standard setup. Partial evaluations remain separate from the complete All score. End-to-end research is included in All; retain its original verifier output in the submitted evidence.