# SLE preliminary submission kit

Status: research preview, 24 October 2026. This kit prepares a review packet. It does not contain runnable task packages, reference answers, an evaluator, or an approved benchmark release. Validation is not leaderboard acceptance.

## Prepare a packet

1. Get the agreed code, task inputs and environment version from the SLE team. Public downloads are still being prepared. Contact junyangwang@stu.pku.edu.cn. The website's generated illustrations are not benchmark inputs.
2. Record the exact model version, harness commit, task-content revision, budgets, network profile, command and environment. Do not reuse historical results without recording their origin. Current website configurations do not all use the same harness.
3. Copy `task-results.template.json` or `discovery.template.json`. Fill empty metadata and replace the template submission ID and timestamp. The task template includes all 198 draft public identities as pending; do not turn pending/errors into zero scores.
4. For Representative or Last-Exam, set `benchmark.subset` to `representative` or `last_exam` and retain exactly the matching records in `catalog.json`. For a pilot or different release, use `custom` and document the selection; it cannot establish a full-public result.
5. Add links and SHA-256 hashes for the logs and evaluator output archives. Other evidence types are code, environment and delivered outputs. Access-request links are supported. The validator does not fetch them. Do not include access tokens, model keys, private reference answers, restricted datasets or personal data in public links.
6. Validate locally on the website, or with a supported Node.js installation (Node 18 or later):

```sh
node validate.js submission.json > validation-report.json
```

Exit code 0 means structurally valid; 1 means validation errors; 2 means a file/JSON/usage error. The validator uses only Node's standard library and makes no network requests. It requires `schema.json` and `catalog.json` beside `validate.js`.

7. Email `submission.json` and `validation-report.json` to junyangwang@stu.pku.edu.cn, with subject `SLE result review: <submission_id>`. For large evidence, include authorized access links and checksums. The website can open a draft in your mail application, but it cannot send mail or attach files for you. There is no automatic receipt, review SLA, public PR intake or hosted evaluation service yet.

## Import existing ScienceBench aggregates

For the `aggregate/results.jsonl` format containing `ability`, `task`, `status`, `score`, `score_metric`, and `reused`, fill the template metadata, then run:

```sh
node import-results.js metadata.json aggregate/results.jsonl > submission.json
node validate.js submission.json > validation-report.json
```

Use a new output filename; never redirect over an input. The importer rejects duplicate tasks, unknown formats, out-of-scope records, and Coding scores without the correct primary metric. Missing selected tasks are explicitly filled as pending, and running tasks become pending in the snapshot. It copies only allowed result fields, not source paths or raw traces. Metadata and evidence still need to be completed and checked. Some historical aggregates omit reuse flags for newly executed trials. The default import rejects these. Only after verifying those records were newly executed, add `--missing-reuse-is-new`; the importer records this declaration in notes. Other runner formats require an explicit adapter; this tool does not guess their meaning.

## Task score rules

- Primary scores are fractions between 0 and 1, not percentages. Use `reward_repair` for Scientific Coding; use `reward` for the other three capabilities. Do not substitute raw Coding reward.
- `scored`: score is numeric, including a genuine zero; `error_type` is null.
- `error`: score is null and `error_type` describes the failed evaluation/environment/model stage.
- `pending`: score and `error_type` are null. Include these records so coverage cannot disappear.
- One record per capability + stable task ID. Retries need a declared selection policy in notes; do not submit only successful retries.
- Means and full-credit rates use scored records and report coverage beside them. No complete aggregate is returned for incomplete/custom runs. A full aggregate weights present capabilities equally, not tasks.
- The catalog checksum identifies the internal selection manifest from which public metadata was extracted. It does not fingerprint task content. `task_content_revision` is separate and must be checked by reviewers.

## Hypothesis Discovery

Use only Inspire Rate: actionable versions / generated versions. The conditional rate is actionable / filtered. Counts must satisfy actionable ≤ filtered ≤ generated. Document the literature cutoff, database snapshot, judge and exact judging protocol; supply version-level evidence via the evidence archives. This preview format covers overall counts, not independent domain comparisons. Five-level ratings are not an accepted substitute. The executable Inspire Rate protocol and database access are still being finalized, so these packets are preliminary review requests.

## Review and publication

The team checks access and hashes, verifies task/version coverage and evaluation settings, inspects/reproduces evidence, and requests corrections where needed. A successful format check grants no rank. Keep contact details out of public result exports. New results are added to the public website only after an explicit organizer review and publication decision. See `review-checklist.md`.