Send your evals. Get one honest chart.

EvalPal stores the outcome of your eval runs, not your eval content. You send case IDs, an optional fingerprint of each case, and whether each case passed. EvalPal keeps the test set's history in an append-only ledger and draws the dashboard from it.

Quickstart

1. Create an account, then register an app in the console. An app is one product or system you evaluate, for example "Sous kitchen assistant".
2. Copy the upload key shown after registering. It starts with ep_ and is shown once.
3. After your eval harness runs, post the results:

curl https://evalpal.io/v1/runs \
  -H "Authorization: Bearer $EVALPAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "feature": "allergen-check",
    "feature_name": "Allergen checker",
    "pipeline_version": "v3.0",
    "set_version": "v4",
    "cases": [
      { "id": "ac-0001", "fingerprint": "sha256:9f2c1e…", "pass": true },
      { "id": "ac-0002", "fingerprint": "sha256:41ab07…", "pass": false }
    ]
  }'

4. Open the app in the console to see the chart. Turn on sharing to get a link for anyone else.

Concepts

POST/v1/runs

Records one run of one pipeline version for one feature. Send Authorization: Bearer ep_… and Content-Type: application/json.

The response tells you exactly how the run was scored:

{
  "run_id": 41,
  "feature": "allergen-check",
  "pipeline_version": "v3.0",
  "set_version": "v4",
  "total": 230,          // cases in the set at run time
  "ran": 230,            // of those, how many you sent
  "passed": 218,
  "not_run": 0,          // counted as failed
  "new_cases": 4,        // joined the set with this run
  "changed_cases": 0,    // fingerprint differed from before
  "ignored_removed_cases": [],
  "backfilled": false,
  "results_hash": "b8d2…",
  "ledger": { "seq": 213, "hash": "0a9c…" }
}

POST/v1/cases

Adds cases to a feature's test set without recording a run. Use it when your team finds new failure cases before any version has been tested on them.

{
  "feature": "allergen-check",
  "set_version": "v5",
  "added_at": "2026-10-21T09:00:00Z",   // optional, defaults to now
  "cases": [ { "id": "ac-0231", "fingerprint": "sha256:…", "slice": "hidden-nuts" } ]
}

POST/v1/cases/remove

Removes cases from the set. A reason is required and is shown publicly on the verify page. Removed cases stop counting from the moment of removal. They stay in the ledger, and the chart marks the day in red.

{ "feature": "allergen-check", "ids": ["ac-0198"], "reason": "Duplicate of ac-0201" }

A removed case can't be added back under the same ID. Results for it are ignored and listed in ignored_removed_cases.

GET/v1/whoami

Returns the app a key belongs to. Useful for checking a key in CI.

Errors and limits

Errors return JSON as { "error": "…" } with a status code: 400 for invalid input, 401 for a missing or revoked key, 413 for bodies over 8 MB, 429 when rate limited.

• 120 upload requests per minute per app.
• 50,000 cases per request. Split larger sets across several /v1/cases calls, then send runs.
• 25 apps per account and 10 live keys per app.

Scoring rules

These rules are what make the chart comparable across teams. They can't be switched off.

• A run is scored against the whole set at run time. Cases in the set that you leave out count as failed and are reported as not_run.
• Unknown case IDs join the set with the run's time as their added date.
• Removals need a reason and are shown on the chart and the verify page.
• Edits are flagged. If a case arrives with a different fingerprint than before, it counts as edited.
• Backfills are labelled. Runs uploaded more than a day after their run_at are listed as backfilled under the chart.
• Versions are ordered by first run. The chart shows the three most recent. The latest one is the headline number.

Fingerprints

A fingerprint is any stable hash of the case content: input, expected output and grading criteria. EvalPal never sees the content, only the hash. If the content changes, the hash changes, and the edit is recorded. Use the same recipe every time:

import hashlib, json

def fingerprint(case: dict) -> str:
    canonical = json.dumps(case, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest()

import { createHash } from "node:crypto";

const canonical = (v) => Array.isArray(v) ? `[${v.map(canonical).join(",")}]`
  : v && typeof v === "object" ? `{${Object.keys(v).sort().map((k) => `${JSON.stringify(k)}:${canonical(v[k])}`).join(",")}}`
  : JSON.stringify(v);
export const fingerprint = (c) => "sha256:" + createHash("sha256").update(canonical(c)).digest("hex");

Run from CI

Store the key as a secret and post after the eval step. A GitHub Actions example, assuming your harness writes results.json in the run format:

- name: Run evals
  run: python evals/run.py --out results.json
- name: Upload to EvalPal
  env:
    EVALPAL_KEY: ${{ secrets.EVALPAL_KEY }}
  run: |
    curl --fail-with-body https://evalpal.io/v1/runs \
      -H "Authorization: Bearer $EVALPAL_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @results.json

Re-run your two or three most recent versions on the latest set every day. That's what makes older lines flatten as new cases arrive, and it's the part of the chart that shows real progress.

Reading the chart

• Bars are how many cases were in the set at the end of each day.
• Lines are how many cases each version passed on its last run that day. Hover for percentages and skipped cases.
• Notes above bars mark a new set version (Set v4 +56) or removals (−2 removed).
• The headline is the latest version's pass rate on its latest run, and the change in points against the previous version's latest run.
• The integrity line lists removals, edits, skipped cases, backfilled history and hidden older versions.

Verification

Every write becomes a ledger entry. Each entry's hash is:

sha256( prev_hash + "\n" + seq + "\n" + received_at_ms + "\n" + kind + "\n" + body )

Here body is the entry's JSON exactly as stored, and the first entry's prev_hash is 64 zeros. Entry kinds are feature_registered, feature_renamed, cases_added, cases_changed, cases_removed and run_recorded. Run entries carry a hash of the sorted id\tpass lines, so per-case results can be checked against your own records.

Every hour EvalPal records the head of each changed ledger as an anchor. The verify page at /d/<link>/verify downloads the full ledger from /api/share/<link>/ledger and then, in the viewer's browser:

1. recomputes every hash and checks each entry points at the one before it,
2. checks the chain against every recorded anchor,
3. replays the entries to rebuild each feature's set size and headline numbers, and compares them with the dashboard,
4. lists every removal and edit with its reason, and any backfilled runs.

Anchors are kept by EvalPal, so they prove history hasn't changed since they were recorded, not that EvalPal is honest. For an independent check, save the ledger head yourself, or ask us to verify by running your test set ourselves.