# TypeSafe Parallel Judgment Lab — experiment report

Generated from saved live responses: 2026-08-28T19:55:23.191Z

## Bottom line

11 experiments produced 1,709 typed judgments across 299 TypeSafe API calls for an estimated $0.0081492. The site replays the recorded probability, retrieval, agent-tool, and timing data from these runs.

| # | Experiment | Calls | Judgments | Measured wall time | TypeSafe cost | Primary result |
| ---: | --- | ---: | ---: | ---: | ---: | --- |
| 1 | Code repository RAG | 8 | 48 | 385.1 ms | $0.0001856 | Recall@1 100%; MRR 1 |
| 2 | Agent navigation race | 30 | 30 | 558.6 ms | $0.0004649 | Claude + TypeSafe 25.0s vs manual 32.9s; both correct |
| 3 | Company brain | 10 | 70 | 509.5 ms | $0.0002385 | Recall@3 100%; MRR 1 |
| 4 | Judge grid | 30 | 180 | 191.7 ms | $0.0006304 | 89% label agreement; 0.0013 repeat variation |
| 5 | Agent action firewall | 10 | 50 | 325.9 ms | $0.0001772 | 100% decision agreement |
| 6 | VC diligence matrix | 8 | 64 | 386.9 ms | $0.0003406 | 75% tier agreement |
| 7 | AI writing-pattern checker | 37 | 777 | 611.9 ms | $0.0026451 | 100% paired directional separation; +0.245 mean P(AI) lift |
| 8 | Which customers need help first? | 24 | 144 | 675.5 ms | $0.0004594 | Highest support priority: Maple Labs (0.907) |
| 9 | Which decisions need the CEO? | 18 | 126 | 206.2 ms | $0.0004791 | Highest CEO priority: Should the publication keep a hard paywall? (0.759) |
| 10 | Which emails need a reply today? | 24 | 120 | 348.4 ms | $0.0005504 | 100% synthetic-label agreement; 12 reply-today emails |
| 11 | Which ad would 100 AI personas click? | 100 | 100 | 527.3 ms | $0.0019780 | Ad B won; 49% modeled click share; 50 persona selections |

## Experiment setup

Each runner sends one document or scenario plus a JSON object of typed questions to `https://api.typesafe.ai/v1/systemone` with model `speed_latest`. The lab demonstrates Noul yes/no probabilities, Choice classifications with full probability distributions, and Score rubrics with expected values. Retrieval ranks probabilities; routing and triage experiments combine typed answers with visible formulas in code.

The exact inputs are in `fixtures/`. The exact question text and saved row-level probabilities are in `public/experiments.json`. The complete runnable walkthrough is in `notebooks/TypeSafe_Parallel_Judgment_Lab.ipynb`.

## Scope and limitations

- The saved JSON includes every displayed probability, per-call latency, token usage, and estimated cost.
- `experiments/RUN_MANIFEST.json` records totals, runner commands, and SHA-256 hashes for the data artifacts.
- Customer-support and email examples are synthetic.
- The CEO radar uses synthetic, de-identified Slack-style threads inspired by recent company themes; raw Slack messages are not published.
- The agent-navigation race is one run per condition on a synthetic repository, not a statistically powered agent benchmark.
- The company-brain comparison uses QMD 2.8.3 keyword and vector search over the same ten markdown documents; QMD index and embedding setup are reported separately from query time.
- The AI-writing checker detects style patterns, not authorship.
- The 100-person persona panel predicts synthetic stated preference, not real ad clicks.
- Artifact hashes make it easy to detect changes to the recorded experiment data.