MEASURED 2026.08.28 PROBABILITIES, NOT PROSE

What if judgment
were parallel?

Eleven practical experiments with a model that returns probabilities instead of prose. Find an answer, watch an agent navigate, compare retrieval methods, sort an inbox, or ask 100 synthetic personas which ad they would click.

1709typed judgments
299live API calls
$0.0081estimated TypeSafe spend

a search box for a codebase

Find the right file for a coding question

An engineer asks a normal question. The agent turns it into a yes-or-no check for each file, TypeSafe scores every file at once, and the best match rises to the top.

wall time385 mslive measurement
judgments488 parallel calls
TypeSafe cost$0.000186at $0.042 / 1M input

Recorded run · speed_latest

0 ms

0/8 calls returned · 0/48 judgments1× saved timing

yes/no question ↓ / source →
auth/session
cache/profile
db/ledger
email/digest
flags/rollout
observability/tracing
payments/reconcile
webhooks/stripe
Does this file contain the implementation of refresh-token rotation and replay protection?
·
·
·
·
·
·
·
·
Does this file contain the logic that prevents a retried request from creating a duplicate payout?
·
·
·
·
·
·
·
·
Does this file contain code that verifies Stripe webhook signatures and ignores duplicate events?
·
·
·
·
·
·
·
·
Does this file explain or implement how to correct a ledger entry without deleting it?
·
·
·
·
·
·
·
·
Does this file contain the logic that keeps a user in the same feature-flag cohort across deploys?
·
·
·
·
·
·
·
·
Does this file contain code that continues a request trace after work moves onto a queue?
·
·
·
·
·
·
·
·
TypeSafe Recall@1
Recall@3
MRR
BM25 Recall@1
Readout

On this small corpus, the correct file ranked first for every question. In ordinary terms: it behaves like very fast semantic search for an agent that needs to decide what to read next.

experiment design

What went in, what we asked, and how the score became a decision.

01 · Input8 files in a small, deliberately fake payments codebase: authentication, payments, webhooks, database rules, feature flags, tracing, and two distractors.
02 · TypeSafe call6 file-level yes/no questions were sent in parallel.
03 · Decision codeSort files by P(the file-level question is true). No answer text is generated.
See all 8 input items +
auth/session.tssession.tsread full input +
export async function rotateRefreshToken(sessionId: string, presentedToken: string) { const session = await sessions.lock(sessionId); if (session.refreshTokenHash !== hash(presente
cache/profile.tsprofile.tsread full input +
export async function readProfile(userId: string) { const key = `profile:${userId}`; const hit = await redis.get(key); if (hit) return JSON.parse(hit); const profile = await db.pro
db/ledger.sqlledger.sqlread full input +
CREATE TABLE ledger_entries ( id UUID PRIMARY KEY, account_id UUID NOT NULL, amount_cents BIGINT NOT NULL, reversal_of UUID REFERENCES ledger_entries(id), created_at TIMESTAMPTZ NO
email/digest.tsdigest.tsread full input +
export async function sendWeeklyDigest(user: User, stories: Story[]) { if (!user.preferences.weeklyDigest) return; const html = renderDigest(stories.slice(0, 10)); await mailer.sen
flags/rollout.tsrollout.tsread full input +
export function isInRollout(flag: string, actorId: string, percentage: number) { const bucket = stableHash(`${flag}:${actorId}`) % 10_000; return bucket < percentage * 100; } // St
observability/tracing.tstracing.tsread full input +
export function publishJob(name: string, payload: unknown) { return queue.publish(name, payload, { traceparent: activeSpan()?.traceparent, correlationId: requestContext.correlation
payments/reconcile.tsreconcile.tsread full input +
export async function createPayout(request: PayoutRequest) { return db.transaction(async (tx) => { const existing = await tx.payouts.findByIdempotencyKey(request.idempotencyKey); i
webhooks/stripe.tsstripe.tsread full input +
export async function stripeWebhook(rawBody: Buffer, signature: string) { const event = stripe.webhooks.constructEvent(rawBody, signature, env.STRIPE_SECRET); if (await events.wasP
See the exact 6 questions +
  1. refresh_rotationDoes this file contain the implementation of refresh-token rotation and replay protection?Noul
  2. duplicate_payoutDoes this file contain the logic that prevents a retried request from creating a duplicate payout?Noul
  3. webhook_verifyDoes this file contain code that verifies Stripe webhook signatures and ignores duplicate events?Noul
  4. ledger_deleteDoes this file explain or implement how to correct a ledger entry without deleting it?Noul
  5. stable_rolloutDoes this file contain the logic that keeps a user in the same feature-flag cohort across deploys?Noul
  6. queue_traceDoes this file contain code that continues a request trace after work moves onto a queue?Noul
Performance and cost +
Modelspeed_latestAPI calls8Judgments48Measured wall time385.1 msEstimated spend$0.000186Results dataJSON ↘

Limitations: This is a clean toy repository. A production test should use a larger repo and human relevance labels.

how the engine works

One item. Many small questions. One useful action.

  1. 01
    Give it an item

    Send a code file, email, customer message, article, or other piece of text.

  2. 02
    Choose the answer shape

    Use Noul for yes/no probability, Choice for a classification, or Score for an ordered rubric.

  3. 03
    Get every judgment together

    TypeSafe evaluates the questions in parallel and returns probabilities, selected classes, scores, and confidence instead of prose.

  4. 04
    Turn scores into an action

    Ordinary code ranks the results, applies thresholds, and sends uncertain cases to a person.

reproduce the lab

Explore every experiment in detail.

Each experiment includes its inputs, exact typed questions, answers, timings, and decision logic. Use the notebook for a guided walkthrough or download the complete source to run the lab yourself.