HOMODYNE BENCHMARK LAB / HMD-002 · E01
01 / 14 · 00:00

3 of 4 answers wrong. Still the winner.

OPENAI o4-mini · ACCURACY24%

Error rate: hidden for now

gpt-5-thinking-mini · ACCURACY22%

Error rate: hidden for now

SimpleQA · as published by OpenAI, 5 Sep 2025 · historical table, not today's ranking

HOMODYNE BENCHMARK LAB / HMD-002 · E01
02 / 14 · 00:44 · WHO BUILDS THE SCORE?

"Can I return these shoes?"

RIGHTStates the real return policy.
WRONG, CONFIDENTLYStates a policy that does not exist.
"LET ME CHECK"Hands the customer to a human.

سه پاسخ، سه هزینهٔ متفاوت. قیمت یک جواب غلط چقدر است؟

HOMODYNE BENCHMARK LAB / HMD-002 · E01
03 / 14 · 01:42 · GUESS FIRST

Which assistant do you hire?

ASSISTANT A · per 100 questions60 correct

30 wrong · 10 "I don't know"

ASSISTANT B · per 100 questions50 correct

5 wrong · 45 "I don't know"

ILLUSTRATIVE · made-up teaching numbers, not a real model

HOMODYNE BENCHMARK LAB / HMD-002 · E01
04 / 14 · 02:44 · THE PRICE OF A WRONG ANSWER

score = correct − k × wrong

k = 0A 60 · B 50
A wins
k = 0.4A 48 · B 48
tie: the crossover
k = 1A 30 · B 45
B wins
k = 3A −30 · B 35
B wins

ILLUSTRATIVE · "I don't know" = 0 · crossover k = (60−50)/(30−5) = 0.4

HOMODYNE BENCHMARK LAB / HMD-002 · E01
05 / 14 · 03:47 · RUN IT YOURSELF
$ python3 tools/bluff_lab.py demo # excerpt "source_status": "ILLUSTRATIVE", ... "3": { "A": -30, "B": 35 }, "9": { "A": -210, "B": 5 } }, "threshold": "k=t/(1-t) assumes ... Stating t does not make a model calibrated."

Excerpt of the real output (2 Nov 2026) · k = 0 → A 60, B 50 · k = 1 → A 30, B 45 · data ILLUSTRATIVE

HOMODYNE BENCHMARK LAB / HMD-002 · E01
06 / 14 · 05:02 · THE SOURCE

Same test. Read the error row.

gpt-5-thinking-mini22% correct

52% abstain · 26% wrong

OpenAI o4-mini24% correct

1% abstain · 75% wrong

SimpleQA · OpenAI, "Why Language Models Hallucinate", 5 Sep 2025 · from the GPT-5 System Card · historical

HOMODYNE BENCHMARK LAB / HMD-002 · E01
07 / 14 · 06:07 · WHY GUESSING PAYS

9 of 10 popular evals give no credit for "I don't know".

GPQAMMLU-ProIFEvalOmni-MATHWildBench · partialBBHMATH L5MuSRSWE-benchHLE

Unknown birthday: "I don't know" = 0 for sure · a random date ≈ 1 in 365 · Kalai et al. 2025, Table 2 (arXiv 2509.04664)

HOMODYNE BENCHMARK LAB / HMD-002 · E01
08 / 14 · 07:06 · THE KONKUR RULE: k = 1/3
$ python3 artifact/two_scores.py t (confidence target) k OpenAI o4-mini gpt-5-thinking-mini winner 0.00 0.000 24.0 22.0 OpenAI o4-mini 0.25 0.333 -1.0 13.3 gpt-5-thinking-mini 0.50 1.000 -51.0 -4.0 gpt-5-thinking-mini 0.75 3.000 -201.0 -56.0 gpt-5-thinking-mini Crossover: k = 0.0408 points per wrong answer (confidence target t = 3.9%). A wrong answer costing 1/24.5 of a right one is enough to flip the winner.

Real output (2 Nov 2026), t = 0.90 row omitted · re-scoring, not re-running

HOMODYNE BENCHMARK LAB / HMD-002 · E01
09 / 14 · 08:09 · BUT ONE PART OF THIS LIMPS

Rescoring ≠ re-running

WHAT WE DIDsame answers

New rule applied after the fact. The models never knew about a penalty.

WHAT BLUFF WILL ASKnew answers

Tell the model "wrong = −1/3" before it answers. Does it guess less, or just answer less?
[not run yet]

HOMODYNE BENCHMARK LAB / HMD-002 · E01
10 / 14 · 09:06 · WHERE DOES k COME FROM?

k = t / (1 − t)

t 0.25 → k 1/3t 0.50 → k 1t 0.75 → k 3t 0.90 → k 9

4-choice test, blind guess: ¼ × (+1) + ¾ × (−⅓) = 0

t = "answer only if at least this sure" · stating t does not make a model calibrated

HOMODYNE BENCHMARK LAB / HMD-002 · E01
11 / 14 · 10:14 · A SCORE'S ID CARD
1 · DECISIONWhich real choice?
2 · TASKWhat did the model do?
3 · DATAWhich items, which language?
4 · RUNVersions, prompts, repeats?
5 · SCOREWhat is "I don't know" worth?
6 · BOUNDARYWhat does it not cover?

artifact/claim-check-card.md · one page

HOMODYNE BENCHMARK LAB / HMD-002 · E01
12 / 14 · 11:02 · CLAIM BOUNDARY
THIS SHOWSWith fixed answers, changing the scoring rule can flip the winner — in a teaching example and in OpenAI's published SimpleQA table.
THIS DOES NOT SHOWWhich model is better today · that accuracy is useless · one right rule for every job · how models behave when told the penalty.
HOMODYNE BENCHMARK LAB / HMD-002 · E01
13 / 14 · 11:42 · THE ANSWER

Better — under which rule?

OPENAI o4-mini24% correct

75% wrong · wins only while wrong is free

gpt-5-thinking-mini22% correct

26% wrong · wins from k ≈ 0.04

تمرین: یک جواب غلط در کار تو چند برابر یک جواب درست ضرر دارد؟ همان k توست.

HOMODYNE BENCHMARK LAB / HMD-002 · E01
14 / 14 · 12:25 · NEXT · HMD-003

One hot dot from space. Fire — or not?

FALSE ALARMWe say fire. It was a factory.
MISSWe say nothing. It was a fire.
THE DATANASA FIRMS hot spots · northern Iran