HOMODYNE BENCHMARK LAB · E03 / HMD-006
01 / 14 · 00:00

The world’s lowest-error assistant says only: “I don’t know.”

ERRORS0%
USEFUL ANSWERS0%
HOMODYNE BENCHMARK LAB · E03 / HMD-006
02 / 14 · 00:45

Every answer is one of three

CORRECTRight answer.
WRONGConfident and false — a “hallucination”.
“I DON’T KNOW”Abstains.

هر نرخ خطا این سه را به یک شکل خاص تقسیم کرده.

HOMODYNE BENCHMARK LAB · E03 / HMD-006
03 / 14 · 01:40 · GUESS FIRST

10 answers: 5 correct · 2 wrong · 3 “I don’t know”. Error rate = ?

✓✓✓✓✓✗✗???

ILLUSTRATIVE · made-up teaching cards

HOMODYNE BENCHMARK LAB · E03 / HMD-006
04 / 14 · 02:40
WRONG ÷ ALL QUESTIONS2/10 = 20%
WRONG ÷ ANSWERED2/7 ≈ 29%
WRONG ÷ NOT CORRECT (AA-Omniscience)2/5 = 40%

چندتا، از چندتا؟

HOMODYNE BENCHMARK LAB · E03 / HMD-006
05 / 14 · 03:40 · MID-CTA · GUESS FIRST

SimpleQA · two OpenAI models

ERROR RATE26% vs 75%

gpt-5-thinking-mini vs o4-mini

ACCURACY22% vs 24%

the older model answered slightly more correctly

ABSTENTION (“I don’t know”)52% vs 1%

the column headlines skip

HOMODYNE BENCHMARK LAB · E03 / HMD-006
06 / 14 · 04:45

When it does answer, how often is it wrong?

gpt-5-thinking-mini26/48 ≈ 54%
o4-mini75/99 ≈ 76%
DRUG DOSE“I don’t know” beats a confident wrong answer.
CAFÉ NAMESA bad guess costs little.
HOMODYNE BENCHMARK LAB · E03 / HMD-006
07 / 14 · 05:50 · GUESS FIRST
$ python3 tools/bluff_lab.py score tools/bluff-demo.csv --condition t75 # correct +1 · abstain 0 · wrong −3 model accuracy coverage risk score/prompt A 0.50 0.70 0.29 −0.10 B 0.40 0.50 0.20 +0.10 source_status: ILLUSTRATIVE

A = the ten cards (5/2/3) · B = 4 correct / 1 wrong / 5 abstain

HOMODYNE BENCHMARK LAB · E03 / HMD-006
08 / 14 · 06:55
RISK OF “ALWAYS I DON’T KNOW”0 ÷ 0

undefined — not 0%

PENALISED SCORE0

the line of silence

ABSTENTIONBENCH · 20 models · 20 datasets−24%

abstention after reasoning fine-tuning (average)

HOMODYNE BENCHMARK LAB · E03 / HMD-006
09 / 14 · 07:50 · RE-HOOK

Both called “hallucination rate”

VECTARA · lowest listed1.8%

Summarise a given document. Rate = 100 − % consistent with it.

AA-OMNISCIENCE · GPT-5 (high)81%

Answer from memory, no document. Rate = wrong ÷ (wrong + partial + abstain).

different task · different formula · different models → not comparable

HOMODYNE BENCHMARK LAB · E03 / HMD-006
10 / 14 · 08:55

What we want ≠ what we count

CONSTRUCTWhat we want: an answer you can rely on.
PROXYWhat we count: wrong ÷ answered, on short questions.
MISSES“I don’t know” share · long answers · Persian · date.

thermometer ≠ health · 445 LLM benchmarks · 29 expert reviewers · 8 recommendations

HOMODYNE BENCHMARK LAB · E03 / HMD-006
11 / 14 · 09:55

Construct card · fill it for any “fewer errors” claim

1 · WANTAn answer you can lean on.
2 · COUNTEDWrong answers on short questions, no document.
3 · NUMERATOR / DENOMINATORWrong ÷ all? ÷ answered? ÷ not-correct?
4 · MISSESAbstention share · long answers · Persian.

Bluff spec starts here · not run yet · measures answering behaviour, not honesty

HOMODYNE BENCHMARK LAB · E03 / HMD-006
12 / 14 · 10:55 · CLAIM BOUNDARY
THIS SHOWSHow each error rate is built — and that “fewer errors” can be bought with abstention.
THIS DOES NOT SHOWWhich model is better · that Vectara or AA are bad tests · what a model “knows” · current rankings. Cards and A/B are ILLUSTRATIVE.
HOMODYNE BENCHMARK LAB · E03 / HMD-006
13 / 14 · 11:40

Out of what? Bought with what?

✓✓✓✓✓✗✗???

پس: «کمتر خطا» معمولاً نمی‌گوید از چندتا، و به قیمت چند «نمی‌دانم».

HOMODYNE BENCHMARK LAB · E03 / HMD-006
14 / 14 · 12:20 · NEXT

Next: what if the document itself is steering the AI?

INVOICE #0417 · Fictional Co. Total: 12,400,000 IRR Due: 30 Aban … [a line written for the assistant, not for you]