HOMODYNE BENCHMARK LAB / E04 · HMD-008
01 / 14 · 00:00

Same 100 questions. The model that knew more lost.

MODEL K · knows more73 / 100

Capital of Australia? → Canberra ✗

MODEL P · repeats popular beliefs80 / 100

Capital of Australia? → Sydney ✓

ILLUSTRATIVE · KEY v1 says “Sydney” on purpose · no real model was called

HOMODYNE BENCHMARK LAB / E04 · HMD-008
02 / 14 · 00:47

Exams that admit their key can be wrong.

01 · PRELIMINARY KEYPublished a few days after the exam
02 · OBJECTIONSCandidates flag disputed items
03 · FINAL KEYItems corrected or removed
AI LEADERBOARD?Often: one table, no objection window

کنکور برای کلیدش مهلت اعتراض دارد؛ خیلی از امتحان‌های AI نه.

HOMODYNE BENCHMARK LAB / E04 · HMD-008
03 / 14 · 01:59 · GUESS FIRST
6.49%of MMLU questions estimated to contain errors · Virology: 57% of analysed questions

10 popular ML test sets: ≥ 3.3% label errors on average · ImageNet validation: ≥ 6%

HOMODYNE BENCHMARK LAB / E04 · HMD-008
04 / 14 · 03:17

A wrong key only takes points from those who answered right.

KEY RIGHTKEY WRONGMODEL RIGHT
+1 · deserved
0 · punished for being right
MODEL WRONG
0 · deserved
0 · nothing to lose

expected loss ≈ accuracy × key error rate → the stronger model pays more

HOMODYNE BENCHMARK LAB / E04 · HMD-008
05 / 14 · 04:30

Random error blurs. A popular myth picks a winner.

RANDOM TYPOKey matches nobody. Every correct answer loses a point.
POPULAR MYTHKey = Sydney · Istanbul · Toronto. The model that shares the myth scores.

اگر این‌جور باز کردن عددها به کارت می‌آید، هومودین هر هفته یکی‌اش را باز می‌کند.

HOMODYNE BENCHMARK LAB / E04 · HMD-008
06 / 14 · 05:36 · GUESS: WHO LEADS?
$ python3 artifact/key_audit_demo.py --write-csv key v1 (8 myth errors) K= 73 P= 80 leader=P key v2 (adjudicated) K= 80 P= 76 leader=K random typos instead K= 73 P= 74 $ python3 tools/bluff_lab.py score artifact/key-v2.csv --condition plain "accuracy_all_prompts": K 0.80 · P 0.76

ILLUSTRATIVE · fictional models · planted errors

HOMODYNE BENCHMARK LAB / E04 · HMD-008
07 / 14 · 07:11
With corrected labels, ResNet-18 beats ResNet-50 once originally mislabeled examples rise by just 6%.Northcutt, Athalye, Mueller · Pervasive Label Errors in Test Sets · NeurIPS 2021 · ImageNet · paraphrase

مدل تشخیص تصویر، نه چت‌بات؛ ولی همان پیام: «بهتر» به کلید بستگی دارد.

HOMODYNE BENCHMARK LAB / E04 · HMD-008
08 / 14 · 08:06

Where wrong keys come from

MYTHSydney ≠ Canberra
DATEAstana → Nur-Sultan (2019) → Astana (2022)
ALIASBrasília = Brasilia
TRANSLATIONGlobal-MMLU: 28% of questions need cultural knowledge · 84.9% of geography Qs on N. America / Europe
HOMODYNE BENCHMARK LAB / E04 · HMD-008
09 / 14 · 09:24 · RE-HOOK

The denominator decides what the number means.

suspicious only 8/3622%
random 30 · 3/3010%
planted truth8%

random audit 95% Wilson interval: 3% – 26% · ILLUSTRATIVE

HOMODYNE BENCHMARK LAB / E04 · HMD-008
10 / 14 · 10:52
q001 Capital of Australia? R1=error R2=error q005 Capital of Nigeria? R1=error R2=error q006 Capital of Myanmar? R1=error R2=ok ← disagreement raw agreement 11/12

Disagreement → primary source → decision → logged reason. Not a vote.

پیدا نکردن منبع یعنی «پیدا نکردم»، نه «جوابی نیست».

HOMODYNE BENCHMARK LAB / E04 · HMD-008
11 / 14 · 11:59
BLUFF PILOT KEY AUDITNOT RUN YET

60 question files · audit results will be published separately

SIMPLEQA VERIFIED1,000 prompts

rebuilt to fix incorrect labels, topical bias, redundancy

HOMODYNE BENCHMARK LAB / E04 · HMD-008
12 / 14 · 13:08 · CLAIM BOUNDARY
THIS SHOWSA wrong key removes points only from correct answers; a key that repeats a popular myth can reward the model that shares it. Published audits found key errors in famous benchmarks.
THIS DOES NOT SHOWThat any specific leaderboard today is wrong. Our quiz is ILLUSTRATIVE. 6.49% is one estimate for MMLU, not every benchmark.
HOMODYNE BENCHMARK LAB / E04 · HMD-008
13 / 14 · 13:54 · ANSWER

Judge the key before you judge the model.

1Was the key audited?
2Out of how many — chosen how?
3Which key version?

جواب کوتاه: معمولاً همانی می‌بازد که درست جواب داده.

HOMODYNE BENCHMARK LAB / E04 · HMD-008
14 / 14 · 14:43 · NEXT

What if reality wrote the answer key?

ETA 22 min → 22 minutes later, you know. Next: how Google Maps predicts traffic that does not exist yet.