HOMODYNE BENCHMARK LAB / HMD-004 · E02
01 / 14 · 00:00

Rank 6 → ? Same votes. One thing removed.

BEFORE6

a well-known model, summer 2024

AFTER?

answer at 01:36

HOMODYNE BENCHMARK LAB / HMD-004 · E02
02 / 14 · 00:41 · GUESS FIRST

Cooking-show winner. Restaurant manager?

JOB 1Answer a factual question
JOB 2Summarise a document
JOB 3Change a booking with a tool
QUESTIONWhich one does a vote-based leaderboard test?
HOMODYNE BENCHMARK LAB / HMD-004 · E02
03 / 14 · 01:36 · WHAT DOES A VOTE COUNT?

Preferred ≠ verified.

Style control: GPT-4o-mini 6 → 11 · Claude 3 Opus 16 → 10

ANSWER 1 · ILLUSTRATIVElong, confident

Headings, bullets, a warm tone — and the wrong return window.

ANSWER 2 · ILLUSTRATIVEshort, plain

"Seven days, unworn, with receipt." Correct.

LMSYS, 29 Aug 2024: response length was the dominant style factor; markdown effects were second-order; "our analysis is still observational"

HOMODYNE BENCHMARK LAB / HMD-004 · E02
04 / 14 · 02:50 · A REPORT CARD, NOT A RANK

A model has a profile, not a rank.

X · knowledge80
Y · knowledge65
X · grounded55
Y · grounded78
X · tool use70
Y · tool use60

ILLUSTRATIVE · made-up scores · one overall rank hides whose weights were used

HOMODYNE BENCHMARK LAB / HMD-004 · E02
05 / 14 · 03:57 · START FROM THE FAILURE
DECISIONWhich model answers our shop's customers?
COSTLY FAILUREA wrong refund policy, stated confidently.
REQUIRED EVIDENCECorrectness against a reference + abstaining when unsure.
FAMILY…falls out last.

نیمهٔ مفید زنجیره، حذف‌هاست: کدام جدول مشهور به سؤال تو جواب نمی‌دهد؟

HOMODYNE BENCHMARK LAB / HMD-004 · E02
06 / 14 · 04:50 · MY OWN TOOL GOT IT WRONG
$ python3 -m m2bench families \ --decision "customer support bot for my online shop" \ --failure "tells customers the wrong refund policy" Closest worked example: decision Can we let this system act unattended in a customer workflow? costly failure It completes the wrong action irreversibly. required evidence Task success and failure modes in a live environment. families agents_tool_use, safety_security, robustness NOT knowledge, reasoning

Real output, excerpt (2 Nov 2026) · our bot only answers — it does not act

HOMODYNE BENCHMARK LAB / HMD-004 · E02
07 / 14 · 05:44 · SAME NEED, DIFFERENT WORDS
$ python3 -m m2bench families \ --decision "question answering assistant for my online shop" \ --failure "a confident wrong answer about refunds" families calibration_abstention, factuality, multilingual_cultural diagnostic only knowledge NOT human_preference, reasoning, agents_tool_use why not Preference votes reward length and confident style — the behaviour under test. ... built as bluff-v1

Real output, excerpt (2 Nov 2026) · the tool matches words (longer than 4 letters) with worked examples

HOMODYNE BENCHMARK LAB / HMD-004 · E02
08 / 14 · 06:43 · LEARN THE BLIND SPOTS, NOT THE NAMES
CAPABILITY · codingIncomplete tests miss real bugs.
TRUST · safetyRefusing more ≠ being safer.
VALUE · human preferencePreference is not truth.
TRUST · "I don't know"Abstain on everything → zero errors, zero use.

16 families · 4 groups · two axes: static ↔ live, reference answer ↔ human/judge/outcome · a navigation aid, not a formal taxonomy

HOMODYNE BENCHMARK LAB / HMD-004 · E02
09 / 14 · 08:01 · BUT IS THIS TEST STILL ALIVE?
≥ 59.4%of 138 audited SWE-bench Verified problems had flawed tests · OpenAI, 23 Feb 2026

The 138 were problems the model kept failing — not a random sample of the whole set · contamination signs also reported

HOMODYNE BENCHMARK LAB / HMD-004 · E02
10 / 14 · 08:57 · A SUCCESSOR NEEDS AN AUDIT TOO
FEB 2026Stop reporting Verified; point to SWE-bench Pro
8 JUL 2026Pro recommendation retracted · ~30% of tasks estimated broken
freshdiscriminatingsaturatingsaturatedcompromisedretired

Lifecycle states are dated (as of 2026-09-29) and need not form a line · saturated ≠ compromised

HOMODYNE BENCHMARK LAB / HMD-004 · E02
11 / 14 · 10:03 · OPEN ANY BENCHMARK IN FIVE QUESTIONS
1 · CLAIMKnows short facts?
2 · TASKShort questions, one verifiable answer
3 · METRICcorrect / wrong / not attempted (HMD-002)
4 · BLIND SPOTLong answers, summaries, answering from a document
5 · LIFERevised as SimpleQA Verified · 1,000 prompts
HOMODYNE BENCHMARK LAB / HMD-004 · E02
12 / 14 · 10:57 · CLAIM BOUNDARY
THIS SHOWSA rank depends on what is counted · a reference benchmark can break and be retired · how you write the decision and failure changes the evidence you get.
THIS DOES NOT SHOWThat preference leaderboards are useless · which model is best today · a formal taxonomy (families are our map) · a fixed lifecycle timeline · a tool that decides for you.
HOMODYNE BENCHMARK LAB / HMD-004 · E02
13 / 14 · 11:34 · THE ANSWER

#1 for you only if it measures your costly failure.

TASKAnswer customers' questions
COSTLY FAILUREConfident wrong policy
EVIDENCECorrect vs reference + abstains when unsure
TESTFactuality + abstention, not a vote table

تمرین: کارت چهارخانهٔ کار خودت را پر کن. کدام جدول مشهور را کنار می‌گذاری؟

HOMODYNE BENCHMARK LAB / HMD-004 · E02
14 / 14 · 12:18 · NEXT · HMD-005

Same sentence, English vs Persian: count the pieces.

Themodelreadsthis·مدلاینرامی‌خواند

Schematic split, not a real tokenizer output · the real count comes from the next episode's run