HOMODYNE BENCHMARK LAB / E05 · HMD-010
01 / 15 · 00:00

Question 137. Two options left. Answer or skip?

ABCD

هم داوطلب کنکور و هم AI، هر بار بین «جواب» و «نمی‌دونم» انتخاب می‌کنند.

HOMODYNE BENCHMARK LAB / E05 · HMD-010
02 / 15 · 00:47 · GUESS FIRST
RIGHT+3
WRONG−1
BLANK0

Blind guess, 4 options: ¼ × 3 − ¾ × 1 = 0

HOMODYNE BENCHMARK LAB / E05 · HMD-010
03 / 15 · 01:36
$ python3 artifact/penalty_threshold.py # section 2 blind guess, 4 options left expected value = 0 one option eliminated, 3 left expected value = 1/9 two options eliminated, 2 left expected value = 1/3

in units of one right answer · an average, not a guarantee · only if your 50–50 is really 50–50

HOMODYNE BENCHMARK LAB / E05 · HMD-010
04 / 15 · 02:39

Silence line: answer only if P(correct) > k/(k+1)

k = 00%
Konkur k = 1/325%
k = 150%
k = 375%
k = 990%
HOMODYNE BENCHMARK LAB / E05 · HMD-010
05 / 15 · 03:38

Most AI tests: right +1 · wrong 0 · “I don’t know” 0 → silence line = 0

THE PAPER’S ANALOGYLike students facing hard exam questions, models guess when uncertain.
EXAMS WITH PENALTIES (CITED)JEE · NEET · GATE · AMC · SAT / AP / GRE (earlier years)
PROPOSALState the target in the prompt: answer only if > t confident

اگر این‌جور باز کردن قانون‌های امتحان AI به کارت می‌آید، هومودین هر هفته یکی را باز می‌کند.

HOMODYNE BENCHMARK LAB / E05 · HMD-010
06 / 15 · 04:52
“t = 0.75 (penalty 2)”Kalai et al. 2025 · arXiv v1 · sec. 4.2 — but their own formula t/(1−t) gives 0.75 / 0.25 = 3

جواب‌نامه‌ها هم غلط دارند؛ E04 را یادت هست؟

HOMODYNE BENCHMARK LAB / E05 · HMD-010
07 / 15 · 05:56
RESCOREsame answers

new rule → new number. The model did nothing.

INSTRUCT · NOT RUN YETnew prompt

fresh answers → maybe new behaviour

$ python3 tools/bluff_lab.py score tools/bluff-demo.csv --condition t75 "plain_note": "plain is RESCORED at k=3 for comparison; its prompt did not state the penalty."
HOMODYNE BENCHMARK LAB / E05 · HMD-010
08 / 15 · 07:04 · CALIBRATED ANSWERER · k = 3 · GUESS FIRST
ALWAYS ANSWER−0.79

points per question

ANSWER ONLY ABOVE 75%+0.13

covers 29% · wrong in 14% of answers

ILLUSTRATIVE · 1,000 simulated questions · simulated answerers, not language models

HOMODYNE BENCHMARK LAB / E05 · HMD-010
09 / 15 · 08:24 · RE-HOOK · SAME RULE, k = 3
CALIBRATED+0.13

answers 29% · wrong 14% of answers

OVERCONFIDENT (+25 pts)−0.05

answers 59% · wrong 27% of answers

The rule is fine. The compass is broken.

ILLUSTRATIVE · simulated answerers

HOMODYNE BENCHMARK LAB / E05 · HMD-010
10 / 15 · 09:28

Does a real model listen?

PLAINno rule stated
t50wrong = −1
t75wrong = −3
t90wrong = −9

same questions · fresh answers per prompt · labels: correct / wrong / abstain · read coverage + risk + score · “more cautious” defined before the run

HOMODYNE BENCHMARK LAB / E05 · HMD-010
11 / 15 · 10:33 · NOT RUN YET
A · MORE CAREFULfewer errors, same correct answers
B · NO CHANGEthe score moves, the behaviour does not
C · JUST QUIETERmore “I don’t know”, but correct answers lost too

آزمایش طراحی شده، هنوز اجرا نشده. نتیجهٔ واقعی در ویدیوی جدا.

HOMODYNE BENCHMARK LAB / E05 · HMD-010
12 / 15 · 11:20

There is no universal k.

MOVIE TRIVIAwrong = a laugh → small k
STORE RETURN POLICYwrong = a real problem → larger k
AA-OMNISCIENCEindex −100…100 · 0 = as many right as wrong (≈ k = 1)
“I DON’T KNOW”is not always free (hand-off to support)
HOMODYNE BENCHMARK LAB / E05 · HMD-010
13 / 15 · 12:18 · CLAIM BOUNDARY
THIS SHOWSWith +1 / 0 / −k the silence line is k/(k+1). The common Konkur rule makes blind guessing worth zero. In simulation, the rule only helps an honest confidence.
THIS DOES NOT SHOWWhether today’s models are calibrated. Our answerers were not language models. Rescoring old answers never replaces a fresh run.
HOMODYNE BENCHMARK LAB / E05 · HMD-010
14 / 15 · 12:56 · ANSWER

On paper: yes. In practice: only if the model listens and its confidence is honest.

Try it: “If you are less than 90% sure, say I don’t know.” Then check one answer yourself.

HOMODYNE BENCHMARK LAB / E05 · HMD-010
15 / 15 · 13:49 · NEXT

Not “rain” or “I don’t know” — “70% chance of rain”.

Why do weather forecasts change — and where does AI help?