STEP 1647 — LLM-as-judge comparator v0.1

STEP 1647 2026-09-01 test 50/50 PASS STEP 1631 regression 48/48 clean stub adapter で fan noise ゼロ
rei-aios · invariance-checker (STEP 1631) の pluggable async comparator layer

1. 契機

STEP 1637 v0.1 empirical で判明した pain point: numericMatch は「最後の整数抽出 + tolerance 比較」の表層 comparator。モデルが答えを冒頭に書いたり途中で分岐 output を出すと抽出誤り = FALSE 側の artifact に偏る。 chat-Claude 2026-08-31 「self-referential ゆえ discipline 必要 + 陰性対照必須」 warning に応答して primitive 側で 2 discipline を設計に埋め込む。

2. 骨格

AsyncAnswerEquivalence   (Promise-returning cousin of AnswerEquivalence)
   ↑
llmJudgeComparator(adapter, opts)
   ↑
JudgeAdapter    (pluggable HTTP client)
   ├── stubJudge*             deterministic, 6 factory (test/fan-zero)
   ├── ollamaJudgeAdapter     local, no cloud budget
   └── anthropicJudgeAdapter  cloud, requires ANTHROPIC_API_KEY

multiVoteJudge(base, votes, minAgreement) → self-consistency wrapper
singleInvarianceVerdictAsync / statisticalInvarianceVerdictAsync
   = async cousins of sync primitives (I1-I5 decision table identical)

3. 決定表 (judge → EquivalenceOutcome)

DECISION	mapping (default)	rationale
EQUIVALENT	`{ equivalent: true, confidence: 1 }`	judge 確信、 invariance 保存
DIFFERENT	`{ equivalent: false, confidence: 1 }`	judge 確信、 invariance 破壊
ABSTAIN	`{ equivalent: null, confidence: 0 }`	NEITHER 側で吸収 (default)
PARSE_ERROR	`{ equivalent: null, confidence: 0 }`	judge の出力 format 崩れ = NEITHER

abstainAsNeither: false オプションで ABSTAIN → false 転換 (safer 側)。

4. Judge prompt (canonical、 module-level 固定)

You are judging whether two answers to the same math problem are numerically equivalent.

--- Answer A ---
<text A>

--- Answer B ---
<text B>

Rules:
- Focus ONLY on the final numeric answer, not the working steps.
- Different phrasings of the same number are equivalent (e.g., "200", "two hundred", "\boxed{200}", "answer: 200").
- If either answer is missing / incomplete / not-a-number, output ABSTAIN.
- Do not solve the problem. Only compare the final answers as given.

Respond in this EXACT format, one field per line, no extra text:
DECISION: EQUIVALENT | DIFFERENT | ABSTAIN
REASON: <one short sentence>
NUMERIC_A: <extracted number or NULL>
NUMERIC_B: <extracted number or NULL>

JUDGE_PROMPT_ID = 'invariance-checker/judge/v1' — audit trail invariant。 caller custom prompt は別 promptId 宣言責任 (v0.2 defer)。

5. 順序原則 (定数化禁止)

adapter.model = caller 注入 (qwen2.5:7b / claude-haiku-4-5 etc.)
temperature = caller (default 0 = deterministic)
votes / minAgreement = caller (pre-registration で事前宣言)
abstainAsNeither = caller (safety vs discipline 選択)
Judge prompt 自身は module-level 固定 (audit trail invariant)

6. 陰性対照 (caller 責任)

judge 妥当性は単独 test では立証できない (self-referential trap)。 caller が empirical loop で以下を実行:

恒等 pair (答え同一) → 全 EQUIVALENT 期待
明確 mismatch pair (無関係な別問題答え) → 全 DIFFERENT 期待
abstain pair (片方 empty / non-numeric) → 全 ABSTAIN 期待

3 種の期待通り出れば judge は少なくとも明白 3 case で動作。微妙中間 case は STEP 1645 destructive transforms + STEP 1637 corpus で 4 象限 confusion matrix 経由で測定。

7. Test

npm run test:step1647 — 50/50 PASS (9 section):

JUDGE_PROMPT_ID 安定 + prompt shape (5 assert)
parseJudgeResponse 全 4 decision + numeric 抽出 (7)
Stub adapters 6 factory (7)
llmJudgeComparator (8: absence / EQ / DIFF / ABSTAIN toggle / err / PARSE_ERROR)
multiVoteJudge 過半数 + 満場一致 discipline (6)
promoteSync (2)
singleInvarianceVerdictAsync I1-I5 mirror (4)
statisticalInvarianceVerdictAsync (5)
Negative-control self-test with numeric-simulated stub (3)

STEP 1631 primitive regression: 48/48 clean (sync path 影響なし)。

8. 反証条件 (Popper self-application)

Stub always-equivalent adapter → equivalent=false を返す
Stub always-different adapter → equivalent=true を返す
multiVoteJudge(votes=3, minAgreement=3) に 2-1 split → null 以外
abstainAsNeither: false で ABSTAIN → null (should → false)
promoteSync(exactMatch()) が sync exactMatch と異なる outcome

全 5 条件、 test Section 3-6 で発火せず。現時点で反証なし。

9. Honest scope

v0.1 は infrastructure。実 empirical (実 judge model で陰性対照 3-pattern を pass するか) は別 STEP
Judge model の自動選定なし — caller が adapter 明示指定 (self-reference 予防のため)
Confidence calibration の数式化なし — judge raw output の confidence は使わない (raw 出力保存のみ)
意味論的等価の数学的定義なし — numerical-answer 等価に focus、 wider semantic invariance は別 STEP
陰性対照の自動混入なし — STEP 1645 destructive transforms 経由で caller 責任
Judge の bias 監視 primitive なし — 「always-equivalent pathology」の早期検知は v0.2 candidate

10. Trade-off

Cost: 1 pair あたり 1 API call (multiVote なら N 倍)
Latency: numericMatch の 1μs → judge の 100-500ms (cloud) / 500-2000ms (local)
Judge bias: 別 primitive で監視必要 (v0.2 candidate)
Advantage: numericMatch brittleness 解消 + 意味等価 (「200」と「\boxed{200}」) + audit trail (JudgeResponse に adapter id + raw text + numeric 記録)

11. v0.2+ candidate

Judge 応答の分布 caching (同 pair 判定不変仮定、 caller 明示 opt-in)
Concurrency wrapper (adapter 並列 OK なら N 並列 pool)
Confidence calibration (judge 自己申告 confidence → p-value 空間 mapping)
多 judge cross-verify (judge_A × judge_B pair、一致率で 3-tier verdict)
LLM-as-judge bias 監視 (「always-equivalent」 pathology 早期検知)