41 lines
3.5 KiB
JSON
41 lines
3.5 KiB
JSON
{
|
|
"_comment": "The grid and the CPM network were solved by brute force, not by re-reading the derivation. Both confirmed unique.",
|
|
|
|
"test1.txt": {
|
|
"note": "Commonsense mechanisms. Only the four items with an unambiguous physical answer are checked: water expands on freezing so that jar cracks (honey does not); the 3 a.m. chirp is a low battery, whose voltage sags as the house cools overnight; resting a roast lets juices redistribute; metal feels hotter than wood at equal temperature because it conducts heat away from the skin faster.",
|
|
"must_contain_any": ["expand", "expansion", "less dense", "more space", "greater volume"],
|
|
"regex": "(?i)(batter(y|ies))",
|
|
"_reasoning": "Items 2 and 3 - the coffee mug and the indirect refusal - are deliberately unchecked. Both are correct in too many phrasings to pattern-match without risking a false failure. The remaining two mechanisms are covered by test1_extra below, kept separate so one missing term does not mask another."
|
|
},
|
|
|
|
"_test1_extra_note": "Held as documentation rather than a matcher: 'juice'/'redistribute' for the roast and 'conduct' for the spoons were considered, but folding four independent facts into one score made a partial answer indistinguishable from a wrong one. Split them only if per-item scoring is ever added.",
|
|
|
|
"test2.txt": {
|
|
"note": "Causal structures. The discriminating content is the lurking variable behind finding A - ice cream and drowning are both driven by hot weather - because that word appears nowhere in the prompt. The structure names do appear in the prompt as a list of options, so containing them proves little; they are checked anyway because a correct answer certainly contains them and the check cannot fail correct work.",
|
|
"must_contain_any": ["temperature", "summer", "hot weather", "warm weather", "heat"],
|
|
"regex": "(?i)regression to the mean",
|
|
"_reasoning": "Finding F is regression to the mean and finding E is a selection effect, but both terms are supplied by the prompt, so a model could echo them while assigning them to the wrong findings. The weather confounder is the one answer the prompt does not hand over."
|
|
},
|
|
|
|
"test3.txt": {
|
|
"note": "Unique solution: Alvarez/South/comets, Brandt/North/dialects, Chen/East/basalt, Dube/West/algae. Clue 5 is redundant - removing it still leaves exactly one solution; removing any other clue does not.",
|
|
"must_contain": ["Alvarez", "Brandt", "Chen", "Dube"],
|
|
"must_contain_any": ["South", "north"],
|
|
"regex": "(?i)clue\\s*5|fifth clue",
|
|
"_reasoning": "Part (d) asks which clue is redundant. Verified by re-solving with each clue dropped in turn."
|
|
},
|
|
|
|
"test5.txt": {
|
|
"note": "Critical path A-D-E-F-H, project finish 32 days. Slack: B=12, G=11 (and C=5). D slipping 3 days pushes the finish to 35; G slipping 3 days changes nothing.",
|
|
"numbers": [32, 12, 11],
|
|
"must_contain_any": ["A → D → E → F → H", "A->D->E->F->H", "A-D-E-F-H", "A, D, E, F, H"],
|
|
"_reasoning": "Path notation varies, so several renderings are accepted."
|
|
},
|
|
|
|
"test6.txt": {
|
|
"note": "Cognitive-reflection answers: ball is 5 cents (not 10), 100 machines take 5 minutes (not 100), lake is half covered on day 47 (not 24).",
|
|
"numbers": [5, 47],
|
|
"_reasoning": "The intuitive-but-wrong answers are 10c, 100 minutes and 24 days. Only the correct values are checked: Stage 2 explicitly asks the model to state the wrong answer it was meant to elicit, so banning '24 days' would fail a correct answer. The 5-minute answer collides numerically with the 5-cent one and is not checked separately."
|
|
}
|
|
}
|