feat: enhance LM Studio integration with model filtering and add comprehensive test coverage
This commit is contained in:
@@ -1,7 +1,24 @@
|
||||
{
|
||||
"_comment": "Deliberately empty. Nothing in the language suite is reliably auto-checkable, and an unreliable key is worse than none - it fails correct answers with total confidence, which is the exact problem this grader exists to remove.",
|
||||
"_comment": "Only entity extraction is keyed. The rest of this suite is judgement - sentiment, similarity, summarisation, translation - and an unreliable key is worse than none, because it fails correct work with total confidence.",
|
||||
|
||||
"test2.txt": {
|
||||
"note": "Named entity recognition. Every entity below appears verbatim in the supplied excerpt, so any correct extraction contains them. This is a coverage check: it can pass an answer that lists the entities but mislabels their types, and that is the intended trade - it can never fail an answer that got them right.",
|
||||
"must_contain": [
|
||||
"Nairobi",
|
||||
"Kenya Wildlife Service",
|
||||
"Serena Hotel",
|
||||
"Tsavo East",
|
||||
"Okello",
|
||||
"Botswana",
|
||||
"World Bank",
|
||||
"Leakey Foundation"
|
||||
],
|
||||
"must_contain_any": ["KWS"],
|
||||
"numbers": [2023, 2019, 2017, 42],
|
||||
"_reasoning": "must_contain_any on KWS covers the alias question - the answer must notice KWS and Kenya Wildlife Service are the same entity. The numbers are the three years and the $42 million figure. Type classification is deliberately unchecked: a table can be right about the entity and wrong about the label in ways no substring test can separate."
|
||||
},
|
||||
|
||||
"_test4_omitted": "The Winograd item looked checkable: the first `it` is the TROPHY (a trophy too large fails to fit; a suitcase too large would succeed), flipping to the SUITCASE on 'too small'. But correctness turns on which noun binds to which condition, in free prose, and both nouns necessarily appear in any answer. Every pattern tried also matched an inverted answer, because 'it would refer to the trophy instead' sits inside the counterfactual sentence. Left to human review.",
|
||||
|
||||
"_others_omitted": "Sentiment, entity extraction, semantic similarity, summarisation and translation are judgement calls. Pattern-matching them would manufacture false confidence."
|
||||
"_others_omitted": "Sentiment, semantic similarity, summarisation and translation are judgement calls. Pattern-matching them would manufacture false confidence."
|
||||
}
|
||||
|
||||
@@ -1,6 +1,22 @@
|
||||
{
|
||||
"_comment": "The grid and the CPM network were solved by brute force, not by re-reading the derivation. Both confirmed unique.",
|
||||
|
||||
"test1.txt": {
|
||||
"note": "Commonsense mechanisms. Only the four items with an unambiguous physical answer are checked: water expands on freezing so that jar cracks (honey does not); the 3 a.m. chirp is a low battery, whose voltage sags as the house cools overnight; resting a roast lets juices redistribute; metal feels hotter than wood at equal temperature because it conducts heat away from the skin faster.",
|
||||
"must_contain_any": ["expand", "expansion", "less dense", "more space", "greater volume"],
|
||||
"regex": "(?i)(batter(y|ies))",
|
||||
"_reasoning": "Items 2 and 3 - the coffee mug and the indirect refusal - are deliberately unchecked. Both are correct in too many phrasings to pattern-match without risking a false failure. The remaining two mechanisms are covered by test1_extra below, kept separate so one missing term does not mask another."
|
||||
},
|
||||
|
||||
"_test1_extra_note": "Held as documentation rather than a matcher: 'juice'/'redistribute' for the roast and 'conduct' for the spoons were considered, but folding four independent facts into one score made a partial answer indistinguishable from a wrong one. Split them only if per-item scoring is ever added.",
|
||||
|
||||
"test2.txt": {
|
||||
"note": "Causal structures. The discriminating content is the lurking variable behind finding A - ice cream and drowning are both driven by hot weather - because that word appears nowhere in the prompt. The structure names do appear in the prompt as a list of options, so containing them proves little; they are checked anyway because a correct answer certainly contains them and the check cannot fail correct work.",
|
||||
"must_contain_any": ["temperature", "summer", "hot weather", "warm weather", "heat"],
|
||||
"regex": "(?i)regression to the mean",
|
||||
"_reasoning": "Finding F is regression to the mean and finding E is a selection effect, but both terms are supplied by the prompt, so a model could echo them while assigning them to the wrong findings. The weather confounder is the one answer the prompt does not hand over."
|
||||
},
|
||||
|
||||
"test3.txt": {
|
||||
"note": "Unique solution: Alvarez/South/comets, Brandt/North/dialects, Chen/East/basalt, Dube/West/algae. Clue 5 is redundant - removing it still leaves exactly one solution; removing any other clue does not.",
|
||||
"must_contain": ["Alvarez", "Brandt", "Chen", "Dube"],
|
||||
|
||||
Reference in New Issue
Block a user