Can small language models read analytical English? A zero-shot baseline
We gave four small open models 120 questions about aggregation, thresholds, consecutive periods and negation. The best scored 46.7%; a rule that picks the longest answer scores 41.7%.
People ask data questions in plain English: “Which agents had a streak of 6 months without any reopened tickets?” Turning that into an exact query means getting small details right: > or >=, 5 or 0.05, six months in a row or six months in total. Before building anything for this, we wanted to know where small, off-the-shelf language models start. So we measured it, with no fine-tuning, in a controlled setup.
This is a baseline to beat, not a product claim. The short version: none of the four models is clearly good at this yet.
The test
Each item is a question over a small table schema, with four candidate formal interpretations. Exactly one is correct, and the score is how often the model picks it. There are 120 items, 20 in each of six categories:
| Category | What it tests |
|---|---|
| Aggregation | sum, average, maximum or count; grouping; distinct counts |
| Percentage threshold | percent to fraction, > against >=, share of total, growth on the previous period |
| Consecutive periods | a run of periods against a count of periods |
| Historical comparison | same period last year, rolling windows, a new all-time high |
| Negation | not, never, no more than, not below |
| Ambiguous condition | questions with two valid readings, where the right answer is to ask, mixed with matched unambiguous controls |
The wrong answers are the typical mistakes: the wrong aggregate, a missing grouping, 5 instead of 0.05, a window that includes the current period. The items cover four table domains (sales, factory, support tickets and energy), and they are generated deterministically from a fixed seed, so every model sees exactly the same questions.
Each model ran alone in its own container with 8 CPUs, a 7.2 GB memory limit and swap switched off, in 32-bit precision. Model versions were pinned by commit and the runs were offline, so download time never entered the measurements. Two ways of scoring were used, both zero-shot with the same wording for every model: ranking the four candidates by likelihood, and showing them as options A to D.
We also scored four rules that understand nothing: always pick the first option, the longest, the shortest, or the one that shares the most words with the question. A model has to beat these to show it has learned something.
Results
| Model | Parameters | Accuracy (likelihood) | Peak RAM | Median time per question |
|---|---|---|---|---|
| Flan-T5 small | 77M | 29.2% | 0.7 GB | 175 ms |
| Flan-T5 base | 248M | 25.0% | 1.4 GB | 627 ms |
| Qwen2.5 0.5B Instruct | 494M | 43.3% | 3.2 GB | 1.7 s |
| TinyLlama 1.1B Chat | 1.1B | 46.7% | 6.3 GB | 5.0 s |
| Random guessing | 25.0% | |||
| Always the longest candidate | 41.7% |
On 120 items, the 95% interval around each score is roughly plus or minus 9 points.
What we found
Nothing is clearly good yet. The best score was 46.7%. The rule that picks the longest answer, with no language understanding, scored 41.7% on the same items. Qwen2.5 and TinyLlama are above random but not distinguishable from each other, or from that rule. Both Flan-T5 models are at chance.
Scale did not give a clean gain. Accuracy does not rise steadily with size. The models differ in architecture and training, not only in size, so this comparison does not isolate the effect of scale.
The mistakes are systematic. Qwen2.5 reached 80% on aggregation but 0% on consecutive periods, which is below chance. It prefers a wrong reading, counting periods that are not consecutive. On percentage thresholds the longest-answer rule (70%) beat every model.
Ambiguity is not handled at all. Half of the ambiguous-category items are genuinely ambiguous, and the right answer is to ask for clarification. Both decoder models never chose to ask when ranking by likelihood: they got all 10 controls right and all 10 ambiguous items wrong. That is exactly the score of a model that always answers.
Letter-choice prompting is unreliable at this size. When the candidates were shown as options A to D, predictions piled onto one or two letters: about 80% A for Flan-T5, A or B every time for TinyLlama, and 92% C or D for Qwen. The correct answers were spread 28%, 28%, 22% and 22%. This method mostly measures the bias, so we treat likelihood ranking as the informative one.
Cost grows faster than size. From 77M to 1.1B parameters, a 14× increase, the median time per question rose about 28×, and peak memory went from 0.7 GB to 6.3 GB.
How far to trust it
- With 20 items per category, one item is 5 points, so category differences under about 15 points are noise.
- The candidate interpretations are written in a query-like notation that none of these models was trained on. That is part of what the baseline measures.
- T5’s vocabulary has no
<character. We spell itltfor those models by default and also ran them without the fix; the difference was within noise. - Running the smallest model a second time gave 240 of 240 identical predictions and scores. Latency moved by up to 17%, so treat the ratios between models as the portable part, not the absolute milliseconds.
- These are small models, chosen to fit on ordinary hardware. They say nothing about large hosted models.
What it means for us
The zero-shot baseline to beat is about 45%. That justifies a more structured approach than asking a small model to pick a reading, such as fine-tuning or constrained output. The next evaluation needs the ambiguity category, where asking is the right answer, and a version of the items with the candidates equalised in length, since the longest-answer shortcut is competitive.
If you are working on turning business questions into exact queries and want to compare notes, get in touch.