Any Language Model Is a Classifier: Calibrated One-Pass Decisions over Arbitrary Label Sets from Pretrained LLMs

Abstract

Many language-model calls select from a known set of options but use generation followed by parsing. We instead compute option likelihoods and normalize over the set. Packing the options after a shared prompt with a block attention mask computes all likelihoods in one forward pass (a pack-readout). Without training, this is the log-likelihood protocol of standard evaluation harnesses, with its known limitations: it favours short options, splits probability across paraphrases and is poorly calibrated. We train the option distribution by cross-entropy over the option set on a mixture of 18 tasks and evaluate on 12 held-out tasks. The mean primary score on the held-out tasks (macro-F1, or accuracy where options differ per example) rises from 0.37 to 0.57 at 0.6B parameters, from 0.55 to 0.64 at 1.7B and from 0.63 to 0.72 at 8B. A single set of weights handles every label set tested. Against the 35 zero-shot classifiers of the BTZSC leaderboard, it lies on or above the size–accuracy frontier on the three shared tasks; under the benchmark’s own protocol, the 4B model exceeds the best listed system at half its size. Prompting the same-size instruction-tuned model is less accurate at every size; eleven prompted hosted models, including GPT-6 Astra and Claude Fable 5.1, score 0.71 to 0.81 against 0.72 for our trained 8B. A learned scalar head transfers better than the language-model head; at 1.7B, mean top-1 calibration error decreases from 0.114 to 0.098 without temperature scaling. A 77-way decision from a 1.7B model takes 54 ms on an A100, 3× faster than constrained decoding and 19× faster than JSON generation with a rationale; the conversion requires 16,000 training examples and 20 minutes of LoRA training and applies unchanged to SmolLM2, OLMo-2, Qwen2.5 and gpt-oss-20b. Model implementation, training code and trained adapters are released.

1 Introduction

In many applications, a language model selects one option from a fixed set: a routing label, a severity tier, yes or no. Generating text and parsing it has drawbacks. Decoding is sequential and memory-bound, so an n-token label requires n sequential forward passes; the output must be parsed and may fall outside the schema; and a sampled label carries no usable probability.

An alternative is maximum-likelihood selection among admissible options [5, 7]. Known options can be scored by teacher forcing without decoding. Restricting outputs to these options provides valid decisions with associated probabilities. Commercial “decision models” such as Jev [67] expose this interface (a typed question, a set of options, a probability over them, one pass), but do not disclose how they are trained. We investigate a simple, fully specified adaptation of a pretrained language model toward this interface: a classifier that accepts a new label set at inference time and returns probabilities over it in one forward pass.

Zero-shot option scoring has at least three documented limitations. Summed log-probabilities penalize long options and predictions change with the scoring method [66]; probability leaks to paraphrases outside the option set [24]; and the renormalized distribution is poorly calibrated [27, 81, 83]. Per-task decoder methods train a classification head [38, 63]; Universal Discriminator [76] transfers encoder-based candidate scoring to held-out tasks. We combine cross-task discriminative training with pack-readout.

Our contributions are:

  1. Pack-readout. One forward pass over the prompt followed by every option, with a block attention mask and per-option position ids, gives the same log-likelihoods as scoring each option separately, without KV-cache replication (Section 3.2); the same pass implements three decision primitives: choice, ordinal score and yes/no (Section 3.3).
  2. Training the option distribution. We minimize cross-entropy over the option set on a public task mixture with at most 2,000 LoRA steps (16,000 example presentations, approximately 2.5M tokens); training takes 20 minutes on one A100 at 1.7B. On 12 held-out tasks, the mean primary score improves over zero-shot scoring by 9 points at 1.7B (three seeds), 4B and 8B, by 20 at 0.6B and by 2 at 14B with a reduced learning rate and step count (Section 5.2); the scalar head transfers to label sets and label names it never saw (Section 5.3).
  3. Comparison with dedicated zero-shot classifiers. Against the 35 systems of the BTZSC leaderboard, the trained models are above every listed system on MASSIVE from 0.6B on and on Banking77 from 4B on (Figure 3); under the benchmark’s own protocol, trained without its datasets, the 4B model is above every leaderboard system at half the size of the best one (Figure 8).
  4. Calibration. The objective is a proper scoring rule: Brier score, log-likelihood and confidence ranking improve on unseen tasks, and top-1 calibration improves for the scalar head without temperature scaling (Section 5.4).
  5. Cost comparison against prompting. A 77-way decision from a 1.7B model takes 54 ms on an A100, 3× faster than constrained decoding and 19× faster than JSON generation with a rationale, with little variation across inputs (Section 5.6). Prompting the instruction-tuned model of the same size is less accurate at every size, and up to 7% of its replies name no option; eleven hosted frontier models score 0.71 to 0.81 against 0.72 for our trained 8B, at 4 to 177 decode steps per decision (Section 5.5).
  6. A general recipe. The same procedure raises SmolLM2-1.7B and OLMo-2-1B by 25 and 21 accuracy points, gpt-oss-20b by 15 and Qwen2.5-32B by 3; more training label sets help the many-label tasks and calibration most, and removing two training sets that neighbour held-out tasks raises the neighbours’ scores (Section 5.7).

Option scoring. Scoring candidate continuations by log-likelihood is the multiple-choice protocol of GPT-3 [7] and lm-evaluation-harness [5], which reports raw (acc) and per-character normalized (acc_norm) variants. Holtzman et al. [24] identified surface-form competition and proposed domain-conditional PMI; Min et al. [46] score the prompt given the option; Tsvilodub et al. [66] show that predictions change with the scoring method. Scoring the letter of an option [57] is cheaper but biased toward particular option letters [82]; choices extracted from generated answers are more robust than first-token probabilities when the two disagree [70]. We score the full option text.

Discriminative fine-tuning of decoders. LS-LLaMA [38] and Sood and Savelka [63] attach a classification head to a decoder and train per task; the latter finds that cross-entropy over candidate scores outperforms label-generation SFT by 2–3 points at 0.6–1.7B. Neither trains across tasks, evaluates on unseen label sets, or measures calibration or latency. Kowsher et al. [31] argue that generation preserves more information than a pooled representation; our LM-head logit is a token-level likelihood, the scalar head a pooled representation. We compare both with generation.

Zero-shot classification over arbitrary label sets. T0 [59] and FLAN [10, 72] train on many prompted tasks and transfer to held-out ones; both train to generate the target text, and T0 and the first FLAN evaluate classification by rank classification, the likelihood of each option (the later Flan collection evaluates by direct and chain-of-thought prompting). We adopt the held-out protocol and train the scoring readout directly. Universal Discriminator [76] trains an encoder-based scalar scorer across tasks with binary cross-entropy on input–option pairs and transfers to held-out tasks. We use a decoder, train with cross-entropy over the option set, and score all options in a shared-prefix pass. GLiClass [64] prepends the labels to the input of an encoder. BTZSC [1] benchmarks NLI cross-encoders, embeddings, rerankers and first-token LLM scoring on 22 datasets (Section 5.8). Laya [13], released concurrently, is an open encoder with Jev’s three primitives: ModernBERT-large scores each option at a mask token and is trained per domain with REINFORCE on proper scoring rules. Its model card reports 0.36 zero-shot on its own decision set and 0.43 on Banking77, which indicates limited transfer to unseen label sets.

Shared-prefix attention. Hydragen [28] computes exact attention for sequences that share a prefix. Wang and Hegde [69] pack chosen and rejected responses behind a shared prompt with a block-sparse mask for direct preference optimization. We apply this construction to an option set, using likelihood or scalar readouts. Block-Attention [41] and Parallel Context Windows [56] mask independent blocks that a final block reads, and require fine-tuning or lose some accuracy. Pack-readout uses the same prefix-plus-block pattern for option scoring and needs no adaptation: each option block attends to only the prompt and itself.

Calibration. Temperature scaling [21], contextual calibration [81] and batch calibration [83] correct a fixed model’s outputs; ConfTuner [36] trains verbalized confidence with a Brier loss. We train the option distribution itself with a proper scoring rule [20] and test whether calibration transfers. G-Eval [39] takes the probability-weighted sum of rating tokens; our ordinal primitive is the same estimator. Table 21 (Appendix C) compares these methods.

Concurrent work. Cheng et al. [9] concurrently introduce a 2B typed decision model that reads each option at a single token and is trained with full-parameter cross-entropy and Brier losses. We score the full option text, compare two readouts, and measure transfer to held-out tasks across model families with LoRA; Section 5.3 trains their single-token readout under our protocol.

3 Method

Figure 1 summarizes the model: a pretrained decoder run once over the prompt and all options, with a block attention mask and per-option position ids, read out at the option tokens and normalized over the option set.

3.1 Option scoring

Given a prompt x (state, question and the listed options) and options o1, …, on, each rendered as a leading space, the option text and a newline terminator, the score of option i is its teacher-forced log-likelihood and the decision distribution its softmax:

si=∑t=1Lilog⁡Pθ(oi,t∣x,oi,<t),pi=exp⁡(si/τ)∑jexp⁡(sj/τ)(1)

The terminator is scored so that an option whose tokens are a prefix of another’s (“New York”, “New York City”) can score lower. A token prefix shared by several options contributes equally to their log-probabilities, leaving their relative odds unchanged while counting against options that do not share it. We also consider per-token, per-byte and per-character normalization of si, and the PMI variant si − log Pθ(oi ∣ x0) for the content-free prompt x0 that keeps only the cue line, the domain-conditional PMI of Holtzman et al. [24].

3.2 Pack-readout: one forward pass for all options

Eq. 1 requires no decoding. A direct implementation prefills x once, replicates its KV cache n times and scores the padded option batch; the replication costs O(nP) memory traffic and dominates for many options (Table 27). Instead we run one sequence x ‖ o1 ‖ ⋯ ‖ on, where ‖ denotes token concatenation, with (i) a block attention mask in which a token attends to the prompt and earlier tokens of its own option only, and (ii) position ids that restart at P = |x| for every option (Figure 1). The hidden states of option tokens then equal those of a separate pass over x ‖ oi, so the log-likelihoods are exact (verified to 2×10−3 nats in fp32). Each option token’s log-probability is read at the position that predicts it: the prompt’s last position for the first token of every option, then the option’s own positions, including the one that predicts its terminator; the LM head is applied only at these positions. We call this pass a pack-readout: the options are packed, then read out. The cost is one prefill of P + Σi Li tokens: an option adds Li tokens to a compute-bound prefill, whereas each decoded token costs a memory-bound step.

Prompt x (P tokens)

Route the message to one intent.
Labels: refund, refund status, cancel
Text: Where is my refund?
Label:

Packed sequence and position ids

• the LM head reads the log-probability of each option token at the position that predicts it

xxo₁o₁o₂o₂o₃o₃

“status” (option 2, position P+1) attends to the prompt and to “refund”: the same as in a separate pass over the prompt and option 2.

Softmax over the options

Figure 1. The model, on an intent example whose options include a prefix pair (“refund”, “refund status”). The example is formatted as a prompt that lists the options, and each option comprises a leading space, its text and a scored newline terminator (↵), allowing a prefix option to score below its extension. Each option continues the prompt from position P, as if it were the only continuation; the n continuations are packed into one sequence for the forward pass. A pretrained decoder runs once over that sequence. A block attention mask lets option tokens attend to the prompt and to their own option only, so each option’s hidden states equal those of a separate pass over prompt and option. Each option receives one score from its own positions: the LM head sums the log-probabilities of its tokens (read at the positions that predict them, •), the scalar head applies a learned vector to its last hidden state (■). A softmax over the n scores gives the decision distribution; training minimizes its cross-entropy against the gold option.

3.3 Decision primitives

Three primitives cover the target decisions, each defined by Eq. 1. Choice lists the options in the prompt and returns p. Score lists K ordered levels with their indices, scores the index strings “0”, …, “K−1”, and returns the expected level Σk k pk, as in G-Eval [39]; single-token indices (K ≤ 10) remove length bias from the ordinal case. True/false scores “yes” and “no” and returns pyes.

3.4 Training the option distribution

We fine-tune with LoRA [26] on a mixture of classification and multiple-choice tasks, minimizing the negative log-probability of the gold option y under Eq. 1 with τ = 1:

Lset=−log⁡exp⁡(sy)∑jexp⁡(sj)(2)

This trains the inference distribution with a strictly proper scoring rule, whose unrestricted population-risk minimizer is the true conditional distribution [20], and allows the model to correct length bias. We compare two readouts on the same packed computation: the LM head, si as in Eq. 1, and the scalar head of Sood and Savelka [63], si = w⊤hi, where hi is the final hidden state after the last norm at option i’s newline terminator. This is the vector the LM head reads, computed with attention to the prompt and the whole option. Both readouts can score unseen option strings; p is the softmax over the n scores. We also compare the gold-only objective −sy, which raises the gold option’s likelihood without comparing it to the alternatives and is the label-generation objective of T0 and FLAN with the option text as target.

Is reinforcement learning needed for calibration? Jev is reported to be trained by “reinforcement learning for calibrated decisions” (RLCD), which its developers describe as optimizing for calibrated probabilities on decision tasks [68]; the reward and the data are not public. A reward that targets calibration is a proper scoring rule of p against a verified label. When the model emits the whole distribution p in one pass, such a reward, for instance the Brier score Σi (pi − 𝟙[i = y])2, is deterministic and differentiable: there is no action to sample, gradient ascent on the expected reward is gradient descent on the loss, and a sampling estimator can only add variance. We therefore compare three objectives: the log score (Eq. 2), the Brier score as a loss, and REINFORCE [75]: sample a decision from p, then update the log-probability of that option in proportion to the reward minus a running mean of past rewards. With reward 1 when the sample is y, the expected reward is py rather than log py, an improper scoring rule maximized by putting all mass on the likeliest option; it optimizes accuracy alone.

4 Experimental setup

Models and compute. We train Qwen3-Base at 0.6B, 1.7B, 4B, 8B and 14B parameters [55] in bf16. Most runs use one NVIDIA A100-40GB; the main 8B and 14B runs use a DGX Spark (GB10, 128 GB), and the 14B sweep, the BTZSC ladder and the label-set sweep use a Lambda 8×A100-40GB with one run per GPU, except gpt-oss-20b and Qwen2.5-32B, each sharded over two GPUs. Training uses LoRA of rank 16 (α = 32) on all attention and MLP projections, AdamW [40] with learning rate 2×10−4, 100 warm-up steps and cosine decay, and batches of 8 examples for 2,000 steps (16,000 examples of the 36,000-example mixture, at most 2,000 per training task); the scalar head is one fp32 vector trained with the adapter, and the main runs use seeds 0–2 (Appendix G describes the 4B seeds). A 1.7B run takes about 20 minutes of training and 3 minutes of evaluation on the subsets (12 on the full test sets). Training and evaluation jobs total 77 GPU hours in our records (13 on the Colab A100, 12 on the DGX Spark and 52 on the Lambda A100s), mostly for the 14B and larger models, the two sweeps of Section 5.7, and the prompting and latency benchmarks; the records omit the full-test evaluations, whose logs were not kept.

Tasks. Table 1 lists the mixture and the held-out set. Held-out tasks share no dataset and no label set with the mixture, but two have close neighbours in it: three of the four TweetEval emotion labels (anger, joy, sadness) also appear in the six-label Emotion training set, which lacks “optimism”, and TweetEval hate is closely related to TweetEval offensive, a broader category. Section 5 reports the effects; Section 5.7 evaluates training without these datasets. Options are always listed in the prompt. Classification prompts read “instruction / Labels: … / Text: … / Label:”; question prompts read “Question: … / Options: … / Answer:” (Appendix A).

RoleTasks|𝒪|Kind
trainAG News, DBpedia-14, Yahoo Answers4 / 14 / 10topic
trainYelp-5, IMDB, TweetEval sentiment5 / 2 / 3sentiment, ordinal
trainEmotion (dair-ai), TweetEval offensive, irony6 / 2 / 2emotion, binary
trainCLINC-150151intent
trainSciQ, OpenBookQA, CommonsenseQA, QASC, ARC-Easy, HellaSwag, WinoGrandeper ex.multiple choice
trainBoolQ2yes/no with passage
held outBanking77, MASSIVE77 / 59intent
held outSST-5, Rotten Tomatoes5 / 2sentiment, ordinal
held outTweetEval emotion, hate; GoEmotions4 / 2 / 28emotion, binary
held outSMS Spam, Amazon Counterfactual2 / 2binary
held outARC-Challenge, MMLUper ex.multiple choice
held outStrategyQA2yes/no
Table 1. Training mixture and held-out tasks. |𝒪|: number of options; “per ex.”: options differ per example. Sources: Appendix I.

Baselines. At the same base model: zero-shot option scoring with raw, per-byte and PMI scores and with batch calibration; five-shot option scoring, five training examples of the task in the prompt and no training; first-token symbol scoring, the LLM protocol of BTZSC; gold-only SFT and the scalar head on the same mixture; the scalar head trained on each held-out task’s own training split, the supervised ceiling of the readout (Table 3). External baselines are the family leaders and small LLMs reported by BTZSC, and the best published zero-shot and supervised results per held-out task (Appendix E).

Metrics. Macro-F1 for tasks with a shared label set and accuracy for the two whose options differ per example (the primary metric); negative log-likelihood, Brier score [6] and expected calibration error [52] (ECE, 15 equal-width bins) of the top-1 confidence; AUROC of the confidence as an error detector; accuracy at 50% coverage. Macro-F1 averages per-class F1 over the classes present in the gold labels of the evaluated examples; a reply naming no option counts as an error for its gold class only. Latency is the wall time of one decision, measured once per input over 20 inputs after warm-up, on A100-40GB GPUs with the same eager PyTorch setup for every method and size (Appendix F names the hosts). Table 3 evaluates on the full test sets (40,117 examples over the twelve tasks); the ablations, sweeps and other sizes use 500 seeded test examples per task, and Tables 11 and 12 check these subsets against the full test sets at every size (means within a point, except the 0.6B zero-shot mean, 1.2 lower on the subset). Bootstrap 95% intervals on the subsets have a half-width of about 0.04 per task (median; 0.07 at most, on the 335-example Counterfactual set) and 0.012 on the held-out mean. Per-task differences of a few points between single runs are within noise, so we compare means. Held-out means weight tasks equally; standard deviations describe variation across training seeds.

5 Results

5.1 Zero-shot option scoring

Our scorer reproduces lm-evaluation-harness on ARC-Easy (first 400 test questions, Qwen3-0.6B-Base: acc 0.6275 for both; acc_norm 0.560 for both). Table 2 gives zero-shot held-out accuracy for Qwen3-1.7B-Base. Text scoring is a strong baseline (mean 0.626; 0.73 on ARC-Challenge, 0.87 on Rotten Tomatoes) and, on the nine tasks with at most 26 options, 8.9 points above first-token symbol scoring (0.688 vs. 0.599), which fails on SMS Spam (0.24). Four observations motivate training. No normalization wins everywhere: the raw sum is best on average, and every length normalization reduces accuracy on short labels (Banking77: 0.43 raw, 0.17 per character). PMI with a generic null prompt collapses on large label sets (0.01 on Banking77): the option with the rarest surface form is predicted for every example. Small models follow the label prior: the 0.6B model predicts “spam” for 499 of 500 messages and “hate” for 494 of 500 tweets; batch calibration [83] improves the tasks dominated by the label prior (+4.6 on Banking77, +5.2 on StrategyQA at 1.7B) and degrades the imbalanced ones (−20.6 on SMS Spam, −23.6 on Counterfactual). Calibration is poor and inconsistent: ECE spans 0.05–0.32 across tasks at 1.7B (0.07–0.73 at 0.6B), and the best-fitting temperature spans 0.4–2.4 at 1.7B (0.5–20 at 0.6B, where 20 is the ceiling of the search), so no single post-hoc temperature suffices.

TaskrawtokenbytecharPMI+BCsymbolECE (raw)
Banking770.4320.2280.1980.1740.0140.478–0.083
MASSIVE0.6560.5920.5920.5780.0560.666–0.047
SST-50.4280.2980.3040.2960.2440.3840.2920.164
TweetEval emotion0.7580.7580.6780.6200.6900.7620.7340.078
TweetEval hate0.6120.5880.5880.5880.5880.6220.6760.062
Rotten Tomatoes0.8720.8720.8720.8720.7300.8980.8540.062
SMS Spam0.7960.7960.6340.5060.8780.5900.2360.162
Counterfactual0.8240.8270.8270.8270.8270.5880.8270.107
GoEmotions0.2320.2300.1620.1520.1860.258–0.316
ARC-Challenge†0.7340.7220.7300.7240.512–0.7160.058
MMLU†0.5840.5660.5560.5580.430–0.5480.128
StrategyQA0.5820.5820.6080.6080.5100.6340.5120.102
Mean0.6260.5880.5620.5420.4720.5880.5990.114
Table 2. Zero-shot held-out accuracy, Qwen3-1.7B-Base: raw score, per-byte normalization, PMI, batch calibration (+BC), first-token symbol scoring, and ECE of the raw score. Per-token and per-character normalization behave like per-byte.

5.2 Transfer to unseen label sets

Training on the mixture raises the held-out mean on the full test sets (Table 3; Figure 2) from 0.552 to 0.621 with the LM head and to 0.641 ± 0.010 (three seeds; population s.d.) with the scalar head at 1.7B; five in-context examples of the task itself reach 0.571 (Table 13), so cross-task training yields about five times the gain of within-task prompting. The gain is largest where zero-shot scoring is weakest: 20 points at 0.6B (0.37 to 0.57), and at 1.7B Banking77 (77 unseen intents) from 0.40 to 0.66, Counterfactual from 0.45 to 0.63 and SST-5 from 0.36 to 0.45. The gain is 9 points at 4B and 8B and under one point at 14B (0.708 to 0.714). The 14B base model starts high: half of its 8-point zero-shot lead over the 8B base reflects a label prior aligned with three tasks (with batch calibration, which removes the prior, the zero-shot means are 0.58, 0.61 and 0.65 at 4B, 8B and 14B). The fixed 2,000-step run also produces regressions on short tweets that offset its gains on many-label tasks (Appendix B). Section 5.7 shows that a gentler run recovers the loss (0.731) and attributes the tweet regressions to two training sets that neighbour them. Accuracy and macro-F1 move differently on the imbalanced binary tasks: on Counterfactual the zero-shot model predicts the majority label for 334 of 335 examples (accuracy 0.82, macro-F1 0.45) and the trained model predicts both classes (0.84, 0.63). Training attenuates the prompt’s label prior in favour of the text.

Zero-shot, this modelTrained on other label sets
Rotten TomatoesARC-Challenge*TweetEval emotionSMS SpamMASSIVEMMLU*StrategyQATweetEval hateCounterfactualBanking77SST-5GoEmotions

Full test sets; trained columns are means of three seeds.

Figure 2. Before and after training, per held-out task, Qwen3-1.7B-Base, on the full test sets. Grey: zero-shot option scoring; orange: after training on the mixture (scalar head), which never saw these datasets. Open triangles: the best published zero-shot result on the same metric, from dedicated classifiers or earlier LLM reports of 1.7B to 20B parameters or of undisclosed size (Table 3 adds the supervised and prompted-frontier numbers, which the legend can show here). Tasks ordered by zero-shot score. The other sizes and gpt-oss-20b are on the 500-example subsets (Tables 16–19).

Against published numbers. The last columns of Table 3 report the best published zero-shot and supervised results on the same metric, with model sizes (Appendix E), and the best prompted frontier model from Section 5.5; Figure 2 marks the published zero-shot results. The frontier models exceed every published zero-shot number except on SST-5, defining the current zero-shot ceiling; our trained 8B trails the best model (Claude Fable 5.1, 0.806) by 8.5 points on the held-out mean, with 8B parameters against models of undisclosed or trillion-parameter size. The published zero-shot leaders are 8B embedding and reranker models, Flan-UL2 (20B) and ChatGPT or GPT-4, 5 to 12 times our model’s parameters where the size is known. On MASSIVE, our 1.7B model, which never saw its 59 intents, reaches 0.74 against 0.67 for the 8B reranker; on Banking77 it reaches 0.66 against 0.72 for the 8B embedding model, and models trained on the task’s own labels remain well ahead there (0.94 accuracy with BERT-large). Both are on the full test sets; the published results use each paper’s prompts and label wording, so one-point differences do not establish rankings.

Against the BTZSC leaderboard. Figure 3 places the model among the 35 zero-shot classifiers of the BTZSC leaderboard on the three held-out tasks the benchmark also evaluates, with, for each family, the best score at or below each size. One set of weights serves every label set, and none of our models was trained on the evaluation task. On MASSIVE the trained 1.7B model (0.74) is above every leaderboard system, whose best is an 8B reranker at 0.67, and the 0.6B model ties that reranker at 13× fewer parameters. On Rotten Tomatoes the 1.7B model is within two points of the best system and the 8B model above it. On Banking77, where embedding models that retrieve intents by similarity are the specialists, the 1.7B model lies just below their best score (0.66 against 0.72) and the 4B model above every system (0.73). Every LLM on the leaderboard, scored by the first-token protocol, is more than 20 points below ours on the intent tasks at up to 12B parameters. The four other base models (lettered discs) fall on or above the Qwen3 curve; gpt-oss-20b exceeds every system on Banking77 and MASSIVE, and Qwen2.5 32B every system on Rotten Tomatoes. The comparison is not exact: leaderboard numbers use the benchmark’s verbalized label descriptions, ours the label names; Table 30 (Appendix D) gives our numbers under the benchmark’s own protocol.

Ours, trained
0.6B1.7B4B8B14BOOSSGGQQ
Figure 3. Macro-F1 against parameters on the three held-out tasks that BTZSC also evaluates. Grey staircases: for each family of the BTZSC leaderboard [1], its Pareto frontier (the best score at or below each size), continued flat to the family’s largest system; small grey markers, one shape per family: the individual systems as published (btzsc/btzsc-results, fetched 2026-09-20; full test sets, verbalized label descriptions). Orange: our trained model at 0.6B to 14B (scalar head, label names, full test sets, Table 12; three seeds at 1.7B; outline only: the gentler 14B run of Section 5.7), which never saw these datasets; open circles: the same base models zero-shot; lettered discs: the four other base models of Section 5.7, one size each (O: OLMo-2 1B, S: SmolLM2 1.7B, G: gpt-oss-20b at 21B total parameters, Q: Qwen2.5 32B; 500-example subsets), open when zero-shot. A dotted segment joins each base model’s zero-shot point to its trained point.

Size and compute per decision. Table 22 (Appendix C) estimates the cost of one decision for each of these systems as 2 × parameters × tokens processed. The packed pass processes the prompt and every option once, 2.4 TFLOPs on Banking77 and 0.2 on SST-5 at 1.7B; a cross-encoder reranker runs one pass per label and costs 8× more on Banking77, and a 20B generative model 6× more before it decodes. A bi-encoder that caches the label embeddings costs less on Banking77, but it returns one similarity per label, each scored without seeing the others. Among systems that condition on the whole option set, ours is the smallest and costs no more per decision than any other.

How far from a model trained on the task? The “trained on the task” column of Table 3 runs the same recipe on each task’s own training split for 400 steps. On the many-label and emotion tasks the gap is large: 0.87 against 0.66 on Banking77, 0.84 against 0.74 on MASSIVE, 0.85 against 0.69 on TweetEval emotion, 0.89 against 0.63 on Counterfactual. On ARC-Challenge and Rotten Tomatoes the general model is within two points or above (0.83 against 0.84, 0.91 against 0.89), on SST-5 and StrategyQA it trails by 11 and 6 points, and on TweetEval hate the task-trained model is worse (0.42 against 0.56): 400 steps on the imbalanced binary training split overfit. On the question and review tasks the general model is close to the task-trained one; elsewhere task-specific training adds 6 to 26 points, most where the label set is large and specific.

Gold-only SFT reaches 0.605 with the same readout and data (Appendix B): comparing the gold option against the alternatives adds about two points of macro-F1, with larger distributional improvements (Section 5.4). First-token symbol scoring stays below text scoring on every task where it applies, except TweetEval hate, where the letter prompt avoids the “hate” prior.

TaskZero-shotLM headScalar headTrained on the taskBest prompted frontier (model)Best published zero-shot (size)Best supervised (size)
Banking770.3990.6500.6600.8740.868 (Claude Fable 5.1)0.715 (8B)–
MASSIVE0.6460.7160.7400.8360.890 (GPT-6 Astra)0.673 (8B)–
SST-50.3580.4290.4510.5640.544 (Claude Fable 5.1)––
TweetEval emotion0.6970.7010.6940.8490.802 (gpt-oss-120b)0.728 (n.d.)0.810 (0.12B)
TweetEval hate0.4840.5620.5600.4190.767 (Kimi K3)0.708 (20B)0.581 (0.12B)
Rotten Tomatoes0.8620.8830.8880.9110.952 (Claude Fable 5.1)0.903 (8B)–
SMS Spam0.6760.6730.703–0.986 (Claude Fable 5.1)––
Counterfactual0.4510.4980.6260.8900.847 (gpt-oss-120b)––
GoEmotions0.2060.2700.2950.4680.362 (gpt-oss-120b)––
ARC-Challenge†0.7450.8370.8380.8280.976 (Kimi K3)0.786 (3.2B)–
MMLU†0.5520.6050.592–0.938 (GPT-6 Astra)0.626 (1.7B)–
StrategyQA0.5480.6280.6460.7080.923 (Claude Fable 5.1)––
Mean0.5520.6210.641
Table 3. Held-out performance, Qwen3-1.7B-Base: macro-F1, or accuracy for the two tasks marked † whose options differ per example. The LM-head and scalar-head columns never saw these datasets (both readouts: mean of three seeds); “trained on the task” is the same recipe run on the task’s own training split (400 steps, scalar head), the supervised ceiling of this model. The last three columns are context: the best of the eleven prompted frontier models of Table 7 on the 500-example subsets (free replies), and the best published numbers on the same metric, on full test sets, from dedicated classifiers or earlier LLM reports not trained on the task (zero-shot) or trained on it (supervised), with the model’s parameter count (n.d.: not disclosed); sources and non-comparable cases in Appendix E. Our columns are on the full test sets, three seeds for both readouts (Table 11 gives the 500-example subsets used elsewhere). Ablation columns are in Appendix B.

5.3 Which readout transfers?

In most cases, the scalar head transfers better than the LM head. At 1.7B it leads by 2.0 points of mean primary score over three seeds (0.641 vs. 0.621, Table 3), by 13 points on Counterfactual, by 2–3 on MASSIVE, SST-5, SMS Spam, GoEmotions and StrategyQA and by 1 on Banking77, while the LM head leads by about a point on MMLU and TweetEval emotion and the rest are within a point. It is also more robust to renaming. Replacing every label of the seven held-out tasks with small label sets by a paraphrase (“ham” to “legitimate message”, “optimism” to “hopeful”; Appendix A) costs the scalar head 6.9 points of accuracy, the LM head 9.3 and the zero-shot model 10.0 (Table 4). Both readouts depend on pretrained representations that need not be invariant to label paraphrases. This sensitivity is already present before fine-tuning (Section 5.7) and may reflect lexical familiarity or benchmark exposure during pretraining; this experiment cannot distinguish these explanations. At 0.6B the ordering is the same (0.568 vs. 0.527). Training both heads and summing their scores does not help: 0.617 held-out at 2,000 steps, below either head alone, and 0.600 on the renamed labels. These results favour the scalar head for training and the LM head for zero-shot scoring.

Single-token readout. The concurrent this-that-model [9] reads its decision at one designated position. To isolate the readout, we train it under otherwise identical conditions: the option letters’ logits at the answer position (the first-token protocol of Section 5.1), with the same LoRA, objective and mixture (Qwen3-1.7B-Base, seed 0). On the same 500 test examples per task it reaches 0.626 against 0.652 for the scalar pack-readout and 0.617 for the LM head (Table 24, Appendix C). Paired bootstrap 95% intervals over examples are +1.5 to +3.8 points for the scalar head over the single-token readout and −2.2 to +0.4 for the LM head; the scalar head is also better calibrated (Brier 0.414 against 0.447, ECE 0.097 against 0.116). These results come from one seed, and the intervals do not include variation across seeds.

Readout (seed 0)Held-out mean7 renamable tasksSame, labels renamed
zero-shot0.6260.6960.596
scalar head0.6980.7300.661
LM head0.6610.6790.586
both heads0.6440.6480.600
LM head, byte-normalized logits0.6780.713–
single-token readout0.679––
Table 4. Readout comparison, Qwen3-1.7B-Base, seed 0: accuracy on all held-out tasks, on the seven whose labels can be renamed, and on the same seven with every label paraphrased. The single-token readout is measured on the 5,835 matched examples of Table 24, where the scalar head scores 0.696 and the LM head 0.650; it was not evaluated with renamed labels.

5.4 Calibration on unseen tasks

Table 5 and Figure 4 separate two notions of calibration. The proper scores that the objective optimizes improve on unseen tasks for both readouts: Brier from 0.507 to 0.452 (LM head) and 0.418 (scalar head), NLL from 1.149 to 0.994 and 0.881, together with confidence ranking (AUROC 0.69 to 0.73 and 0.75; accuracy on the most confident half 0.74 to 0.79 and 0.81). Top-1 ECE at T = 1 does not improve with the LM head (0.114 to 0.121) but does with the scalar head (0.098): the LM head’s logits retain the language model’s scale, improving confidence ranking without improving top-1 calibration; the scalar head learns its own scale. One cross-fitted temperature per task approximately halves ECE for both readouts, so a small labelled sample of a new task still improves its absolute calibration.

The objective matters more for the distribution than for accuracy. Gold-only SFT matches the log score on accuracy (0.664 vs. 0.667, LM head) but has the worst distribution of the models trained with a supervised loss (NLL 1.22 vs. 0.99; ECE 0.150 vs. 0.121): its loss never compares incorrect options with the gold one. Training on the Brier score, a calibration reward an RLCD-style method would optimize (Section 3.4), reduces accuracy by 2.9 points, improves ECE slightly (0.111) but produces a worse held-out Brier score (0.471 vs. 0.452) and AUROC (0.70 vs. 0.73); of the two proper scores, the log score is the better training objective. REINFORCE with an accuracy reward matches the log score on accuracy (0.657) but has a mean confidence of 0.92 at accuracy 0.66; its ECE (0.266) is twice the untrained model’s and its NLL (2.45) twice that of any proper-score objective. This is the reward-induced over-confidence that Damani et al. [14] report for generators trained by reinforcement learning with verifiable rewards (RLVR) [32], reproduced in a one-pass decision model. When labels are available, the proper-score objective attains the same accuracy without this cost.

5,835 predictions
Reliability
Selective prediction
0.780Zero-shot, answers when confidence ≥ 0.670.862Trained (scalar head), answers when confidence ≥ 0.77
Figure 4. Calibration on held-out tasks, Qwen3-1.7B-Base, zero-shot vs. the trained scalar head, pooled over the twelve tasks (5,835 predictions; bins with at least 20). Left: reliability diagram; the dashed diagonal indicates equality between confidence and accuracy. Both models remain close to it without temperature scaling in this pooled view (pooled ECE 0.04 for both). Pooling can hide opposite errors on different tasks, so the text and tables report ECE per task, which is stricter (Table 5); the task menu shows each task alone. Right: accuracy on the most confident fraction of examples; the curve rises as coverage shrinks because the confidence ranks correct decisions above wrong ones.
ModelaccECEBrierNLLAUROCacc@50% cov.
zero-shot raw0.6260.1140.5071.1490.6900.743
zero-shot raw + TS0.6260.0520.4821.0630.6910.744
set-CE, LM head0.6670.1210.4520.9940.7290.789
set-CE, LM head + TS0.6670.0530.4260.8780.7190.783
gold-only SFT0.6640.1500.4751.2200.7360.792
set-CE, scalar head0.6920.0980.4180.8810.7490.814
Brier loss0.6380.1110.4711.0080.6970.743
REINFORCE (accuracy reward)0.6570.2660.5862.4500.7120.770
Table 5. Calibration on held-out tasks, Qwen3-1.7B-Base, mean over tasks. “+TS”: one temperature fitted on half of each task’s test set, evaluated on the other half.

Ordinal decisions use the whole distribution. SST-5 has five ordered levels. Table 23 (Appendix C) compares the argmax with the expected level Σk k pk, the Score estimator (Section 3.3). For all five models, the rounded expected level reduces mean absolute error and errors of two or more levels, and improves sentence ranking. The trained 1.7B scalar head improves most: mean absolute error falls from 0.64 to 0.55 levels, errors of two or more levels from 8.7% to 5.5%, and rank correlation with the gold level rises from 0.76 to 0.83 (three seeds). The distribution retains information discarded by the argmax.

5.5 Against prompting a language model to name the label

Prompted classification asks an instruction-tuned model for a label and parses its reply. Decision models restrict outputs to the option set, eliminating parsing failures and invalid labels [68]. Table 6 compares these approaches at matched model sizes. Instruction-tuned Qwen3-1.7B, 4B and 8B receive the scorer’s prompt without its cue line, followed by an instruction to return a listed answer, in chat format with thinking disabled. Free replies are sampled with the settings the model’s authors recommend (temperature 0.7, top-p 0.8, top-k 20; seed 0) on the full test sets, matched to options, and unmatched replies count as wrong. We also evaluate greedy decoding constrained to the option strings.

First, prompting is less accurate than scoring the same model’s base version zero-shot on most tasks, and far below the trained model: a mean primary metric of 0.52 for free and for constrained replies, against 0.55 zero-shot and 0.64 trained. Second, the free replies are invalid 1.7% of the time on average and 6 to 7% on Banking77 and TweetEval emotion, even though the instruction asks for one of the listed answers exactly as written. Table 25 (Appendix C) gives examples: on Banking77 the model produces plausible intents absent from the list (“reverted top up” for “top up reverted”, “freeze my account” for “compromised card”), on emotion tasks it answers with a synonym of a label (“angry”, “sad”), and on the question tasks it explains instead of choosing. Constrained decoding removes the invalid replies but leaves the mean unchanged, consistent with the model keeping its preference for its own phrasing in the first tokens. Third, every prompted decision requires sequential decoding: 99 ms median for free replies and 63 ms constrained on a Lambda A100-40GB, against 47–54 ms for the packed pass on the Colab A100-40GB (Section 5.6); the ratio is approximate across the two hosts. At 4B and 8B the same pattern holds with a smaller gap: the prompted models reach 0.63 and 0.66 with 1% invalid replies at 128–131 ms, against 0.60 and 0.63 zero-shot and 0.69 and 0.72 trained; our trained 1.7B (0.64) is ahead of the prompted 4B, and our trained 4B (0.69) of the prompted 8B.

ours 1.7Bprompted Qwen3-1.7B
TaskZero-shotTrainedfreeconstrained
Banking770.4010.6800.4140.417
MASSIVE0.6730.7460.5180.518
SST-50.3410.4550.2520.255
TweetEval emotion0.6980.6580.5050.537
TweetEval hate0.5010.5820.3520.352
Rotten Tomatoes0.8700.9030.8590.861
SMS Spam0.6450.6780.7310.719
Counterfactual0.4520.6410.6160.599
GoEmotions0.1940.2730.1250.130
ARC-Challenge†0.7340.8250.7130.719
MMLU†0.5840.6310.5420.535
StrategyQA0.5460.6180.5910.591
Mean0.5530.6410.5180.520
Invalid replies1.7%0.0%
Median ms per decision48489963
ours 4Bprompted Qwen3-4B
TaskZero-shotTrainedfreeconstrained
Banking770.5780.7560.6060.606
MASSIVE0.7280.7720.6980.697
SST-50.3930.5080.3690.369
TweetEval emotion0.7100.6720.6740.675
TweetEval hate0.6240.5660.6100.609
Rotten Tomatoes0.8870.9150.9050.902
SMS Spam0.5100.7660.5370.537
Counterfactual0.5280.7640.7110.719
GoEmotions0.1990.2820.2820.284
ARC-Challenge†0.8360.8850.8670.868
MMLU†0.6580.7020.6590.669
StrategyQA0.5230.7100.6550.649
Mean0.5980.6910.6310.632
Invalid replies0.8%0.0%
Median ms per decision606013181
ours 8Bprompted Qwen3-8B
TaskZero-shotTrainedfreeconstrained
Banking770.6100.7550.5810.583
MASSIVE0.8020.8090.7410.727
SST-50.2570.5260.4230.425
TweetEval emotion0.7340.6610.6910.706
TweetEval hate0.5930.5630.7040.702
Rotten Tomatoes0.8910.9270.9120.915
SMS Spam0.6920.8790.7710.774
Counterfactual0.4700.7960.6200.620
GoEmotions0.2250.3200.2310.238
ARC-Challenge†0.8840.9270.9100.916
MMLU†0.6800.7600.7150.715
StrategyQA0.6850.7260.6760.680
Mean0.6270.7210.6650.667
Invalid replies0.7%0.0%
Median ms per decision12884
Table 6. Prompting the instruction-tuned model of the same size vs. option scoring, Qwen3 at 1.7B, 4B and 8B, twelve held-out tasks (macro-F1, or accuracy for †); ours on the 500-example subsets, the prompted models on the full test sets. Free: sampled reply matched to an option; constrained: greedy decoding restricted to the option strings. Invalid replies name no option and count as wrong. Latency per decision at batch size one, median over tasks: prompting timed on the task prompts on a Lambda A100-40GB; ours read from the sweep of Table 27 on the Colab A100-40GB at each task’s number of options, one packed pass whether zero-shot or trained (Section 5.6).

Larger models. Table 7 repeats the free-reply protocol for eleven hosted models (Appendix F). Their held-out means range from 0.710 (DeepSeek V3.2) to 0.806 (Claude Fable 5.1), against 0.721 for our trained 8B (two seeds; the trained 14B scores 0.714, and 0.731 with 1,000 steps at half the learning rate, two seeds). DeepSeek V3.2, an open-weight model of 671B parameters, scores below it (our 8B is ahead on 6 of the 12 tasks); the current frontier models, GPT-6 Astra (0.793), Claude Fable 5.1 (0.806), Muse Spark 1.3 (0.790) and Kimi K3 (0.778, 2.8T parameters), are 6 to 9 points above it; GPT-5 and Claude Sonnet 4.6, of the previous generation, are 3 to 5 points above it. At most 0.9% of their replies name no option. They return no probability, their replies must still be parsed, and each decision requires sequential decoding: about 4 steps for Qwen3-235B and DeepSeek V3.2, 6 to 138 on the closed ones and 177 for Kimi K3, which reasons before it answers, against one prefill.

Figure 5 compares the three approaches by model size. The trained curve exceeds the prompted curve at every tested size, but the gap narrows from 12 points at 1.7B to 6 at 4B and 8B. Trained performance plateaus from 8B to 14B; hosted prompted performance continues to increase, reaching 0.78 for gpt-oss-120b. The four other trained base models (lettered discs) follow the same curve despite different initial scores: SmolLM2 1.7B reaches the Qwen3-1.7B score from 25 points lower, OLMo-2 1B follows the curve at 1B, and gpt-oss-20b and Qwen2.5 32B extend it. These results do not establish performance at frontier scale; at the sizes tested, the trained model outperforms prompting at its own size and prompted models two to eighty times larger.

Ours, trained
size undisclosed0.6B1.7B4B8B14BOOSSGGQQ1.7B4B8Bgpt-oss-120bKimi K3GLM-5.3Qwen3-235BDeepSeek V3.2
Figure 5. Held-out mean vs. parameters for three ways of using a language model as a classifier: prompting an instruction-tuned model to name the label (Qwen3 on our GPU; hosted open-weight models at their public size), zero-shot option scoring, and our trained models (Qwen3 at each size; outline only: the gentler 14B run of Section 5.7; lettered discs: the four other base models, one size each, gpt-oss-20b at its 21B total parameters). A dotted segment joins each base model’s zero-shot point to its trained point; closed models of undisclosed size sit in the shaded column. Lines join each Pareto frontier (best at or below each size). The arrow compares our largest model with the prompted Qwen3-235B, the same family and pretraining, so the ratio is a within-family comparison (hosted sizes are total parameters; Qwen3-235B has 22B active). The menu shows each task alone (ours and the 1.7B to 8B prompted Qwen3 at their table settings).
TaskOurs, trainedClaude Fable 5.1adaptive thinkingGPT-6 Astraminimal reasoningMuse Spark 1.3minimal reasoninggpt-oss-120bKimi K3Claude Sonnet 4.6no thinking
Banking770.7550.8680.8570.8330.8430.8470.800
MASSIVE0.8090.8900.8900.8600.8380.8590.832
SST-50.5260.5440.5100.4830.4930.4950.525
TweetEval emotion0.6610.7710.7750.7690.8020.7580.761
TweetEval hate0.5630.7010.6050.7530.7340.7670.752
Rotten Tomatoes0.9270.9520.9480.9300.9080.9440.936
SMS Spam0.8790.9860.9760.9580.8970.9110.930
Counterfactual0.7960.8180.8210.8210.8470.7360.721
GoEmotions0.3200.3280.3370.3080.3620.2750.285
ARC-Challenge†0.9270.9680.9720.9700.9500.9760.964
MMLU†0.7600.9220.9380.9240.8620.9120.898
StrategyQA0.7260.9230.8830.8680.8000.8510.829
Mean0.7210.8060.7930.7900.7780.7780.769
Invalid replies0.0%0.0%0.0%0.1%0.7%0.1%
Mean reply tokens2513138–1776
TaskOurs, trainedClaude Sonnet 5adaptive thinkingGLM-5.3low reasoningGPT-5minimal reasoningQwen3-235BDeepSeek V3.2
Banking770.7550.8110.7840.7870.7330.726
MASSIVE0.8090.8550.8090.8320.8260.833
SST-50.5260.5170.4840.5110.5150.397
TweetEval emotion0.6610.7410.7360.7480.7680.761
TweetEval hate0.5630.6760.7410.6690.6970.544
Rotten Tomatoes0.9270.9460.9400.9400.9360.928
SMS Spam0.8790.9730.9200.8680.8620.907
Counterfactual0.7960.5640.7180.8010.7370.689
GoEmotions0.3200.3500.2460.2770.2110.269
ARC-Challenge†0.9270.9700.9740.9580.9500.942
MMLU†0.7600.9080.8940.8760.8260.848
StrategyQA0.7260.8540.8680.7570.7820.674
Mean0.7210.7640.7590.7520.7370.710
Invalid replies0.1%0.1%0.1%0.9%0.1%
Mean reply tokens13132144
Table 7. Hosted instruction-tuned models prompted to name the label (free replies), against our trained 8B model: macro-F1, or accuracy for †, the share of invalid replies, and the mean number of decode tokens per reply, reasoning included. GPT-5, GPT-6 Astra and Muse Spark 1.3 run with minimal reasoning effort, GLM-5.3 with low effort, and the other models with their default settings, which for Claude Sonnet 5 and Claude Fable 5.1 means adaptive thinking on (Fable cannot turn it off) and for Claude Sonnet 4.6 means no thinking. Model ids, sizes, providers and settings in Appendix F.

5.6 Efficiency

Figure 6 and Table 27 (Appendix C) compare the packed pass with KV-cache replication, one sequence per option, constrained greedy decoding of the label (a trie over option tokens) and JSON generation of a one-sentence rationale plus label, on the same base model and eager PyTorch execution. Absolute times are dominated by per-forward overhead in this setup (about 40 ms at 1.7B), so the ratios are the comparable quantity. Packed scoring uses the LM head; the JSON baseline adds an instruction to generate a rationale and label. Each method is timed once per input over 20 Banking77 queries and three random label sets per option count, so the bands of Figure 6 show variation across inputs and label draws. Packed latency is 47–48 ms from 2 to 16 options, 54 ms at 77, 91 ms at 255 (a 926-token prompt), with p90 within 2 ms of the median. KV replication costs about 2× throughout and 4.8× at 255 options, and at 4B does not fit in 40 GB at 255 options (Appendix H). Generation incurs a cost per decode step, with the total determined by the generated text: constrained decoding takes 164–217 ms median across the sweep, 2.4–4.3× the packed pass, with p90s up to 200 ms above its median. After prefill, each additional label token requires one decode step (3.0 to 3.7 steps on average over three random label sets per count), so decoding cost depends directly on the selected label’s length; JSON with a rationale takes 1.0–1.2 s median, 13–22× the packed pass, with p90s 0.2–0.4 s higher. One sequence per option, the lm-evaluation-harness default, is competitive up to 8 options and then grows with options times prompt length. At on-demand A100 prices ($1.5–2 per hour), a 77-way decision costs about $0.03 per thousand.

Cost of the conversion. Converting a pretrained model requires one LoRA run: 2,000 steps of 8 examples, 16,000 presentations of public classification examples with 154 packed tokens each, 2.5M training tokens total, without synthetic data, labels from a larger model or reinforcement learning. At 6 × parameters × tokens that is 25 PFLOPs at 1.7B and 120 at 8B, 20 minutes on one A100 at 1.7B (38 for the 1,000-step 4B run), 2.1 and 3.3 hours on a DGX Spark at 8B and 14B, with the time dominated by kernel-launch overhead. Table 26 (Appendix C) compares adaptation data and compute for dedicated zero-shot classifiers (Section 5.8) and instruction-tuned models. The best BTZSC system near our size, an NLI cross-encoder, was fine-tuned on about 885,000 NLI pairs for three epochs, roughly ten A100 hours for a comparable run; the best 8B system, Qwen3-Reranker-8B, on 19 million retrieval pairs, twelve million of them synthetic; the best embedding model on 800 million web-mined pairs and then 3 million annotated ones; Flan-T5-XXL’s instruction tuning cost 3,000 times our run; and the post-training of the instruction-tuned and frontier models is not disclosed. Our conversion uses three orders of magnitude fewer examples than the retrieval-trained systems, every example is public, and it leaves the base model’s weights untouched.

Packed (ours)
3.23.03.73.43.73.5JSON generationKV replicationConstrained decodingPacked (ours)
Figure 6. Latency per decision vs. number of options, Qwen3-1.7B-Base on an A100, both axes logarithmic: lines are medians over the same 20 inputs, bands the central 80% of inputs and label draws. Packed latency rises from 47–48 ms at 2–16 options to 91 ms at 255; its per-input spread is small. JSON generation with a rationale is an order of magnitude slower, while constrained decoding is 2.4–4.3× slower. Annotations report mean decode steps. Larger option sets may have longer labels, which lengthens decoding; each option count uses three random draws from pooled intent names, so this sweep does not separate the two effects. The 0.6B and 4B sweeps are those of Appendix H; one sequence per option can be shown from the legend.

5.7 The recipe: base models and what drives transfer

Base models. Table 8 applies the identical procedure to Qwen3-Base at five sizes, SmolLM2-1.7B (Llama architecture, the size of Qwen3-1.7B), OLMo-2-1B (open training data), Qwen2.5-32B and gpt-oss-20b. SmolLM2-1.7B starts 19 points below Qwen3-1.7B zero-shot and finishes within a point (0.68 vs. 0.69 held-out accuracy); OLMo-2-1B, the weakest base, gains 21 points; Qwen3-4B and 8B gain 6 each and 14B one, all ending with ECE between 0.08 and 0.09: the accuracy gain shrinks as the base model grows, the calibration gain does not. gpt-oss-20b [51] differs from the others: a post-trained mixture of experts (21B parameters, 3.6B active) with sliding-window layers and attention sinks, run through a second, complete implementation of the method that builds its layer-wise masks. Zero-shot, it scores options poorly (0.51 macro-F1), as is typical of post-trained chat models; the recipe takes it to 0.70 with the scalar head, with ECE from 0.21 to 0.10, the largest gain of any model above 1.7B.

Base modelZero-shotLM headScalar headECE zero-shotECE LM headECE scalar
Qwen3-0.6B0.4310.5560.6340.2600.1690.113
OLMo-2-1B0.3970.5330.6050.2920.2070.133
SmolLM2-1.7B0.4340.5810.6800.1620.1580.117
Qwen3-1.7B0.6260.6670.6920.1140.1210.098
Qwen3-4B0.6670.6750.7230.1130.1070.094
Qwen3-8B0.6920.7410.7470.1220.1100.087
Qwen3-14B0.7360.7320.7490.1220.1160.086
gpt-oss-20b0.5900.7260.7370.2080.1220.100
Qwen2.5-32B0.734–0.7680.105–0.080
Table 8. The recipe across base models: held-out mean accuracy and ECE (T = 1), zero-shot and after training with each readout.

Training larger models. The recipe’s learning rate and step count were chosen at 1.7B. At 14B the unchanged recipe fits the mixture more tightly (final training loss 0.35 vs. 0.44 at 8B) and gains only a point: short-text regressions offset many-label improvements. Table 9 repeats the run with a lower learning rate, fewer steps, or both: every gentler run reaches 0.729 to 0.731, above the trained 8B, with the short-text tasks within two to four points of zero-shot, the many-label gains retained and ECE at 0.07 to 0.08. The amount of adaptation is decisive, whether controlled by step count or learning rate. The setting chosen at 1.7B, where increases in both continued to help, over-adapts a 14B model (the LM head at 14B falls below zero-shot on the mean, 0.699). Applied to Qwen2.5-32B, sharded over two GPUs, the same gentler run takes the held-out mean from 0.704 to 0.737, the highest of any trained model here, in 1.7 hours. The ladders of Figures 5 and 3 show this 1,000-step, half-learning-rate run at 14B as their outline-only point; the tables keep the unchanged recipe.

RunHeld-out meanMany-label tasksShort-text tasksECE
zero-shot0.7080.5580.6920.122
recipe: lr 2×10⁻⁴, 2,000 steps0.7140.6350.6130.086
lr 2×10⁻⁴, 1,000 steps0.7310.6320.6680.080
lr 10⁻⁴, 2,000 steps0.7290.6320.6570.074
lr 10⁻⁴, 1,000 steps0.7310.6260.6710.071
lr 5×10⁻⁵, 2,000 steps0.7300.6340.6540.081
recipe, LM head0.6990.6010.5980.116
Table 9. Gentler runs at 14B: held-out mean, the three many-label tasks (Banking77, MASSIVE, GoEmotions), the four short-text tasks (SST-5, TweetEval emotion and hate, SMS Spam), and ECE. One seed each, two for the 1,000-step run at half the learning rate.

Number of label sets. At a fixed budget of 16,000 example presentations and 2,000 steps, we train the scalar head on 4 to 18 of the training tasks, three random subsets of tasks per count (Figure 7; Table 28 in Appendix C). Held-out accuracy increases with the number of training label sets at fixed data, with the largest increase on many-label tasks. Every count remains above zero-shot on the held-out mean (0.638 with 4 label sets against 0.626), which then rises to 0.69 at 18, with a spread across draws about as large as the trend. The three many-label tasks improve from 0.47 with 4 sets to 0.61 with 18, and ECE falls from 0.15 to 0.10. This is consistent with a model trained on few label sets learning those sets, and a model trained on many label sets learning to interpret the listed labels.

All held-out tasks
Figure 7. Held-out accuracy vs. number of training label sets at fixed data, Qwen3-1.7B-Base, scalar head; band: ±1 s.d. over three random task subsets per count, each subset a faint dot. Accuracy increases with the number of training label sets at fixed data, most steeply on the many-label tasks; the mean is above zero-shot from four sets on. The menu shows ECE and AUROC from the same runs (Table 28).

Amount of data. Table 29 (Appendix C) trains the scalar head for the same 2,000 steps on pools of 2,000 to 36,000 examples. Accuracy saturates early and calibration does not: the 2,000-example pool, seen eight times, already gives 0.595 against 0.553 zero-shot and the full pool 0.641, while ECE falls from 0.31 to 0.10. Repeated examples stop improving accuracy long before they stop increasing over-confidence, so the pool should be as large as the label sets allow.

Sensitivity to label names. Label names are part of the prompt and affect performance. On SMS Spam, “ham”/“spam” gives the 0.6B base model 0.12 accuracy (it calls nearly every message spam) and “legitimate message”/“unsolicited junk” 0.75; at 1.7B the order reverses (0.80 and 0.48), and the trained scalar head keeps the base model’s preference (0.90 and 0.52).

Neighbouring label sets in the mixture. Two training sets closely resemble held-out ones (Section 4): Emotion shares three of TweetEval emotion’s four labels, and TweetEval offensive is a broader version of TweetEval hate. Table 10 reports the same recipe without them. Removing these datasets improves their held-out counterparts: TweetEval emotion rises from 0.66 to 0.73 and TweetEval hate from 0.58 to 0.68 (its ECE from 0.18 to 0.06), three to five times the seed spread. The held-out mean is unchanged (0.641 to 0.634): SMS Spam and Counterfactual, the two imbalanced binary tasks, lose about what the tweet tasks gain, within their seed spread; TweetEval offensive was the only imbalanced binary task in the mixture. At 4B, with 1,000-step runs, the same removal adds 2 points on emotion and 5 on hate, which returns to its zero-shot level, and again leaves the mean unchanged (one seed each). This suggests that the form of the label sets determines a training set’s value to the mixture. A training set whose label set nearly duplicates a held-out one lowers that task’s score, because the model learns a prior and class boundaries for labels that carry a different meaning at test time; a training set of the same form helps regardless of its domain.

SizeMixtureStepsSeedsTweetEval emotionTweetEval hateGoEmotionsHeld-out mean
1.7Ball 18 tasks2,00030.6580.5820.2730.641
1.7Bwithout Emotion, TweetEval offensive2,00020.7260.6760.2330.634
4Ball 18 tasks1,00010.7320.5660.2910.693
4Bwithout Emotion, TweetEval offensive1,00010.7490.6180.2640.698
Table 10. The mixture without the two training sets that neighbour held-out tasks (Emotion, TweetEval offensive), scalar head, 500-example subsets: the neighbours’ held-out relatives improve and the held-out mean is unchanged. Each row is the mean over its seeds; the 4B pair are 1,000-step runs at the recipe’s learning rate (the “short” setting of Table 9).

What the training teaches. These results suggest that training teaches classification: interpreting the prompt’s label set, comparing options and assigning probabilities. Six observations support this interpretation. Training uses 2.5M tokens of public labels against trillions of pretraining tokens, and accuracy saturates by a pool of 8,000 examples while the number of label sets continues to help (Tables 29 and 28). The gains occur where zero-shot scoring fails on form, the label prior and surface competition, and they are smallest for the base models that already handle the tasks (Section 5.2). With paraphrased labels, the scalar head still outperforms zero-shot scoring and the trained LM head (Section 5.3). The same procedure works on every tested base model, including a post-trained mixture of experts. Removing the two training sets that neighbour held-out tasks raises the neighbours’ scores (Table 10), which is consistent with the mixture transferring form and a near-duplicate label set transferring a prior. The task-trained ceiling of Table 3 quantifies the additional value of task knowledge: 10 to 25 points on many-label tasks, where the general model relies on pretrained label semantics. This hypothesis predicts that training on label sets from an unrelated domain should transfer nearly as well; testing it remains future work.

5.8 Comparison with dedicated zero-shot classifiers

BTZSC [1] evaluates zero-shot classifiers on 22 datasets from texts and a verbalized description of every label (“This example news text is about business news”). We list the descriptions in the prompt and score each as an option. Five BTZSC datasets (AG News, Yelp, IMDB, Emotion, Yahoo) are in our mixture, so we also train a model on the mixture without them. Figure 8 places our model among the 35 leaderboard systems by size; Table 30 in Appendix D gives macro-F1 overall and per task group against the best model of each BTZSC family and the small LLMs in its leaderboard, which are scored by the first-token protocol (0.43 overall for a 3B instruct model, 0.65 for Qwen3-4B).

On the two intent datasets shared with our held-out set, the overall-best model of each family scores 0.69 (reranker), 0.63 (embedding), 0.24 (NLI) and 0.35 (12B LLM) on Banking77 and 0.67, 0.57, 0.40 and 0.36 on MASSIVE; our 1.7B model reaches 0.66 and 0.74 with its own label names (Table 3). Under the benchmark’s protocol, the untrained 1.7B base model scores 0.53 overall: above the 3B first-token LLM (0.43), below the 4B and 12B LLMs (0.65, 0.67) and below every specialist family, with the intent group (0.35) reduced most by the verbalized descriptions.

Trained on the mixture without the five BTZSC datasets, the same 1.7B model reaches 0.67 overall with the LM head and 0.70 with the scalar head: above every family leader except the 8B reranker (0.72), above the 12B first-token LLM (0.67) at a seventh of its size, and 27 points above the 3B LLM. By task group the scalar head is within 3 points of the reranker on sentiment (0.90 vs. 0.92) and emotion (0.48 vs. 0.49), 4 points below it on intent (0.66 vs. 0.70), and on topic (0.65) just below the reranker (0.66) and the 12B LLM (0.69). The same recipe at the other sizes gives 0.67 at 0.6B, 0.73 at 4B, 0.74 at 8B and 0.75 at 14B (the gentler run of Section 5.7; two seeds at 1.7B, 4B and 8B): from 4B on it is above the best system of the leaderboard, at half its size (0.727 vs. 0.722), and at the reranker’s own size it is the best system on the leaderboard (0.740; intent 0.73 vs. 0.70, topic 0.69 vs. 0.66, emotion 0.45 vs. 0.49). Under the benchmark’s own protocol, designed for classifiers that read a label description, the model trained on other label sets is above every family’s frontier at each size we trained, on 500-example subsets of its datasets (Figure 8). Figure 3 compares individual systems on the three datasets shared with our held-out set. BTZSC reports inference speed only as a normalized figure without per-model numbers, on other hardware, so we make no speed claim against these families.

Ours, trained
0.6B1.7B4B8B14B
Figure 8. Overall macro-F1 on BTZSC (22 datasets, the benchmark’s protocol) against parameters. Small grey markers, one shape per family: the 35 leaderboard systems, on full test sets; grey staircases: each family’s Pareto frontier, continued flat to the family’s largest system. Orange: our Qwen3 models trained on the mixture without the five BTZSC datasets (scalar head, 500-example subsets), at each size, with a line through their frontier; the outline-only point is the 14B trained for 1,000 steps at half the learning rate (Section 5.7); open circles: the same base models zero-shot, a dotted segment joining each to its trained point. The menu shows each task group of Table 30.

6 Limitations

Two limitations follow from the design. Chain-of-thought prompting and reinforcement-trained reasoning raise accuracy on tasks that require intermediate steps [16, 73]; a one-pass decision has none, so the gain is unavailable to our model, and StrategyQA, the one held-out task that needs multi-hop inference, is among the tasks where our models trail the frontier most (0.65 at 1.7B and 0.76 at 14B against 0.92 for the best hosted model). For such questions, low-confidence cases could be routed to a reasoning model. The classifier interprets option text literally and therefore requires descriptive label names (Section 5.7).

The evaluation also has limitations. Held-out datasets are excluded from our fine-tuning mixture (two binary label sets recur, Section 4), but benchmark exposure during pretraining cannot be excluded. Our results therefore do not establish generalization to tasks absent from pretraining. Calibration is measured on the twelve held-out tasks with 500 examples each (335 for Amazon Counterfactual), and on TweetEval hate the trained model trades accuracy for macro-F1. All latencies come from one eager PyTorch implementation; speed ratios may differ under optimized serving. The recipe was tested with LoRA, on models of up to 32B parameters, in English, with hyperparameters tuned at 1.7B and only the gentler setting tested above 14B.

7 Conclusion

Selecting from a known option set requires no decoding. A pack-readout, one packed forward pass, computes exact option likelihoods at the cost of a prefill, and cross-entropy over the option set on public classification data turns a small pretrained model into a classifier that accepts a new label set at inference time: on 12 held-out tasks, the mean primary score rises from 0.55 to 0.64 at 1.7B and from 0.63 to 0.72 at 8B, while calibration improves (Brier 0.51 to 0.42 at 1.7B). One set of weights serves every tested label set: on unseen tasks it is on or above the size–accuracy frontier of dedicated zero-shot classifiers, leads their leaderboard from 4B on under their own protocol, and is more accurate than prompting the instruction-tuned model of its size, at one prefill per decision. The method consists of a block attention mask, a softmax over the option set and one LoRA training run; it applied unchanged to all five base models tested. Model implementation, training code and trained adapters are released.

Acknowledgements and availability

Acknowledgements and disclosures. Claude Code and OpenAI Codex assisted with training-code development, manuscript revisions, and figure-generation code. The research idea originated with the author, who takes full responsibility for the work. The study was entirely self-funded, and the author declares no competing interests.

Data availability. Benchmark sources are cited in Appendix I; the repository’s DATA.md lists dataset identifiers and access links. The evaluation reports (per-example scores on the held-out tasks, aggregate scores on BTZSC, the hosted models’ replies), the latency measurements and the BTZSC leaderboard snapshot are released as a results archive (github.com/maxz411/classifier-from-llm/releases/tag/v0.1.0).

Code availability. The model implementation and training and evaluation code are available under Apache-2.0 at github.com/maxz411/classifier-from-llm, as the Python package packreadout. The trained adapters are on Hugging Face under CC BY-NC 4.0 (huggingface.co/collections/maxz411/pack-readout-classifiers).

References

  1. Ilias Aarab. BTZSC: A benchmark for zero-shot text classification across cross-encoders, embedding models, rerankers and LLMs. In ICLR, 2026.
  2. Loubna Ben Allal et al. SmolLM2: When smol goes big – data-centric training of a small language model. arXiv:2502.02737, 2025.
  3. Tiago A. Almeida, José María Gómez Hidalgo, and Akebo Yamakami. Contributions to the study of SMS spam filtering: New collection and results. In ACM Symposium on Document Engineering, 2011.
  4. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of EMNLP, 2020.
  5. Stella Biderman et al. Lessons from the trenches on reproducible evaluation of language models. arXiv:2405.14782, 2024.
  6. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1–3, 1950.
  7. Tom Brown et al. Language models are few-shot learners. In NeurIPS, 2020.
  8. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. In Workshop on NLP for Conversational AI, 2020.
  9. Zehua Cheng, Wei Dai, and Jiahao Sun. this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent. arXiv:2609.23886, 2026.
  10. Hyung Won Chung et al. Scaling instruction-finetuned language models. JMLR, 2024.
  11. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
  12. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv:1803.05457, 2018.
  13. Convai Innovations. Laya. Model card, https://huggingface.co/convaiinnovations/laya, 2026. Apache-2.0 weights released 2026-09-18; accessed 2026-09-21.
  14. Mehul Damani et al. Beyond binary rewards: Training LMs to reason about their uncertainty. arXiv:2507.16806, 2025.
  15. DeepSeek-AI. DeepSeek-V3 technical report. arXiv:2412.19437, 2024.
  16. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948, 2025.
  17. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In ACL, 2020.
  18. Jack FitzGerald et al. MASSIVE: A 1M-example multilingual natural language understanding dataset with 51 typologically-diverse languages. arXiv:2204.08582, 2022.
  19. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. TACL, 2021.
  20. Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007.
  21. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, 2017.
  22. Franz A. Heinsen. An algorithm for routing vectors in sequences. arXiv:2211.11754, 2022.
  23. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
  24. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. In EMNLP, 2021.
  25. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, 2019.
  26. Edward J. Hu et al. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
  27. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? On the calibration of language models for question answering. TACL, 2021.
  28. Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput LLM inference with shared prefixes. arXiv:2402.05099, 2024.
  29. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A dataset for question answering via sentence composition. In AAAI, 2020.
  30. Kimi Team. Kimi K2: Open agentic intelligence. arXiv:2507.20534, 2025.
  31. Md Kowsher et al. Predicting through generation: Why generation is better for prediction. arXiv:2502.17817, 2025.
  32. Nathan Lambert et al. Tülu 3: Pushing frontiers in open language model post-training. arXiv:2411.15124, 2024.
  33. Stefan Larson et al. An evaluation dataset for intent classification and out-of-scope prediction. In EMNLP, 2019.
  34. Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. Building efficient universal classifiers with natural language inference. arXiv:2312.17543, 2023.
  35. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs as generalist embedding models. In ICLR, 2025.
  36. Yibo Li et al. ConfTuner: Training large language models to express their confidence verbally. arXiv:2508.18847, 2025.
  37. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv:2308.03281, 2023.
  38. Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu-lee Wang, Qing Li, and Xiaoqin Zhong. Label supervised LLaMA finetuning. arXiv:2310.01208, 2023.
  39. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. In EMNLP, 2023.
  40. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
  41. Dongyang Ma, Yan Wang, and Tian Lan. Block-attention for efficient prefilling. In ICLR, 2025.
  42. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011.
  43. Meta. Introducing Muse Spark 1.3. https://research.meta.ai/blog/introducing-muse-spark-1-3, 2026.
  44. Meta AI. Llama 3.2 model card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md, 2024.
  45. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In EMNLP, 2018.
  46. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In ACL, 2022.
  47. Mistral AI. Mistral NeMo. https://mistral.ai/news/mistral-nemo, 2024.
  48. Minxue Niu and Emily Mower Provost. Contrastive distillation of emotion knowledge from LLMs for zero-shot emotion recognition. arXiv:2505.18040, 2025.
  49. OLMo Team. 2 OLMo 2 furious. arXiv:2501.00656, 2025.
  50. James O’Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. I wish I would have loved this one, but I didn’t – a multilingual dataset for counterfactual detection in product review. In EMNLP, 2021.
  51. OpenAI. gpt-oss-120b & gpt-oss-20b model card. arXiv:2508.10925, 2025.
  52. Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In AAAI, 2015.
  53. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005.
  54. Qwen Team. Qwen2.5 technical report. arXiv:2412.15115, 2024.
  55. Qwen Team. Qwen3 technical report. arXiv:2505.09388, 2025.
  56. Nir Ratner et al. Parallel context windows for large language models. In ACL, 2023.
  57. Joshua Robinson, Christopher Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. arXiv:2210.12353, 2022.
  58. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. In AAAI, 2020.
  59. Victor Sanh et al. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.
  60. Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In EMNLP, 2018.
  61. Ali Şenol, Garima Agrawal, and Huan Liu. Measuring reasoning quality in LLMs: A multi-dimensional behavioral framework. arXiv:2605.24661v4, 2026.
  62. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
  63. Bhavesh Sood and Jaromir Savelka. Punching above their weight: Classification-head fine-tuning of tiny language models (TLMs) for verifiable multiple-choice tasks. arXiv:2607.03801, 2026.
  64. Ihor Stepanov et al. GLiClass: Generalist lightweight model for sequence classification tasks. arXiv:2508.07662, 2025.
  65. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In NAACL, 2019.
  66. Polina Tsvilodub, Hening Wang, Sharon Grosch, and Michael Franke. Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods. arXiv:2403.00998, 2024.
  67. TypeSafe AI. Introduction – TypeSafe AI. Documentation of Jev, https://docs.typesafe.ai/introduction, 2026.
  68. TypeSafe AI. Introducing system one models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026.
  69. Franklin Wang and Sumanth Hegde. Accelerating direct preference optimization with prefix sharing. arXiv:2410.20305, 2024.
  70. Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul Röttger, and Barbara Plank. Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think. In COLM, 2024.
  71. Zhiqiang Wang, Yiran Pang, and Yanbin Lin. Large language models are zero-shot text classifiers. arXiv:2312.01044, 2023.
  72. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
  73. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
  74. Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Workshop on Noisy User-generated Text, 2017.
  75. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992.
  76. Haike Xu, Zongyu Lin, Jing Zhou, Yanan Zheng, and Zhilin Yang. A universal discriminator for zero-shot generalization. In ACL, pages 10559–10575, 2023.
  77. Z.ai. GLM-5.3. https://docs.z.ai/guides/llm/glm-5.3, 2026.
  78. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In ACL, 2019.
  79. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NeurIPS, 2015.
  80. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 Embedding: Advancing text embedding and reranking through foundation models. arXiv:2506.05176, 2025.
  81. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
  82. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In ICLR, 2024.
  83. Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In ICLR, 2024.

A Prompts

Every option is rendered as a leading space, the option text and a newline, and the newline is scored. Options are listed in dataset order.

Text classification (classify_prompt):

<instruction>
Labels: <label 1>, <label 2>, ..., <label n>
Text: <text, cut at 1500 characters>
Label:

Banking77, for example, uses the instruction “Classify the customer’s banking query into exactly one intent.” and the 77 intent names with underscores replaced by spaces.

Multiple choice and yes/no (mcq_prompt):

<context, if any>
Question: <question>
Options:
- <option 1>
- <option 2>
Answer:

Yes/no tasks use the options “yes” and “no”.

First-token symbol baseline (symbol_prompt): the cue line is replaced by lettered choices “A. <option 1>” … and the letters are scored.

Decision primitives. choice: “<state> / Question: <question> / Options: <o1>, … / Answer:”. score: levels listed as “<i>: <level>” under “Levels:”, ending “Answer (level number):”; the options are the level indices. true_false: “Answer (yes or no):” with options yes and no.

Renamed labels (Table 4): SST-5 (very negative to strongly unfavorable, negative to unfavorable, neutral to mixed or indifferent, positive to favorable, very positive to strongly favorable); TweetEval emotion (anger to furious, joy to happiness, optimism to hopeful, sadness to sorrow); TweetEval hate (non-hate to not hateful, hate to hateful); Rotten Tomatoes (negative to thumbs down, positive to thumbs up); SMS Spam (ham to legitimate message, spam to unsolicited junk); Counterfactual (not-counterfactual to describes what actually happened, counterfactual to describes something that did not happen); StrategyQA (yes to true, no to false).

BTZSC. “Pick the label that describes the text.” followed by the benchmark’s verbalized label descriptions, in the classification template.

B Additional results

TaskTest sizeZero-shot, 500Zero-shot, fullTrained, 500Trained, full
Banking773,0760.4010.3990.6800.660
MASSIVE2,9740.6730.6460.7460.740
SST-52,2100.3410.3580.4550.451
TweetEval emotion1,4210.6980.6970.6580.694
TweetEval hate2,9700.5010.4840.5820.560
Rotten Tomatoes1,0660.8700.8620.9030.888
SMS Spam5,5740.6450.6760.6780.703
Counterfactual3350.4520.4510.6410.626
GoEmotions4,5900.1940.2060.2730.295
ARC-Challenge†1,1720.7340.7450.8250.838
MMLU†14,0420.5840.5520.6310.592
StrategyQA6870.5460.5480.6180.646
Mean0.5530.5520.6410.641
Table 11. The 500-example evaluation subsets against the full test sets, Qwen3-1.7B-Base: zero-shot scoring and the trained scalar head (both columns are three-seed means; the full-test runs are three additional runs with the main configuration, reported in Table 3). Differences are at most four points per task (MMLU, TweetEval emotion) and 0.1 on the mean.
SizeZero-shot, 500Zero-shot, fullTrained, 500Trained, full
0.6B0.3660.3780.5680.567
1.7B0.5530.5520.6410.641
4B0.5980.5940.6910.695
8B0.6270.6260.7210.721
14B0.7080.7110.714–
Table 12. The same check at every size, held-out means: the 500-example subsets against the full test sets, zero-shot and trained (scalar head; at 0.6B one retrained run, at 1.7B three, at 4B and 8B the trained adapter re-evaluated). Means agree within a point except the 0.6B base zero-shot, which the subset understates by 1.2. The one reversal between sizes, the 14B base below the 8B base on MASSIVE, holds on the full test set (0.748 vs. 0.762), so it is not a subset artefact.
Taskzero-shot rawzero-shot bytesymbol5-shotgold-only SFTScalar headboth headsset-CE, LM head
Banking770.4010.153–0.5600.6030.6800.6810.663
MASSIVE0.6730.635–0.7380.7320.7460.7340.718
SST-50.3410.2660.2320.4340.4290.4550.3760.344
TweetEval emotion0.6980.5790.6460.7350.6790.6580.6860.680
TweetEval hate0.5010.3700.6570.3910.5060.5820.5390.548
Rotten Tomatoes0.8700.8700.8510.7980.9000.9030.9120.903
SMS Spam0.6450.5470.2360.7490.6270.6780.6070.659
Counterfactual0.4520.4530.4530.4870.4830.6410.4790.569
GoEmotions0.1940.114–0.1920.2570.2730.2960.283
ARC-Challenge†0.7340.7300.7160.7500.7860.8250.8140.822
MMLU†0.5840.5560.5480.5900.6120.6310.6320.650
StrategyQA0.5460.5970.3750.4280.6420.6180.6540.633
Mean0.5530.4890.5240.5710.6050.641 ± 0.0120.6170.623 ± 0.008
Table 13. Ablations, 1.7B: zero-shot raw and per-byte scoring, first-token symbol scoring, five-shot option scoring (five training examples of the task in the prompt), gold-only SFT, and the trained readouts.
Taskzero-shot rawzero-shot bytesymbol5-shotgold-only SFTScalar headboth headsset-CE, LM head
Banking770.4320.198–0.5940.6340.7070.7080.692
MASSIVE0.6560.592–0.7160.7340.7670.7360.730
SST-50.4280.3040.2920.4900.4540.4720.4340.446
TweetEval emotion0.7580.6780.7340.7900.7560.7290.7620.755
TweetEval hate0.6120.5880.6760.4600.5260.5820.5440.554
Rotten Tomatoes0.8720.8720.8540.8060.9000.9030.9120.903
SMS Spam0.7960.6340.2360.8440.8620.8890.7200.819
Counterfactual0.8240.8270.8270.8330.8240.8510.5070.677
GoEmotions0.2320.162–0.2120.2360.3270.3020.324
ARC-Challenge0.7340.7300.7160.7500.7860.8250.8140.822
MMLU0.5840.5560.5480.5900.6120.6310.6320.650
StrategyQA0.5820.6080.5120.5420.6420.6200.6540.637
Mean0.6260.5620.5990.6360.6640.692 ± 0.0040.6440.667 ± 0.007
Table 14. Accuracy in place of macro-F1 for every held-out task, Qwen3-1.7B-Base, 500-example subsets, columns as in Table 13.
TaskrawtokenbytecharPMI+BCsymbolECE (raw)
Banking770.3760.3340.2580.2500.0140.436–0.073
MASSIVE0.4020.4100.3960.4000.0600.494–0.143
SST-50.3320.2680.2680.2420.1240.3540.2880.368
TweetEval emotion0.6320.6320.6000.5960.6360.6920.5380.163
TweetEval hate0.4240.5480.5000.5480.5880.6020.5480.397
Rotten Tomatoes0.6020.6020.6020.6020.5540.8760.6040.246
SMS Spam0.1240.1240.1220.1220.1220.5140.3220.725
Counterfactual0.6750.8270.8270.8270.8270.4510.6630.085
GoEmotions0.1420.1520.1060.0980.1660.200–0.369
ARC-Challenge†0.5340.5160.5160.5140.386–0.3900.137
MMLU†0.4140.4060.3860.3840.340–0.3360.189
StrategyQA0.5120.5120.4940.4960.5020.5120.4980.227
Mean0.4310.4440.4230.4230.3600.5130.4650.260
Table 15. Zero-shot held-out accuracy, Qwen3-0.6B-Base, columns as in Table 2.
TaskZero-shotLM headScalar headTrained on the taskBest prompted frontier (model)Best published zero-shot (size)Best supervised (size)
Banking770.3340.6110.661–0.868 (Claude Fable 5.1)0.715 (8B)–
MASSIVE0.4110.6860.685–0.890 (GPT-6 Astra)0.673 (8B)–
SST-50.1420.2810.332–0.544 (Claude Fable 5.1)––
TweetEval emotion0.5680.6520.672–0.802 (gpt-oss-120b)0.728 (n.d.)0.810 (0.12B)
TweetEval hate0.3140.4840.531–0.767 (Kimi K3)0.708 (20B)0.581 (0.12B)
Rotten Tomatoes0.5220.8730.884–0.952 (Claude Fable 5.1)0.903 (8B)–
SMS Spam0.1110.3430.455–0.986 (Claude Fable 5.1)––
Counterfactual0.5150.4210.579–0.847 (gpt-oss-120b)––
GoEmotions0.1120.2640.257–0.362 (gpt-oss-120b)––
ARC-Challenge†0.5340.6660.672–0.976 (Kimi K3)0.786 (3.2B)–
MMLU†0.4140.5100.528–0.938 (GPT-6 Astra)0.626 (1.7B)–
StrategyQA0.4120.5340.560–0.923 (Claude Fable 5.1)––
Mean0.3660.5270.568
Table 16. Held-out performance after training, Qwen3-0.6B-Base, 500-example subsets, columns as in Table 3.
TaskZero-shotLM headScalar headTrained on the taskBest prompted frontier (model)Best published zero-shot (size)Best supervised (size)
Banking770.5780.7220.756–0.868 (Claude Fable 5.1)0.715 (8B)–
MASSIVE0.7280.7490.772–0.890 (GPT-6 Astra)0.673 (8B)–
SST-50.3930.4460.508–0.544 (Claude Fable 5.1)––
TweetEval emotion0.7100.6350.672–0.802 (gpt-oss-120b)0.728 (n.d.)0.810 (0.12B)
TweetEval hate0.6240.4530.566–0.767 (Kimi K3)0.708 (20B)0.581 (0.12B)
Rotten Tomatoes0.8870.9080.915–0.952 (Claude Fable 5.1)0.903 (8B)–
SMS Spam0.5100.5920.766–0.986 (Claude Fable 5.1)––
Counterfactual0.5280.6160.764–0.847 (gpt-oss-120b)––
GoEmotions0.1990.2710.282–0.362 (gpt-oss-120b)––
ARC-Challenge†0.8360.9060.885–0.976 (Kimi K3)0.786 (3.2B)–
MMLU†0.6580.7000.702–0.938 (GPT-6 Astra)0.626 (1.7B)–
StrategyQA0.5230.7080.710–0.923 (Claude Fable 5.1)––
Mean0.5980.6420.691
Table 17. Held-out performance after training, Qwen3-4B-Base, 500-example subsets, columns as in Table 3.
TaskZero-shotLM headScalar headTrained on the taskBest prompted frontier (model)Best published zero-shot (size)Best supervised (size)
Banking770.6100.7230.755–0.868 (Claude Fable 5.1)0.715 (8B)–
MASSIVE0.8020.8070.809–0.890 (GPT-6 Astra)0.673 (8B)–
SST-50.2570.5030.526–0.544 (Claude Fable 5.1)––
TweetEval emotion0.7340.6840.661–0.802 (gpt-oss-120b)0.728 (n.d.)0.810 (0.12B)
TweetEval hate0.5930.5360.563–0.767 (Kimi K3)0.708 (20B)0.581 (0.12B)
Rotten Tomatoes0.8910.9300.927–0.952 (Claude Fable 5.1)0.903 (8B)–
SMS Spam0.6920.7490.879–0.986 (Claude Fable 5.1)––
Counterfactual0.4700.8050.796–0.847 (gpt-oss-120b)––
GoEmotions0.2250.2990.320–0.362 (gpt-oss-120b)––
ARC-Challenge†0.8840.9300.927–0.976 (Kimi K3)0.786 (3.2B)–
MMLU†0.6800.7560.760–0.938 (GPT-6 Astra)0.626 (1.7B)–
StrategyQA0.6850.7680.726–0.923 (Claude Fable 5.1)––
Mean0.6270.7070.721
Table 18. Held-out performance after training, Qwen3-8B-Base, 500-example subsets, columns as in Table 3.
TaskZero-shotLM headScalar headTrained on the taskBest prompted frontier (model)Best published zero-shot (size)Best supervised (size)
Banking770.5500.8080.810–0.868 (Claude Fable 5.1)0.715 (8B)–
MASSIVE0.5810.7640.811–0.890 (GPT-6 Astra)0.673 (8B)–
SST-50.1930.3320.429–0.544 (Claude Fable 5.1)––
TweetEval emotion0.4620.6500.720–0.802 (gpt-oss-120b)0.728 (n.d.)0.810 (0.12B)
TweetEval hate0.6310.5760.547–0.767 (Kimi K3)0.708 (20B)0.581 (0.12B)
Rotten Tomatoes0.6360.9020.908–0.952 (Claude Fable 5.1)0.903 (8B)–
SMS Spam0.6720.7850.765–0.986 (Claude Fable 5.1)––
Counterfactual0.5000.7840.792–0.847 (gpt-oss-120b)––
GoEmotions0.0920.2650.314–0.362 (gpt-oss-120b)––
ARC-Challenge†0.8500.9000.894–0.976 (Kimi K3)0.786 (3.2B)–
MMLU†0.6140.7200.704–0.938 (GPT-6 Astra)0.626 (1.7B)–
StrategyQA0.3640.6950.675–0.923 (Claude Fable 5.1)––
Mean0.5120.6820.697
Table 19. Held-out performance after training, gpt-oss-20b (post-trained mixture of experts), 500-example subsets, columns as in Table 3.
ModelaccECEBrierNLLAUROCacc@50% cov.
zero-shot raw0.4310.2600.7541.6830.6800.550
zero-shot raw + TS0.4310.0960.6121.3390.6700.551
set-CE, LM head0.5560.1690.5721.2130.6680.657
set-CE, LM head + TS0.5560.0670.5081.0350.6630.653
gold-only SFT0.5790.1530.5631.4190.6930.707
set-CE, scalar head0.6340.1130.4961.0170.6760.736
Table 20. Calibration, 0.6B, pooled over the twelve held-out tasks (5,835 predictions). The zero-shot plateau between 0.5 and 0.9 confidence is the base model confidently wrong on the imbalanced binary tasks (SMS Spam: confidence 0.84, accuracy 0.12; TweetEval hate: 0.82, 0.42); training removes it. Per task, the trained model remains over-confident on TweetEval hate, MMLU and StrategyQA and under-confident on SMS Spam and Amazon Counterfactual (Figure 9).
5,835 predictions
Reliability
Selective prediction
0.504Zero-shot, answers when confidence ≥ 0.710.783Trained (scalar head), answers when confidence ≥ 0.72
Figure 9. Calibration on held-out tasks, Qwen3-0.6B-Base, zero-shot vs. the trained scalar head, pooled over the twelve tasks as in Figure 4: reliability diagram (left) and accuracy on the most confident fraction (right). The zero-shot plateau between 0.5 and 0.9 confidence is the base model confidently wrong on the imbalanced binary tasks; training removes it (Table 20).

C Supporting tables

These tables supplement the main text: related work, compute per decision, ordinal prediction, prompting failures, adaptation cost, latency, and the label-set and data sweeps.

MethodTrainingReadoutOne passCalibrationUnseen label setsOpen
Log-likelihood scoring [7]noneoption likelihoodno, one seq./optionpoor (Table 2)yesyes
First-token symbol [82]noneP(letter)yesletter biasyes, ≤ 26yes
T0 / FLAN [59, 72]mixture, onceoption likelihoodno, one seq./optionnot measuredyesyes
Universal Discriminator [76]mixture, onceencoder scalar headone seq./optionnot measuredyesyes
Sood et al. [63]per taskscalar headno, one seq./optionnot measurednoyes
GLiClass [64]onceencoderyesnot statedyesyes
NLI / embedding / rerankeronce (NLI or retrieval pairs)pair scoreone pass per labelnot statedyesyes
Jev [67]undiscloseddistributionyesclaimedyesno
Laya [13]per domain (RLCD)encoder mask-token scoreryesECE after temperaturenear chance (reported)yes
Ours, LM headmixture, oncepacked likelihoodyesproper scores, rankingyes, 12 tasksyes
Ours, scalar headmixture, oncepacked w⊤hyesproper scores, ECEyes, 12 tasksyes
Table 21. Positioning. “Unseen label sets”: evaluated on label sets not trained on. “One pass”: all options scored in one forward pass.
SystemParamsOne decisionBanking77 (77 labels)SST-5 (5 labels)
Ours, Qwen3-1.7B-Base1.7Bpacked scoring: prompt + all options, one pass2.40.2
Qwen3-Embedding-8B8Bbi-encoder: the text, one pass; labels cached0.20.4
Qwen3-Reranker-8B8Bcross-encoder: text + label, one pass per label20.31.9
Flan-UL220Bgenerative: the prompt, then decode the label14.72.1
Llama-3.2-3B-Instruct3.2Bgenerative: the prompt, then decode the label2.40.3
ChatGPT / GPT-4n.d.generative: the prompt, then decode the label––
Table 22. Size and estimated compute of one decision, ours vs. the systems behind the best published zero-shot numbers. Entries are TFLOPs per decision, ≈ 2 × parameters × tokens processed; token counts are means over our 500 test prompts (Banking77: 365-token prompt, 334 option tokens, 13-token text, 3.5-token labels; SST-5: 51, 12, 23, 1.4). The reranker estimate omits its fixed prompt template, so it is a lower bound. n.d.: size not disclosed.
ModelAccuracyMAE (levels)Off by ≥ 2 levelsSpearman ρ
1.7B zero-shot0.4280.666 / 0.6420.074 / 0.0540.751 / 0.813
1.7B LM head0.4460.623 / 0.6080.064 / 0.0560.765 / 0.829
1.7B scalar head0.4720.636 / 0.5490.087 / 0.0550.759 / 0.831
8B zero-shot0.2860.982 / 0.8620.230 / 0.1920.774 / 0.847
8B scalar head0.5350.528 / 0.4840.059 / 0.0340.819 / 0.862
Table 23. SST-5 (five ordered levels): the argmax of the option distribution against its expected level, rounded (each cell: argmax / expected). Mean absolute error in levels, share of predictions off by two or more levels, and Spearman correlation with the gold level. The expected level is the better ordinal prediction for every model, by the widest margin for the trained 1.7B scalar head.
ReadoutMean primaryAccuracyBrierNLLECE
single-token0.6260.6790.4471.0450.116
pack-readout, scalar head0.6520.6960.4140.8750.097
pack-readout, LM head0.6170.6500.4661.0360.120
Tasknsingle-tokenpack-readout, scalar headpack-readout, LM head
Banking775000.6310.6660.649
MASSIVE5000.6910.7320.728
SST-55000.4530.4880.388
TweetEval emotion5000.6920.7060.688
TweetEval hate5000.5830.5540.528
Rotten Tomatoes5000.9020.9120.898
SMS Spam5000.6640.7160.655
Counterfactual3350.5940.6500.497
GoEmotions5000.2470.2650.291
ARC-Challenge†5000.7940.8240.808
MMLU†5000.5980.6420.626
StrategyQA5000.6600.6680.654
Table 24. A trained single-token readout against pack-readout, Qwen3-1.7B-Base, seed 0, on the same 500 test examples per task (the first 500 of each shuffled test set; pack-readout: the seed-0 full-test adapters of Table 3). Top: means over the twelve held-out tasks; bottom: primary score per task. Single-token: the option letters’ logits at one answer position, trained with the same LoRA, objective and mixture.
TaskReplyGold option
Banking77“reverted top up”top up reverted
Banking77“freeze my account”compromised card
MASSIVE“lists addorcreate”calendar set
MASSIVE“recommendations movies”recommendation movies
TweetEval emotion“angry”anger
TweetEval emotion“sad”sadness
Rotten Tomatoes“neutral”positive
Rotten Tomatoes“more intellectually scary than dramatically involving”negative
SMS Spam“A”spam
GoEmotions“desperation”neutral
GoEmotions“redeem”neutral
ARC-Challenge“the color of the light bulb they use”the color of light bulb they use
ARC-Challenge“release of chlorofluorocarcs from aerosol cans”release of chlorofluorocarbons from aero
MMLU“18x + 15”18x+15
MMLU“recognized as breaking an important norm of behaviour”seen a temporary aberration from an othe
Table 25. Invalid replies of the prompted Qwen3-1.7B: the first two per task that named no option, with the gold label. Each is a plausible answer outside the option set, which option scoring cannot produce.
SystemAdaptation dataComputeData public
Ours, from Qwen3-1.7B-Base16,000 examples from 18 public datasets, 2.5M tokens20 min, one A100 (25 PFLOPs)yes
Ours, from Qwen3-8B-Basesame2.1 h, one DGX Spark (120 PFLOPs)yes
gte-large-en-v1.5 [37]MLM on C4; 800M web-mined pairs; 3M annotated pairsn.d. (v1: 50k pre-training steps at batch 16k on 8 A100-80GB)sources only
Qwen3-Embedding-8B [80]150M synthetic pairs; 7M labeled + 12M synthetic pairsn.d.sources only
Qwen3-Reranker-8B [80]7M labeled + 12M synthetic pairsn.d.sources only
deberta-v3-large-nli-triplet [1]885k NLI pairs, 3 epochs [34]n.d. (a similar run: 10 h, one A100)yes
Flan-T5-XXL [10]Flan 2022 collection, 1,836 tasks7.6×10¹⁹ FLOPs (76,000 PFLOPs)yes
Mistral-Nemo-Instruct [47]n.d.n.d.no
Llama-3.2-3B-Instruct [44]SFT, rejection sampling, DPO; counts n.d.460k H100 hours with pretrainingno
GPT-5, Clauden.d.n.d.no
Table 26. Adaptation data and compute beyond pretraining, where disclosed. Ours: 16,000 public examples, 20 minutes on one A100 at 1.7B.
OptionsPrompt tok.Packed (ours)KV replicationOne seq./optionConstrained decodingJSON generation
23747.591.447.3203.31,042.8
44447.389.948.2163.61,030.7
85747.390.449.2203.21,014.1
168348.292.350.8164.81,033.8
7729554.4106.9578.0167.91,042.8
25592190.8432.85,606.1217.21,184.6
Table 27. Latency per decision (median ms over 20 inputs and three random label sets per count), Qwen3-1.7B-Base, bf16, A100-40GB, labels listed in the prompt.
Training label setsHeld-out accMany-label accECEAUROC
40.638 ± 0.0140.4710.1490.718
60.663 ± 0.0200.5620.1420.724
80.655 ± 0.0250.5680.1300.731
100.669 ± 0.0250.5740.1090.733
120.667 ± 0.0040.5630.1250.730
160.672 ± 0.0110.5890.1200.738
180.693 ± 0.0030.6070.0950.745
Table 28. Held-out performance vs. number of training label sets at 16,000 example presentations and 2,000 steps, scalar head, Qwen3-1.7B-Base: mean and s.d. over three random subsets of the training tasks per count (at 18, over three training seeds).
Examples in the poolPer taskHeld-out macro-F1Many-label accECE
1,9981110.5950.5700.311
7,9924440.6230.6070.195
16,0028890.6310.5980.115
36,0002,0000.6410.6000.098
Table 29. Held-out performance vs. the size of the training pool at 2,000 steps, scalar head, Qwen3-1.7B-Base; each of the 18 tasks is capped at the per-task count, and the smallest pool is seen eight times.

D BTZSC comparison by task group

Table 30 gives the numbers behind Figure 8 (Section 5.8), overall and per task group.

ModelOverallSentimentTopicIntentEmotion
Qwen3-Reranker-8B (reranker, 8.0B)0.7220.9230.6620.7000.486
Mistral-Nemo-Instruct-2407 (LLM, first token, 12.2B)0.6700.8410.6910.4550.363
Qwen3-4B (LLM, first token, 4.0B)0.6490.8830.6380.4010.373
gte-large-en-v1.5 (embedding, 0.4B)0.6170.8490.5430.5900.372
deberta-v3-large-nli-triplet (NLI, 0.4B)0.5960.8990.5030.4470.419
Llama-3.2-3B-Instruct (LLM, first token, 3.2B)0.4300.4560.4370.4080.348
Ours, Qwen3-1.7B-Base, zero-shot (no training)0.5260.7260.4890.3520.386
Ours, 1.7B trained, LM head (mixture without BTZSC data)0.6720.9130.5980.6050.456
Ours, 1.7B trained, scalar head (same) (2 seeds)0.7030.8990.6480.6580.480
Ours, 0.6B trained, scalar head (same)0.6730.8790.6210.6130.433
Ours, 4B trained, scalar head (same) (2 seeds)0.7270.9290.6690.7210.444
Ours, 8B trained, scalar head (same) (2 seeds)0.7400.9290.6910.7310.454
Ours, 14B trained, scalar head (same; 1,000 steps, half the learning rate)0.7450.9320.6890.7600.473
Table 30. Comparison with dedicated zero-shot classifiers on the BTZSC benchmark (22 datasets; macro-F1 overall and per task group), under the benchmark’s own protocol: every system receives the text and a verbalized description of each label, and none is trained on the task. Top: the best model of each family in the leaderboard and its small LLMs, on full test sets. Bottom: ours, on a seeded 500-example subset per dataset; the trained rows use the training mixture without the five datasets that BTZSC also contains. Compared with 8B retrieval models and 4–12B LLMs, our model is within 2 points of the best system at 1.7B, exceeds it at 4B, half its size, and exceeds it by two points at 8B, the same size.

E Published numbers on the held-out tasks

Best published results we could verify (2026-09-19, corrected 2026-09-30) per held-out task, with model size and the metric each is reported in. Zero-shot means no training on the dataset. Sources: BTZSC results (Aarab [1], full test sets); Zhang et al. 2023 (arXiv 2305.15005; 500 sampled test items); TimeLMs (arXiv 2202.03829); Casanueva et al. 2020 (arXiv 2003.04807); FitzGerald et al. 2022 (arXiv 2204.08582); Emo Pillars (arXiv 2504.16856); the Llama 3.2 and Qwen3 model reports; for SMS Spam, Wang et al. [71] (150 sampled test messages) and Houlsby et al. [25] (full fine-tuning; their 558-message test split); for Counterfactual, Lee et al. [35] (a logistic probe on frozen embeddings; MTEB’s English test set); for GoEmotions, Niu and Mower Provost [48]; for SST-5, Heinsen [22]; for StrategyQA, Şenol et al. [61] (version 4; 250 sampled items of the test split we use). The published supervised StrategyQA results use other splits. GoEmotions numbers are multi-label and not comparable with our single-label subset.

TaskBest zero-shot (model, metric)Best supervised (model, metric)
Banking770.715 Qwen3-Embedding (8B) (macro-F1)0.937 BERT-large (0.3B) (accuracy)
MASSIVE0.673 Qwen3-Reranker (8B) (macro-F1)0.890 mT5-base (0.6B) (accuracy)
SST-50.570 Flan-UL2 (20B) (accuracy)0.598 routing head on frozen RoBERTa-large (0.4B) (accuracy)
TweetEval emotion0.728 ChatGPT (macro-F1)0.810 TimeLM (0.1B) (macro-F1)
TweetEval hate0.708 Flan-UL2 (20B) (macro-F1)0.581 TimeLM (0.1B) (macro-F1)
Rotten Tomatoes0.903 Qwen3-Reranker (8B) (macro-F1)0.900 T5-large (0.8B) (accuracy)
SMS Spam0.973 GPT-4 (accuracy)0.993 BERT-base (0.1B) (accuracy)
Counterfactualnone found0.951 NV-Embed-v1 (8B), probe on 32 examples per label (accuracy)
GoEmotions0.319 GPT-4 (macro-F1, multi-label)0.550 RoBERTa-large (0.4B) (macro-F1, multi-label)
ARC-Challenge0.786 Llama-3.2-3B-Instruct (accuracy)none found
MMLU0.626 Qwen3-1.7B-Base, 5-shot (accuracy)none found
StrategyQA0.524 Phi-2 (2.7B), free-form answer (accuracy)none found
Table 31. Published numbers on the held-out tasks.

F Prompting baselines: protocol

Every prompted model receives the scorer’s prompt without its cue line, followed by “Reply with one of the listed answers, exactly as written, and nothing else.”, as a single user message in the model’s chat format; thinking is disabled where the template allows it. Local models sample free replies with the settings their authors recommend for the non-thinking mode (temperature 0.7, top-p 0.8, top-k 20; seed 0) and decode constrained replies greedily. Hosted requests set temperature 0; the gateway does not report whether a provider applies it to a reasoning model. A reply is matched to an option by exact match after normalization (case, surrounding quotes and punctuation, whitespace), else by the longest option contained in the reply as whole words, else by the single option the reply is part of; otherwise it is invalid and counts as wrong. Every reply is stored with the results. Local models (Qwen3-1.7B, 4B, 8B, the post-trained releases) run on a Lambda A100-40GB one decision at a time; free replies end at the end-of-turn token or at a 32,768-token cap, which two MMLU replies of the 1.7B model reached. Hosted models run through the Vercel AI Gateway. Muse Spark 1.3 and GLM-5.3 have no request-level output-token cap; the other hosted runs used a 1,024-token budget including hidden reasoning. Requests are restricted to the providers listed in Table 32. Serving providers are recorded per task, and per reply for Muse Spark 1.3 and GLM-5.3. GPT-5, GPT-6 Astra and Muse Spark 1.3 run with minimal reasoning effort, GLM-5.3 with low effort; the other models use their default settings, which for Claude Sonnet 5 and Claude Fable 5.1 means adaptive thinking on and for Claude Sonnet 4.6 no thinking (the labels of Table 7 state the effective setting). Table 32 lists the model ids, the providers that served the requests and dates; Appendix I lists available technical reports.

Model idParams (B)Served bySettingsDate
alibaba/qwen-3-235b235deepinfra, novitadefault2026-09-19
anthropic/claude-fable-5.1n.d.anthropicdefault2026-09-20
anthropic/claude-sonnet-4.6n.d.anthropicdefault2026-09-19
anthropic/claude-sonnet-5n.d.anthropicdefault2026-09-20
deepseek/deepseek-v3.2671deepinfradefault2026-09-19
meta/muse-spark-1.3n.d.metareasoning_effort=minimal2026-09-20
moonshotai/kimi-k32,800alibaba, basetendefault2026-09-20
openai/gpt-5n.d.openaireasoning_effort=minimal2026-09-19
openai/gpt-6-astran.d.openaireasoning_effort=minimal2026-09-20
openai/gpt-oss-120b117basetendefault2026-09-19
zai/glm-5.3753deepinfrareasoning_effort=low2026-09-20
Table 32. Hosted models of Table 7: gateway model id, total parameters as stated by the model’s developer (n.d.: not disclosed), serving providers, request settings and date of the run.

G Reproduction

The released repository contains the model implementation, training-data loaders, and training and evaluation entry points. scripts/train.py trains the LoRA adapter and readout; scripts/evaluate.py evaluates a base model or a saved adapter on the specified tasks; scripts/prompting_baseline.py, scripts/prompting_baseline_api.py, scripts/bench_latency.py and scripts/btzsc_eval.py run the prompting, latency and BTZSC comparisons of Sections 5.5, 5.6 and 5.8. Section 4 gives the training settings. The results archive (Data availability) holds the evaluation reports behind every reported number. The trained adapters (LoRA weights and, for the scalar readout, the head vector) of the main runs at every size, the other base models, the LM-head readout, the BTZSC-clean mixture and the additional seeds are released on Hugging Face, named by base model, under CC BY-NC 4.0 because the training mixture includes SciQ (CC BY-NC 3.0).

One training run diverged: the seed-2 full-test run of the 1.7B scalar head on the Lambda A100s (its loss rose from about 0.5 to 1.3 after step 500 and stayed there, and its held-out scores were near chance). It was rerun with the same seed; only the rerun is reported. A concurrent seed-3 run reached a held-out mean of 0.649 and is not used. No other run diverged. At 4B, the scalar-head seed 0 (4b-set-scalar-s0) was an early 1,000-step run and seed 1 the 2,000-step recipe; the two agree within 0.005 on the held-out mean, and the 1,000-step control of Section 5.7 shows that the step count does not change the result at 4B.

H Latency at other sizes

The sweep of Table 27 at 0.6B and 4B, same protocol and GPU (also in Figure 6). At 4B, KV replication does not fit in 40 GB at 255 options.

OptionsPrompt tok.Packed (ours)KV replicationOne seq./optionConstrained decodingJSON generation
23744.785.445.7193.9798.3
44445.387.346.3161.7778.8
85745.287.246.5195.6860.4
168346.187.748.4158.1850.6
7729552.999.5340.8162.9931.9
25592172.6406.63,291.1173.0969.7
Table 33. Latency per decision (median ms), Qwen3-0.6B-Base, A100-40GB, columns as in Table 27.
OptionsPrompt tok.Packed (ours)KV replicationOne seq./optionConstrained decodingJSON generation
23759.8116.661.7260.71,351.5
44459.9114.861.2212.11,355.3
85761.2116.462.6209.31,364.6
168362.4119.689.7210.91,385.4
7729569.2163.91,237.8219.41,373.7
255921159.0OOM12,349.9234.51,433.0
Table 34. Latency per decision (median ms), Qwen3-4B-Base, A100-40GB, columns as in Table 27; OOM: does not fit in 40 GB.

I Sources of datasets and models

Training mixture: AG News, DBpedia-14, Yahoo Answers and Yelp-5 [79]; IMDB [42]; TweetEval sentiment, offensive and irony [4]; Emotion [60]; CLINC-150 [33]; SciQ [74]; OpenBookQA [45]; CommonsenseQA [65]; QASC [29]; ARC-Easy [12]; HellaSwag [78]; WinoGrande [58]; BoolQ [11]. Held out: Banking77 [8]; MASSIVE [18]; SST-5 [62]; Rotten Tomatoes [53]; TweetEval emotion and hate [4]; GoEmotions [17]; SMS Spam [3]; Amazon Counterfactual [50]; ARC-Challenge [12]; MMLU [23]; StrategyQA [19]. Base models: Qwen3 [55], Qwen2.5 [54], OLMo 2 [49], SmolLM2 [2]. Hosted open-weight models with a published report: gpt-oss [51], DeepSeek-V3 [15], the line that V3.2 continues, Kimi K2 [30], the predecessor of K3, and Qwen3-235B [55]. GLM-5.3 [77] and Muse Spark 1.3 [43] are documented by their developers; the closed models are identified by model id and date in Table 32. BTZSC reference systems of Table 30: Qwen3-Reranker-8B [80], gte-large-en-v1.5 [37], Llama-3.2-3B-Instruct [44] and Mistral-Nemo-Instruct [47].