Small Language Models Learn from Large Ones via Retrieved Instructions

👁 118 reads

A new paper from researchers at the University of Michigan shows how small language models can learn from large ones by retrieving ready-made instructions on the fly, without any fine-tuning required.

The work comes from Kenan Alkiek, David Jurgens, and Vinod Vydiswaran and was published on arXiv in October 2025 (latest version from September 2026). The authors tackle a problem anyone who has used small language models on end devices knows from practice: a small model simply does not fit enough knowledge in its parameters to handle difficult, specialized questions.

The Problem with Small Language Models

The authors describe three typical approaches to this problem and point out their limitations. Fine-tuning encodes knowledge in the model’s parameters, but you need to repeat it for each model and each domain separately. Providing retrieved text passages in the prompt leaves the model with the task of independently finding the right fact and applying it. Few-shot prompting, as the results in the paper show, actually worsens results under certain conditions - on the MedQA benchmark, few-shot examples provided by the teacher model lowered the small model’s accuracy.

How Instruction Retrieval Works

The proposed solution is called instruction retrieval. A large teacher model analyzes problems from a given domain, groups them into clusters, and creates one instruction for each cluster. Such an instruction contains three elements: domain knowledge needed to solve problems from that cluster, a procedure for approaching this type of problem, and a list of common mistakes made when solving it. You build a collection of such instructions once for a given domain. Then, when the system is running, the small model - without any fine-tuning, in its frozen version - retrieves instructions closest in content to the asked question and applies them.

An important property of this approach: after building the instruction corpus, no further access to the teacher model is needed. You can reuse the corpus multiple times.

Benchmark Results

The authors tested the method on benchmarks from medicine, law, and mathematics. The instruction corpus improved results compared to zero-shot on each of these tasks. Compared to few-shot prompting, self-consistency, and other methods based on text retrieval - the advantage of the instruction corpus appears in medicine and law (for mathematics, the authors did not formulate the same general claim in the abstract).

The most concrete number: on the MedQA benchmark, the method raises average accuracy by 10.6 percentage points. For comparison, retrieved textbook passages raise it by 3.9 points, while few-shot from the same teacher model actually lowers it.

Which Component Fixes Which Error Type

The authors conducted an error analysis that shows different instruction components help in different situations. Domain background knowledge alone is the element that improves questions the small model always answers incorrectly - that is, where it lacked fundamental knowledge. The procedure and list of common mistakes, on the other hand, help with questions where the model wavers between options - that is, where knowledge is partial but the model does not know how to apply it properly.

What This Means

In our view, the most interesting aspect of this work is not the numerical result itself, but the separation of functions: one instruction component fixes lack of knowledge, another fixes reasoning errors. This is a rarely seen level of detail in analysis. Practically speaking, the method is attractive for scenarios where a small model must run locally - on a phone, medical device, or in an environment without connectivity - and fine-tuning is too expensive. The one-time cost of building the corpus through a large model exists, but the authors treat it as acceptable. A limitation the paper does not explicitly discuss is how well the method scales to new domains and whether the quality of instructions generated by the teacher is always sufficient - this is not described in the available abstract.

Source: arXiv cs.AI: Instruction Retrieval at Inference Time for Small Language Models (document from 2026-10-01). This discussion is based on the full text of the document, not reports from other media.

// read moreLLM: What Is a Large Language Model and How Does It Work?GuidesCan You Train a Language Model on Music and Cellular Automata?BenchmarksDeveloper Matt Webb Used ChatGPT to Learn Quaternion MathsAI Tools

Kinetyra