MIT's Recursive Language Models Let an LLM Read Its Own Prompt as Code, Beating Frontier Long-Context Scaffolds

Overview

Three researchers at MIT’s Computer Science and Artificial Intelligence Laboratory have proposed an inference method that lets a language model handle prompts far larger than its context window by treating the prompt as something to inspect with code rather than something to read all at once. The technique, called Recursive Language Models (RLMs), is described in a paper titled “Recursive Language Models” by Alex L. Zhang, Tim Kraska, and Omar Khattab of MIT CSAIL, first posted to arXiv on December 31, 2025 and last revised on May 11, 2026.

The paper defines an RLM as “a general inference paradigm that treats long prompts as part of an external environment and allows the LLM to programmatically examine, decompose, and recursively call itself over snippets of the prompt,” according to the paper’s full text on arXiv. Instead of stuffing an entire document into a single forward pass, the model writes code to look at the prompt piece by piece and spawns fresh copies of itself to process the pieces.

What We Know

The mechanism. Rather than receiving the long prompt directly, the model is equipped with a Python REPL — an interactive programming environment — in which the prompt is held as a variable. The arXiv paper states that the authors “equip an LLM with a Python REPL” and that “the RLM initializes a persistent REPL programming environment.” The official reference implementation describes the same design: RLMs “offload the context as a variable in a REPL environment that the LM can interact with and launch sub-LM calls inside of,” according to the project’s GitHub repository. The model can grep through, partition, and read slices of that variable, and call recursive sub-instances of itself to do work on individual snippets.

A drop-in interface. The authors frame the approach as a replacement for an ordinary model call rather than a new model. The reference library notes that RLMs “replace the canonical llm.completion(prompt, model) call with a rlm.completion(prompt, model) call, acting as a ‘language model,’” per the GitHub repository. The implementation is released as open source under an MIT license, the repository shows, described there as a “General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.”

The headline results. On shorter prompts where the comparison is apples-to-apples, the method outperformed established long-context and coding scaffolds built on the same base model. The arXiv paper reports that RLMs “dramatically outperform the quality of vanilla frontier LLMs and common long-context and coding scaffolds (e.g., on GPT-5 by a median across the evaluated benchmarks of 26% against compaction, 130% against CodeAct with sub-calls, and 13% against Claude Code) across four diverse long-context tasks while having comparable cost.”

The team also trained a small model specifically around the recursive loop. According to the arXiv paper, “Our model, RLM-Qwen3-8B, outperforms the underlying Qwen3-8B model by a median of 28.3% across the four evaluation tasks,” and the same paper notes the recursive variant “even approaches the quality of vanilla GPT-5 on three long-context tasks.”

Beyond the context window. The central claim is that the recursion sidesteps the hard ceiling on prompt length. The arXiv paper states that “RLMs can successfully process inputs more than an order of magnitude beyond model context window limits.” The abstract as published on the paper’s Hugging Face page puts the figure higher, saying RLMs “successfully handle inputs up to two orders of magnitude beyond model context windows” while keeping “comparable (or cheaper) cost per query.” Both versions evaluate the method across four long-context tasks, including the OOLONG long-context question-answering benchmark named in the reference repository.

What We Don’t Know

The paper is a preprint and, as of its latest arXiv revision, has not completed formal peer review. The two published versions of the abstract state the out-of-context reach differently — “more than an order of magnitude” in the arXiv full text versus “two orders of magnitude” on the Hugging Face page — so the precise upper bound depends on which figure holds up under scrutiny. The reported gains are medians across a specific set of four tasks; how the method generalizes to other workloads, model families, or production latency budgets is not established by the paper alone.

Analysis

RLMs belong to a broader 2026 shift away from simply enlarging context windows and toward giving models tools to manage context themselves. The motivation is “context rot,” the named phenomenon the arXiv paper invokes in which a model’s quality degrades as its prompt grows longer. The conventional fixes — retrieval, summarization, or compaction — discard or compress information before the model sees it. The recursive approach keeps the full input intact as a variable and lets the model decide, programmatically, which parts to actually read.

The design’s most consequential property may be its packaging. Because rlm.completion is meant to be a drop-in for llm.completion, per the GitHub repository, the recursion is invisible to the calling application: a system that already issues chat-completion requests could route them through an RLM without restructuring its prompts. That, combined with an MIT-licensed reference implementation, lowers the barrier to testing whether “comparable cost” holds outside the paper’s four benchmarks — the question that will ultimately decide whether recursion becomes a standard part of the long-context toolkit.