MIT and Harvard Teach Language Models to Ask Better Questions, Lifting a Small Model's Battleship Win Rate From 8% to 82%
An ICLR paper from MIT CSAIL and Harvard shows Monte Carlo inference helps Llama 4 Scout outpace GPT-5 at a Battleship test bed for around 1% of its cost.
Overview
Researchers at MIT and Harvard have shown that giving language models a structured way to reason about what they don’t yet know can dramatically improve their ability to ask useful questions. In a study presented as an oral presentation at the International Conference on Learning Representations (ICLR) in April, the team used a collaborative version of the board game Battleship as a test bed and found that a small model equipped with their inference method could outpace a frontier model at a fraction of the cost, according to MIT News.
What We Know
The work comes out of the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and Harvard University’s School of Engineering and Applied Sciences, as reported by MIT News. Lead author Gabriel Grand, an MIT PhD student and CSAIL researcher, framed the central question the paper investigates. “Today’s language models are primarily optimized to answer complex queries, but it’s less clear whether they learn to ask good questions for themselves,” he told MIT News. “Our work shows that asking informative questions depends on the ability to predict and simulate the world.”
To study that, the team built what they call a “Collaborative Battleship” game in which one participant is a “captain” who inquires about where hidden ships are, while their teammate plays the “spotter” by responding to those questions in real time, as described by Tech Xplore. The researchers gave models a Monte Carlo inference strategy in which, as MIT News describes it, the language models reason about potential guesses as individual particles.
The effect on a small model was large. The relatively small Llama 4 Scout initially beat humans only 8 percent of the time, but with refinements to its inference strategy it reached a Battleship win rate of 82 percent versus humans, according to both MIT News and Tech Xplore. The researchers report that the enhanced small model could outpace the frontier model GPT-5 while operating at around 1 percent of its cost, per MIT News.
The gains were not limited to asking questions. The models saw an accuracy boost of 15 percent on average when they began converting questions into code that explicitly tells them how to verify their answers, according to MIT News. On that answering task, the lightweight GPT-4o-mini saw a nearly 30 percent performance bump, and the larger Claude 4 Opus jumped about eight points, as reported by Tech Xplore.
The team also tested the approach on a second game, Guess Who?. There, Llama 4 Scout improved from a 30 percent success rate to over 72 percent, while GPT-4o leapt from 62 percent to 90 percent, according to MIT News.
Grand and co-author Valerio Pepe, an OpenAI researcher and recent Harvard graduate, wrote the paper with two CSAIL principal investigators, MIT Associate Professor Jacob Andreas and MIT Professor Joshua Tenenbaum, as reported by MIT News.
What We Don’t Know
The method narrows but does not close the gap with skilled humans. “GPT-5 can beat your average ‘Battleship’ player, and gets a hair better with our methods,” Pepe told MIT News. “However, expert players are still hard to beat for all models, unlike in chess, where even top players don’t succeed against AI systems.”
The results come from controlled board-game environments, and the paper does not establish how directly the same inference strategies transfer to open-ended real-world agent tasks. Andreas pointed to that broader potential as the part he finds most compelling. “What I find most exciting about this work is that it opens up the possibility of using these techniques to generate better solutions in the first place, by improving LMs’ exploration and information gathering capabilities,” he told MIT News.
Analysis
The study lands amid a broader argument over whether progress in AI agents depends on ever-larger models. By showing that a small model paired with a deliberate information-gathering procedure can match or beat a frontier system on these tasks at roughly 1 percent of the cost, the paper offers evidence that inference-time reasoning can substitute for raw scale on problems that hinge on asking the right question, per MIT News.
Robert Hawkins, a Stanford University assistant professor of linguistics who was not involved in the work, framed its significance in terms of collaboration rather than raw computation. The work “makes a compelling case that the real bottleneck for AI agents isn’t just the calculation of optimal questions, but the pragmatic reasoning needed to make the most of their answers,” he told MIT News.