MIT's DAAAM Gives Robots a Searchable Long-Term Memory of What They Saw, Where, and When
A new MIT framework lets robots store language descriptions of objects in a spatial map and answer natural-language queries, improving accuracy by 21 to 53 percent over prior methods.
Overview
Researchers at MIT have built a system that gives robots a long-term, searchable memory of the objects they encounter, letting a person retrieve them later with ordinary language. The framework, called DAAAM, creates a spatiotemporal memory that improved accuracy by 21 to 53 percent over competing methods depending on the type of question asked, according to MIT News. The work was presented at the Conference on Computer Vision and Pattern Recognition, with the paper available as a preprint, as reported by Digital Trends.
What We Know
DAAAM stands for “Describe Anything, Anywhere, Anytime, at Any Moment,” according to MIT News. The system combines computer vision with three-dimensional mapping so that, as a robot moves through an environment, it attaches detailed language descriptions to objects it sees and stores them in a spatial map, as described by Digital Trends.
Rather than knowing only that an object sits at a particular coordinate, the robot records what it is and when it was seen, enabling language-based retrieval. A person could ask something like, “Where did I leave my wallet?” or “Go grab the component we started assembling last night,” and the robot could search its memory for the right object and location, according to Digital Trends. To answer such queries, the framework pairs its 3D map representations and object descriptions with a large language model that has tool-calling capabilities for retrieving the relevant information, according to MIT News.
Under the hood, the preprint describes building a hierarchical four-dimensional scene graph that serves as a globally spatially and temporally consistent memory representation, queried by a tool-calling agent for inference and reasoning, according to the arXiv preprint. In the authors’ evaluations, DAAAM improved OC-NaVQA question accuracy by 53.6 percent, reduced position errors by 21.9 percent, reduced temporal errors by 21.6 percent, and improved SG3D task grounding accuracy by 27.8 percent, according to the arXiv preprint. The system maintains real-time performance, the preprint states, and MIT News reports the framework can run fast enough for a mobile robot to use in real time, per Digital Trends.
The paper lists Nicolas Gorlo, a graduate student at MIT, as lead author, alongside Luca Carlone, an associate professor in MIT’s Department of Aeronautics and Astronautics and director of the MIT SPARK Laboratory, and Lukas Schmid, a former MIT research scientist who is now a professor at the University of Technology Nuremberg, according to MIT News. The work was funded in part by the U.S. Army Research Laboratory and the Office of Naval Research, MIT News reports.
“If we want robots to work side-by-side with humans and interact better with humans, they must speak the same language,” Carlone said, according to MIT News. The researchers point to factory robotics, augmented reality systems for maintenance workers, and wayfinding assistance as potential applications, per MIT News.
What We Don’t Know
The framework is a research result rather than a shipping product. As Digital Trends puts it, “This is not a feature coming to your robot vacuum next week,” according to Digital Trends. The reported accuracy gains are measured against specific benchmarks, and how the approach generalizes to messy, changing real-world spaces over long deployments remains to be demonstrated. The paper remains a preprint, and no timeline for deployment in commercial robots has been announced.