Ai2 Open-Sources MolmoMotion, a Model That Forecasts 3D Object Motion From Language Instructions, With a 1.16-Million-Video Dataset
The Allen Institute for AI released MolmoMotion, an open model that predicts an object's future 3D trajectory from a video, marked points, and a language instruction, plus a 1.16-million-video training dataset and a new benchmark.
Overview
The Allen Institute for AI (Ai2) has released MolmoMotion, an open model for predicting how objects will move in three dimensions. Given a video frame, 3D points marked on an object, and written instructions describing the intended action, MolmoMotion predicts where those points will move over the next few seconds in 3D space, according to Ai2. The release, published June 17, 2026, bundles the model weights together with a 1.16-million-video training dataset, a new evaluation benchmark, and code hosted on Hugging Face and GitHub.
What We Know
The accompanying paper, “MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction,” formalizes the task as goal-conditioned 3D point motion forecasting: given a short visual history, a set of 3D query points on an object of interest, and a language description of the intended goal, the model predicts the future 3D trajectory of each point. The authors argue that 3D points in world coordinates provide a general representation that is class-agnostic, view-stable, compact, and directly useful for downstream tasks. All future coordinates are expressed in a world coordinate frame anchored at the camera, in metric scale.
MolmoMotion ships in two variants built on the same backbone. The autoregressive version, MolmoMotion-AR, represents 3D coordinates as structured text and writes out the future trajectory step by step in temporal order. The flow-matching version, MolmoMotion-FM, predicts trajectories in continuous 3D space by transforming noise into motion, which Ai2 says makes it better suited for representing uncertainty. Both use the pretrained 4B Molmo2 vision-language model as their backbone.
The training set, MolmoMotion-1M, is described by Ai2 as the largest corpus of action-described, object-grounded 3D point trajectories assembled to date, spanning 736 motion types and 5.6K distinct objects. The paper details roughly 1.16 million source video clips yielding approximately 11 million clips with motion, 736 unique action verbs, and 5,692 unique manipulated objects, with a median of 88 query points per clip. To measure accuracy, the team also introduced PointMotionBench, a human-validated benchmark of held-out 3D trajectories covering 111 object categories and 61 motion types.
Why It Matters
MolmoMotion is positioned as a component for downstream systems rather than a standalone product. In robotics simulation, a control policy built on the model completed 76.3% of pick-and-place tasks versus 56.0% for the same policy built on the Molmo 2 baseline. The paper reports that success reaches 51% at 10K steps versus 19% for Molmo2, and that on real robots the model reaches the baseline’s error level in only about 2K training steps instead of 12K. For video generation, the paper reports that a trajectory-conditioned setup, DaS+MolmoMotion, improves over CogVideoX-5B on all metrics and outperforms Wan2.2-I2V-A14B on four out of five metrics.
What We Don’t Know
The reported robotics gains come from a mix of simulation and limited real-robot trials, and the paper does not claim general-purpose manipulation across arbitrary environments. As with any newly released benchmark, PointMotionBench’s difficulty and coverage will only be validated as outside groups test their own models against it. How MolmoMotion performs on hardware and tasks beyond those the authors evaluated remains to be seen.
Analysis
MolmoMotion continues Ai2’s practice of shipping open weights, datasets, and evaluation code together rather than releasing a model in isolation, lowering the barrier for other labs to reproduce and extend the work. Its bet is representational: by forecasting motion as language-conditioned 3D point trajectories rather than pixels or 2D flow, the model aims to produce an output that plugs directly into robot planners and video generators. Whether that representation becomes a common interface for embodied AI will depend on adoption by groups building the downstream systems it is designed to feed.