AI that judges left from right may not learn how far away things are

AI systems that watch a video stream and answer questions about the space around them can improve at one kind of question without improving at others, according to a study from researchers at LINs lab. The authors call this pattern interface lock-in. They propose a training recipe, ChronoReplay, which they report improves a single model on three spatial benchmarks at once. The paper is a preprint posted to arXiv and has not yet been peer reviewed.

The work concerns vision-language models, or VLMs: systems that take in images or video alongside text and reply in text. Spatial questions come in different forms. Some ask for a category, such as which side an object lies on. Others ask for a measurement, such as a distance. The authors argue that these are not interchangeable as training signals. A model can learn to name the correct side without preserving the metric displacement that determines the answer, meaning the actual distance and direction. Training a model to estimate distances may likewise leave categorical relations largely unconstrained. Post-training is the fine-tuning done after a model's initial large-scale training. When it emphasises one kind of answer, the gains can concentrate there, and transfer to other answer types can remain weak or even get worse.

ChronoReplay tackles this in three ways. First, it replays spatial training data drawn from different sources as episodes arranged in chronological order. This mirrors the streaming setting, in which a model sees frames one after another. Second, it keeps each source's original answer format rather than forcing everything into a single style. Third, it balances the amount of supervision across families of question types. The authors kept the model's access to video frames the same in each comparison. Under those conditions, they report that one ChronoReplay checkpoint, a single saved version of the trained model, improves on OVO-S-Bench, OST-Bench and UCS-Bench together. The improvements include a 13.41-point gain on OST-Bench. The checkpoint also achieves the highest average across the three benchmarks among the training setups the team compared.

To separate the ingredients, the authors ran controls on the same data. Serialisation is the way answers are written out as text. Changing it shifted gains between OST-Bench and UCS-Bench rather than simply adding to them. Balancing across question families beat ordinary joint supervised fine-tuning, in which all the data is pooled together. It also beat a weighting that follows the natural number of turns in the training conversations. Keeping episodes in chronological order outperformed a version with shifted ordering by about 2.41 points on OST-Bench, across two paired runs with different random seeds. Separately, the team tried adaptive frame selection, which changes which video frames the model sees at test time and leaves the trained model untouched. They report that this gave a further gain while using less visual input.

The authors conclude that streaming spatial performance depends on three things: which distinctions the training signal preserves, how mixed histories are organised during learning, and what evidence is available to the model when it answers. Some of the evidence is narrow. The ordering effect is reported across only two paired seeds, and the main comparison covers only the supervision settings examined in the study. How far the recipe generalises beyond them remains open.