Tencent's single model parses documents, images, audio and video into one format

Researchers at Tencent have built a single AI model that converts documents, images, audio and video into structured records, each anchored to where and when its content appears. In a preprint on arXiv, which has not yet been peer reviewed, the authors report that their 5-billion-parameter system, Youtu-Parsing-Omni, achieves a state-of-the-art score on a document-parsing benchmark and leads all openly released models on a broader test spanning every modality. The interesting part is the design: one encoder for every kind of input, and one output format for every task.

A parser, in this sense, is software that reads raw material such as a scanned page, a chart or a video clip and extracts its contents in an organised, machine-readable form. The authors pitch theirs as a perception layer for AI agents, programs that carry out tasks on a user's behalf and need to read material, locate the relevant part and cite it as evidence, whatever the medium.

The encoder is the part of a model that turns raw input into the numerical representations the rest of the system works with. Here one encoder, using the same learned parameters (its weights) for everything, handles every modality, placing pages, pictures, sound and video in one coordinate system for space and time. It starts from a pre-trained language model, a choice the authors say strengthens its reading of text. A few fusion layers let video frames draw on the audio playing at the same moment, bringing the merging of sound and picture, and their alignment in time, into the perception stage itself. On the output side, a shared format called OmniSchema covers all ten parsing tasks the model performs.

That common format allows a single post-training procedure, the refinement stage that follows a model's initial training, called OmniSchema-Aware On-Policy Distillation, or OSAD. Distillation normally means training a model to imitate a teacher; on-policy means it learns from its own attempts, called rollouts, rather than from a fixed set of examples. In OSAD the teacher is a copy of the model given reference material, which supplies targets for each token, the small chunks of text a model reads and writes. After every round that copy is refreshed with the newly trained weights, so the parser and its supervision evolve together. Because the objective is defined by the role each output token plays, and every task shares the same schema, one procedure serves all ten.

On OmniDocBench v1.6 the authors report an overall score of 96.96, which they describe as state of the art. On OmniParsingBench they report an average of 75.08, ahead of every open-weight model, meaning one whose trained parameters are publicly released, with the advantage concentrated on audio and video. They also report the best overall chart-parsing score, 95.02, and the highest exact agreement on chemical fingerprints, compact encodings used to compare molecular structures, among the systems compared. On typeset sheet music, its character-level transcription is said to match a dedicated optical music recognition model while staying far ahead of general-purpose systems.

These figures come from the authors and have not yet been independently checked, although the public release of code and weights should make that possible. The multimodal comparison is framed against open-weight models, so the abstract does not say how the parser fares against closed commercial systems on that benchmark, and the music result covers typeset rather than handwritten scores.