Leaner streaming model picks one voice out of crowded, noisy scenes

Pulling one person's voice out of a noisy, crowded recording is difficult enough. Doing it in real time, on modest hardware, adds further constraints. In a preprint posted to arXiv, which has not yet been peer reviewed, researchers in the Tsinghua Machine Learning Group tackle both sides of that problem. They release a more demanding benchmark built around realistic scenes. They also propose a streaming model called Falcon that, the authors report, achieves state-of-the-art quality while being much smaller and faster than the baseline it is compared against.

The task is known as audio-visual target speaker extraction. The system receives a recording in which several people talk at once, together with video of the one person whose voice is wanted, and must output that voice alone. The streaming, or causal, version may use only sound and images that have already arrived, never looking ahead. That makes it suitable for applications where delay matters. The authors identify two linked gaps. The first is that standard benchmarks fix the number of speakers at two and ignore features of real scenes. These include near-field and far-field interference (voices close to the microphone and voices further off), reverberation (echoes from surrounding surfaces) and sudden noise. The second is that the best-performing causal models are too heavy for edge deployment, which means running on the device itself rather than on a remote server. Lightweight alternatives compensate by invoking a small separator repeatedly, which, the authors note, offsets the savings.

To address the first gap, the team releases RealSSA. Its simulated part, RealSSA-Sim, offers controllable scenes with one to five dynamic near-field interfering speakers. Scenes with far-field interference add a further 10 to 15 distant interfering speakers. The other part, RealSSA-Real, is a held-out set of real recordings, kept apart from training and used only for evaluation.

Falcon tackles the second gap. Its separator, the component that does the actual pulling apart, works on a time-frequency representation of the sound. This is a map of which frequencies are present at each moment, and the separator analyses it at several scales at once. It does all of this in a single pass through an encoder-decoder network, which compresses the input into an internal representation and then rebuilds the output from it, rather than looping a small module many times. Falcon is also trained with what the authors call an encoder-space speaker contrastive loss. A contrastive loss is a training signal that pulls representations of matching examples together and pushes mismatched ones apart. Here it acts on the encoder's internal representation of speakers. Because it is used only during training, it adds nothing to the cost of running the model. The authors report that it suppresses leakage of nearby voices into the output. They also report that the benefit carries over when the loss is applied to other backbones, meaning other underlying network designs.

The authors report state-of-the-art extraction quality on RealSSA-Sim and on three established benchmarks: LRS2-2Mix, LRS3-2Mix and VoxCeleb2-2Mix. They compare Falcon against a causal version of AV-TFGridNet. Against that baseline, Falcon has 91.9% fewer parameters (the adjustable values a network learns) and needs 20.2 times less computation. Its inference latency on a GPU is 11.1 times lower.

The abstract itself leaves several questions open. It does not give Falcon's scores on the real-recorded RealSSA-Real set, which is the part of the benchmark closest to the conditions the work targets. The speed comparison is measured on a GPU, so the abstract does not address how Falcon would fare on the hardware used for edge deployment. The authors have published code and a demo, which should let others check the claims.