FoleyBench: A Benchmark For Video-to-Audio Models

Video-to-audio generation (V2A) is of increasing importance in domains such as film post-production, AR/VR, and sound design, particularly for the creation of Foley---sound effects synchronized with on-screen actions. Foley requires generating audio that is both semantically aligned with visible events and temporally aligned with their timing. Yet, there is a mismatch between evaluation and downstream applications due to the absence of a benchmark tailored to Foley-style scenarios. We find that 74% of videos from past evaluation datasets have poor audio-visual correspondence. Moreover, they are dominated by speech and music—domains that lie outside the use case for Foley.

To address this gap, we introduce FoleyBench, the first large-scale benchmark explicitly designed for Foley-style V2A evaluation. FoleyBench contains 5,000 (silent video, ground-truth audio, text caption) triplets, each featuring visible sound sources with audio causally tied to on-screen events. The dataset is built using an automated, scalable pipeline applied to in-the-wild internet videos from YouTube-based and Vimeo-based sources. Compared to past datasets, we show that videos from FoleyBench have stronger coverage of a taxonomy specifically designed for Foley sound. Each clip is further labeled with metadata capturing source complexity, UCS/AudioSet category, and video length, enabling fine-grained analysis of model performance and failure modes. We benchmark 11 state-of-the-art V2A models, evaluating them on audio quality, audio–video alignment, temporal synchronization, and audio–text consistency. Our analysis reveals key limitations of existing models and provides actionable guidance for advancing the field.

Please email us to request access to the FoleyBench benchmark dataset: satvikdixit7@gmail.com

^T indicates text-conditioned models. ↑ higher is better, ↓ lower is better. Bold indicates best performance, underlined indicates second best.
FAD, IS, and KLD are calculated on PANN embeddings. We used the AV-benchmark toolkit for evaluation.

Our experiments show that widely used benchmarks such as VGGSound Test often include content unsuitable for Foley evaluation, which can lead to misleading conclusions about model performance.