LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→大型多模态模型（LMM）正在处理越来越长且更丰富的输入。尽管取得了进展，但很少有公开基准来衡量这一发展。为了弥补这一差距，我们引入了 LongVideoBench，这是一个问答基准，其特点是视频-语言交错输入长达一小时。我们的基准包括 3,763 个不同长度的网络收集视频及其字幕，涵盖多种主题，旨在全面评估 LMM 在长期多模态理解方面的能力。为此，我们将主要挑战解释为从长输入中准确检索和推理详细的模态信息。因此，我们提出了一种新颖的视频问答任务，称为指代推理。具体来说，作为问题的一部分，它包含一个指代查询，引用相关的视频上下文，称为指代上下文。然后，模型需要从指代上下文中推理相关的视频细节。遵循指代推理的范式，我们在 17 个细粒度类别中策划了 6,678 个人工标注的多选题，建立了最全面的长视频理解基准之一。评估表明，LongVideoBench 即使对最先进的专有模型（如 GPT-4o、Gemini-1.5-Pro、GPT-4-Turbo）也提出了重大挑战，而其开源对应模型则表现出更大的性能差距。此外，我们的结果表明，只有当模型能够处理更多帧时，其在基准上的性能才会提高，这使得 LongVideoBench 成为评估未来一代长上下文 LMM 的宝贵基准。
Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6,678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.