With the advancement of Office AI and document agents, large models are now directly processing lengthy text and image-based materials such as financial reports, contracts, and research papers.
But being able to receive an entire document does not mean you can correctly use the information within it.
Faced with hundreds of pages of text, images, tables, and charts, the model must first answer: Where are the actual pieces of evidence relevant to the current question?
The newly accepted paper at EMNLP 2026, "Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models," investigates this process from within the model.
From “Can Fit” to “Can Find”
Long-context vision-language models can simultaneously process large amounts of text and images, but the final answer typically depends on only a small portion of that information. The relevant evidence may be a segment of text, or it may be hidden within tables, images, or specific layout areas.
The research team is investigating whether, when evidence is already present in the input, there exists a set of attention heads within the model that directs the question toward the relevant text or visual content.
The team refers to these identified attention heads as Multimodal Retrieval Heads (MMRetHeads).
How to identify a multimodal retrieval head?
Inspired by QRHead, the research team proposed the MMRetHeads detection method.
Specifically, this method analyzes the question-to-evidence attention of each attention head—that is, the attention from question tokens to annotated evidence regions. If the evidence is text, it corresponds to text tokens; if the evidence is within an image, it corresponds to visual tokens. The stronger the attention directed toward the evidence, the higher the retrieval score for that attention head.
△
The study covers six long-context vision-language models, including Qwen3-VL and Gemma3, evaluating tasks such as text retrieval, image retrieval, rendered text retrieval, and identical image retrieval, with context lengths of 8K, 16K, 32K, 64K, and 128K.
The analysis also found that text and image retrieval share some attention heads, but different context lengths and evidence formats activate different attention heads.
Do these attention heads really affect the model's responses?
Focusing attention on the evidence only indicates an association between the two. To test whether the model truly relies on these attention heads, the research team masked the top-scoring retrieval heads and compared the results with those obtained by masking an equal number of randomly selected attention heads.
In text and image retrieval tasks, masking the retrieval head leads to a significant drop in model performance across different context lengths and evidence positions; the impact of random masking is much smaller.
The same discrepancy also appears in more realistic long-document question answering:
- MMLongBench-Doc: 48.2 → 5.7
- SlideVQA: 71.2 → 8.9
After random masking, the two tasks still retained scores of 32.2 and 52.6, respectively. The greater drop caused by masking the retrieval heads indicates that these heads do not merely focus attention near the evidence but also play a causal role in the model’s access to evidence.
Similar results can be observed in multimodal reasoning tasks. After the retrieval head is masked, the model may sometimes respond with “insufficient information” even when the chart is still present in the input, or it may misread the content or generate nonexistent justifications.
△
From internal mechanisms to document retrieval
The research team further utilized the evidence signals from these attention heads for multimodal document retrieval.
Given a query, MMRetHeads computes relevance scores for candidate pages or layout regions and ranks the candidates accordingly. Page-level retrieval identifies entire pages containing evidence, while layout-level retrieval further locates specific text blocks, tables, images, or charts within the page. This entire process requires no additional training of the retriever.
On MMDocIR, Qwen3-VL-8B achieves a page-level Recall@1 of 64.7, an improvement of 7.7 percentage points over the strongest reported baseline; and a layout-level Recall@1 of 39.0, an improvement of 6.3 percentage points over the strongest reported baseline.
△
Paper link: https://arxiv.org/abs/2605.27243
This article is from the WeChat public account "Quantum Bit," authored by the MMRetHeads team.