Dual Text-guided Cross-attention for Global–local Visual Fusion in Vietnamese Visual Question Answering

Abstract

Visual Question Answering (VQA) requires effective cross-modal reasoning between visual content and natural-language questions. This challenge is particularly significant for Vietnamese due to the relatively limited availability of annotated VQA resources. This study proposes a dual-stream architecture that exploits complementary Transformer-based and convolutional visual representations for Vietnamese VQA. A Vision Transformer (ViT) is employed to obtain globally contextualised visual features, while ConvNeXt V2 preserves spatially structured visual information. Questions are encoded using PhoBERT. Instead of directly merging the two visual streams, the proposed model uses the textual representation as a shared query in two independent cross-attention modules, allowing question-relevant information to be retrieved separately from each visual representation prior to fusion. The resulting representations are concatenated along the sequence dimension, pooled, and combined with the sentence-level textual embedding through a residual connection for answer classification. Experiments on the ViVQA benchmark show that the proposed architecture achieves 64.31% accuracy and 62.65% F1-score. Additional experiments across multiple ConvNeXt V2 backbone scales consistently show improvements over the corresponding single-stream convolutional baselines, supporting the complementary role of the two visual representations.