Stella X. Yu : Papers / Google Scholar

Aligning Forest and Trees in Images and Long Captions for Visually Grounded Understanding Byeongju Woo and Zilin Wang and Byeonghyun Pak and Sangwoo Mo and Stella X. Yu Neural Information Processing Systems, Atlanta, Georgia, 9-13 December 2026 Paper | arXiv Abstract Vision-language models such as CLIP often struggle to faithfully understand long, detail-rich captions, relying on dominant scene cues while overlooking fine-grained visual evidence. We propose a hierarchical vision-language learning principle for understanding scenes as part-to-whole compositions: before forming a whole-scene representation, a model should uncover what semantic parts appear where in the image. To this end, we propose CAFT ({\it \underline{C}ross-domain \underline{A}lignment of \underline{F}orests and \underline{T}rees}), a vision-language model that jointly learns local text-region alignment at intermediate representations and global image-text alignment at the final representation. Exploiting the organization of long captions, where local descriptions often correspond to scene parts, CAFT employs a fine-to-coarse image encoder and a part-whole text encoder to discover localized part semantics and progressively compose them into a global image-text representation. Trained on 30M image-text pairs, CAFT achieves state-of-the-art performance on six long-text retrieval benchmarks and exhibits strong scaling behavior. Experiments show that \sname learns fine-grained representations that localize textual semantics in image regions without explicit region-level supervision. Keywords fine-grained image-text alignment, hierarchical representation learning, part-whole composition, long-caption understanding, cross-modal retrieval, weakly supervised visual grounding