Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs

EMNLP 2023

Roei Herzig*1,3   Alon Mendelson*1   Leonid Karlinsky4  
Assaf Arbelle3   Rogerio Feris4   Trevor Darrell2   Amir Globerson1  

1Tel-Aviv University   2UC Berkeley   3IBM Research   4MIT-IBM Lab  

[Paper]
[GitHub]
[Bibtex]



Narrated Overview




Abstract

Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such as object attributes, relations, and action states. In contrast, obtaining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot be used on a large scale. Here we ask whether small SG datasets can provide sufficient information for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VLMs when learning from SGs by integrating components that incorporate structured information into both visual and textual representations. For the visual side, we incorporate a special ``SG Component'' in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene. Our method improves the performance of several popular VLMs on multiple VL datasets with only a mild degradation in ZS capabilities.




Motivation

In recent years, vision and language models (VLMs) have demonstrated impressive results on a wide variety of tasks. However, recent empirical studies have shown that even the strongest VLMs struggle to perform compositional scene understanding, including identifying object attributes and inter-object relations. Understanding the structure of visual scenes is a fundamental problem in machine perception and has been explored extensively. In particular, datasets with scene graph (SG) annotations, such as the Visual Genome (VG), have been collected and used to improve scene understanding models. However, such datasets are expensive to collect at scale and relatively small compared to those used in training VLMs.
This raises the following questions:
(1) Can small datasets containing SG annotations be utilized to finetune VLMs and improve compositional scene understanding?
(2) How should the model and training be adapted to best use this data?
Here we show that it is indeed possible to improve VLMs using image-SG pairs by integrating components that incorporate structure into both visual and textual representations.




SGVL Approach

Our first step is to convert SGs into highly detailed captions. A naive approach would be to finetune VLMs on these image-text pairs, however, this approach does not sufficiently improve performance. This is also aligned with recent work showing that contrastive learning approaches allow the model to concentrate mainly on object labels disregarding other important aspects. Hence, We use the SG to generate hard-negative captions that highlight structural aspects when used with an appropriate loss.

Next, we turn to introduce structure into the visual representation. Inspired by prompt learning approaches, we incorporate into the image transformer encoder a set of ``Adaptive Scene Graph Tokens'', which interact with the patch and CLS tokens via attention. By training these tokens to predict SG information, the encoder can capture better structured representations.




Paper


Roei Herzig*, Alon Mendelson*, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, Amir Globerson
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
Hosted on arXiv