Layout and Context Understanding for Image Synthesis with Scene Graphs

Arces Talavera, Daniel Stanley Tan, Arnulfo Azcarraga, Kai-Lung Hua

Research output: Chapter in Book/Report/Conference proceedingConference Article in proceedingAcademicpeer-review


Advancements on text-to-image synthesis generate remarkable images from textual descriptions. However, these methods are designed to generate only one object with varying attributes. They face difficulties with complex descriptions having multiple arbitrary objects since it would require information on the placement and sizes of each object in the image. Recently, a method that infers object layouts from scene graphs has been proposed as a solution to this problem. However, their method uses only object labels in describing the layout, which fail to capture the appearance of some objects. Moreover, their model is biased towards generating rectangular shaped objects in the absence of ground-truth masks. In this paper, we propose an object encoding module to capture object features and use it as additional information to the image generation network. We also introduce a graph-cuts based segmentation method that can infer the masks of objects from bounding boxes to better model object shapes. Our method produces more discernible images with more realistic shapes as compared to the images generated by the current state-of-the-art method.
Original languageEnglish
Title of host publication2019 IEEE International Conference on Image Processing
Subtitle of host publicationProceedings
Number of pages5
ISBN (Electronic)978-1-5386-6249-6
ISBN (Print)978-1-5386-6250-2
Publication statusPublished - Sept 2019
EventIEEE International Conference on Image Processing - Taipei, Taiwan, Province of China
Duration: 22 Sept 201925 Sept 2019


ConferenceIEEE International Conference on Image Processing
Abbreviated titleICIP 2019
Country/TerritoryTaiwan, Province of China
Internet address


  • Generative Models
  • Scene Graphs
  • Text-to-Image Synthesis


Dive into the research topics of 'Layout and Context Understanding for Image Synthesis with Scene Graphs'. Together they form a unique fingerprint.

Cite this