• Learning from data across modalities such as image and text typically requires proper data and label supervision. How to learn AI models from cross-modality data without observing such supervision would be the challenging yet practical problem to tackle.
• As a core technology project, we target at four different and mutually related vision-and-language tasks in this project: novel image captioning, unsupervised image manipulation, image scene graph understanding & completion, and semantics-guided image completion.

1. Scene Graph (SG) Expansion

＊Unknown semantic inference
＊Self-attention and masked language modelling

2. Semantics-Guided Image Completion

＊Scene graph to layout and layout to image
＊Conditional graph convolutional network (GCN)

3. Novel Object Captioning

＊Pseudo-caption and Self-retrieval Cycle Consistency
＊Self-attention

4. Text-to-Image Manipulation

＊Learning how to modified images
＊Learning where to modified images