Out of Vocabulary

Online, 2019

Out of Vocabulary looks at what happens when a text-to-image system meets a word it doesn’t know. It uses AttnGAN, developed at Microsoft Research in 2017 and one of the first text-to-image models the public could use, through the RunwayML app from 2019. When a word falls outside AttnGAN’s vocabulary, the system doesn’t flag it or ask. It drops the word and carries on as if it was never typed.

AttnGAN learned to draw from Microsoft’s COCO dataset: 123,287 Flickr photographs and 616,265 captions written by paid online workers. Its vocabulary of 27,296 entries comes from those captions. Each of the ten prompts here is built around one word that isn’t in it: a bisexual person, a sequin dress, a matcha latte, a magnolia tree, a platypus model, a nebula painting, a rhomboid tile, a clarinet player, a kaleidoscope pattern and a hologram sticker. To the system, these read as a person, a dress, a latte and so on.

The images were made with the original 2018 COCO model. Each one is shown at its true size, 256 × 256 pixels, the most the model can produce. It draws each image in three passes, moving from rough to detailed.