[P] The Big Sleep: Text-to-image generation using BigGAN and OpenAI's CLIP via a Google Colab notebook from Twitter user Adverb

From https://twitter.com/advadnoun/status/1351038053033406468:

The Big Sleep

Here's the notebook for generating images by using CLIP to guide BigGAN.

It's very much unstable and a prototype, but it's also a fair place to start. I'll likely update it as time goes on.

colab.research.google.com/drive/1NCceX2mbiKOSlAd_o7IU7nA9UskKN5WR?usp=sharing

I am not the developer of The Big Sleep. This is the developer's Twitter account; this is the developer's Reddit account.

Steps to follow to generate the first image in a given Google Colab session:

1. Optionally, if this is your first time using Google Colab, view this Colab introduction and/or this Colab FAQ.
2. Click this link.
3. Sign into your Google account if you're not already signed in. Click the "S" button in the upper right to do this. Note: Being signed into a Google account has privacy ramifications, such as your Google search history being recorded in your Google account.
4. In the Table of Contents, click "Parameters".
5. Find the line that reads "tx = clip.tokenize('''a cityscape in the style of Van Gogh''')" and change the text inside of the single quote marks to your desired text; example: "tx = clip.tokenize('''a photo of New York City''')". The developer recommends that you keep the three single quote marks on both ends of your desired text so that mult-line text can be used An alternative is to remove two of the single quotes on each end of your desired text; example: "tx = clip.tokenize('a photo of New York City')".
6. In the Table of Contents, click "Restart the kernel...".
7. Position the pointer over the first cell in the notebook, which starts with text "import subprocess". Click the play button (the triangle) to run the cell. Wait until the cell completes execution.
8. Click menu item "Runtime->Restart and run all".
9. In the Table of Contents, click "Diagnostics". The output appears near the end of the Train cell that immediately precedes the Diagnostics cell, so scroll up a bit. Every few minutes (or perhaps 10 minutes if Google assigned you relatively slow hardware for this session), a new image will appear in the Train cell that is a refinement of the previous image. This process can go on for as long as you want until Google ends your Google Colab session, which is a total of up to 12 hours for the free version of Google Colab.

Steps to follow if you want to start a different run using the same Google Colab session:

1. Click menu item "Runtime->Interrupt execution".
2. Save any images that you want to keep by right-clicking on them and using the appropriate context menu command.
3. Optionally, change the desired text. Different runs using the same desired text almost always results in different outputs.
4. Click menu item "Runtime->Restart and run all".

Steps to follow when you're done with your Google Colab session:

1. Click menu item "Runtime->Manage sessions". Click "Terminate" to end the session.
2. Optionally, log out of your Google account due to the privacy ramifications of being logged into a Google account.

The first output image in the Train cell (using the notebook's default of seeing every 100th image generated) usually is a very poor match to the desired text, but the second output image often is a decent match to the desired text. To change the default of seeing every 100th image generated, change the number 100 in line "if itt % 100 == 0:" in the Train cell to the desired number. For free-tier Google Colab users, I recommend changing 100 to a small integer such as 5.

Tips for the text descriptions that you supply:

1. In Section 3.1.4 of OpenAI's CLIP paper (pdf), the authors recommend using a text description of the form "A photo of a {label}." or "A photo of a {label}, a type of {type}." for images that are photographs.
2. A Reddit user gives these tips.
3. The Big Sleep should generate these 1,000 types of things better on average than other types of things.

Here is an article containing a high-level description of how The Big Sleep works. The Big Sleep uses a modified version of BigGAN as its image generator component. The Big Sleep uses the ViT-B/32 CLIP model to rate how well a given image matches your desired text. The best CLIP model according to the CLIP paper authors is the (as of this writing) unreleased ViT-L/14-336px model; see Table 10 on page 40 of the CLIP paper (pdf) for a comparison.

There are many other sites/programs/projects that use CLIP to steer image/video creation to match a text description.

Some relevant subreddits:

1. r/bigsleep (subreddit for images/videos generated from text-to-image machine learning algorithms).
2. r/deepdream (subreddit for images/videos generated from machine learning algorithms).
3. r/mediasynthesis (subreddit for media generation/manipulation techniques that use artificial intelligence; this subreddit shouldn't be used to post images/videos unless new techniques are demonstrated, or the images/videos are of high quality relative to other posts).

Example using text 'a black cat sleeping on top of a red clock':

https://preview.redd.it/7xq58v7022c61.png?width=512&format=png&auto=webp&s=a229ae9add555cd1caba31c42b60d907ffe67773

Example using text 'the word ''hot'' covered in ice':

https://preview.redd.it/6kxdp8u3k2c61.png?width=512&format=png&auto=webp&s=5bd078b0111575f5d88a1dc53b0aeb933f3b0da6

Example using text 'a monkey holding a green lightsaber':

https://preview.redd.it/rdsybsoaz2c61.png?width=512&format=png&auto=webp&s=2769d4c6c883c1c35ae0b1c629bebe9bc1d41393

Example using text 'The White House in Washington D.C. at night with green and red spotlights shining on it':

https://preview.redd.it/w4mg90xsf5c61.png?width=512&format=png&auto=webp&s=5f18318de2f77bcd8a86e71e87048fadd30383d1

Example using text '''A photo of the Golden Gate Bridge at night, illuminated by spotlights in a tribute to Prince''':

https://preview.redd.it/cn4ecuafhic61.png?width=512&format=png&auto=webp&s=397c838fdc49f13c5f17110b92c78b95bf0dcac0

Example using text '''a Rembrandt-style painting titled "Robert Plant decides whether to take the stairway to heaven or the ladder to heaven"''':

https://preview.redd.it/h7rb3y6j5jc61.png?width=512&format=png&auto=webp&s=537bfe8210af185647b00e7585c948aa2c4e0ffb

Example using text '''A photo of the Empire State Building being shot at with the laser cannons of a TIE fighter.''':

https://preview.redd.it/cwi7i639c5d61.png?width=512&format=png&auto=webp&s=0510c8b93adb40eee4d3f41607f1c215d41e55ff

Example using text '''A cartoon of a new mascot for the Reddit subreddit DeepDream that has a mouse-like face and wears a cape''':

https://preview.redd.it/wtxbduevcbd61.png?width=512&format=png&auto=webp&s=c5d266258922bc62f25c80a08cd9cabc07d9cb1c

Example using text '''Bugs Bunny meets the Eye of Sauron, drawn in the Looney Tunes cartoon style''':

https://preview.redd.it/gmljaeekuid61.png?width=512&format=png&auto=webp&s=9ea578de165e12afc3a62bf6886bc1ae9dc19bec

Example using text '''Photo of a blue and red neon-colored frog at night.''':

https://preview.redd.it/nzlypte6wzd61.png?width=512&format=png&auto=webp&s=7e10b06f22cfc57c64b6d05738c7486b895083df

Example using text '''Hell begins to freeze over''':

https://preview.redd.it/vn99we9ngmf61.png?width=512&format=png&auto=webp&s=2408efd607f0ab40a08db6ee67448791aa813993

Example using text '''A scene with vibrant colors''':

https://preview.redd.it/4z133mvrgmf61.png?width=512&format=png&auto=webp&s=b78e7a8e3f736769655056093a9904ff09a355a1

Example using text '''The Great Pyramids were turned into prisms by a wizard''':

https://preview.redd.it/zxt6op7vgmf61.png?width=512&format=png&auto=webp&s=53e578cfde14b28afe27957e95e610b89afadd44