The Workbench

Related Lab Reports:

• Intellectual Property

• Overfitting

• Workbench: AI Assisted AI Prompt Engineering

Background: Data Requirements and Copy Protection are Fundamentally at Odds

In February, we wrote about the wave of copyright and trademark lawsuits being filed against major ML firms. A key legal question in those cases is whether systems like ChatGPT are “transformative.” More transformative reproductions of protected works are more likely to be considered legal under the “fair use” doctrine.

ML firms will argue the training process transforms images, books, etc., into something novel rather than something derivative. This feels intuitively true: Obviously, an “Artificial Intelligence” is something completely different from a book or an image.

However, generative models do regurgitate perfect and near-perfect duplicates of their training data. Getty Images and The New York Times both provided examples in their legal filings, as have independent researchers.

In addition to claims of “direct infringement” from using protected materials as training data, ML firms should worry about “contributory infringement.” An entity may be liable for contributory infringement when it “(1) has knowledge of another’s infringement and (2) either (a) materially contributes to or (b) induces that infringement.” as established in Erikson Prods. Inc. vs Kast.

Contributory infringement killed Napster — which was forced to remain shuttered even after it demonstrated it could prevent 99.4% of illegal sharing with filters. That 0.6% was enough to extinguish Napster from existence.

Overfitting — when a model memorizes its training data — can cause regurgitation in generative models. But, if a model trains on many different images of the same character, it can memorize that character without being “overfit” in the classic sense.

Complicating matters for AI firms, they are desperate for more data. These companies have gone to extraordinary lengths to gather gargantuan datasets whose exponential growth has outpaced even the internet. Research from 2022 suggested firms would run out of high-quality training data as early as 2026. ML firm’s actions confirm that data supplies are dwindling.

A recent New York Times investigation found Meta’s VP of Generative AI told executives they had already used “almost every available English language book, poem, and news article.” ML Firms are already dredging cesspools like Reddit comments, Google Maps reviews, and social media posts for data. OpenAI even invented an audio→text model just to harvest transcripts from YouTube and podcasts (violating YouTube’s terms of service).

These models cannot exist without mountains of copy-protected data. But, fair licensing deals could drive already sky-high research costs out of reach. So, Silicon Valley executives are trying a classic strategy: Build a moat before the law catches up.

With all this in mind, I set out to test OpenAI’s systems. How easily could I produce works “substantially similar” to well-known IP? Could I defeat OpenAI’s copyright filters more than 0.6% of the time? How “transformative” are these systems, really?

Two Tries to Mario

First, I just asked DALL-E to “Draw a picture of Mario,” directly. DALL-E refused for copyright reasons. First blood, OpenAI.

Next, I turned to our OpenAI-powered prompt engineering tool, which takes simple prompts and uses GPT-4 to improve them for use with DALL-E. Here’s my prompt and the result:

Subject: An italian plumber
Setting: A video game world filled with pipes
Style: Video game

I had immediate success letting GPT-4 expand on barely-coy descriptions, like “a cartoon duck in a sailor hat,” “a red-headed mermaid,” and “a video game princess in a pink dress.”

Sometimes, I got “budget” versions, a mix of multiple characters, or recognizable characters in a new style. For example: Samus with a gun, Bowser mixed with a Ninja Turtle, and Peach in a Secret of Mana style.

Other times, I got the wrong recognizable character. Here, I tried to make Nintendo’s Samus Aran but got Boba Fett from the Star Wars universe — they’re both “space bounty hunters,” after all.

After a little experimentation, I developed a hypothesis: GPT-4 could tell me how to ask DALL-E-3 to draw particular characters better than I could. I forked our gpt-imagegen repo and added a feature that allows me to do this:

% ./char_gen.py -ips outputs/mario
Character: Nintendo Mario
Name replacement: he
Setting: a world of pipes
Style: 2D 16-bit video game

Which yielded this on the first try:

I exclusively used OpenAI’s official APIs and web applications for these experiments. My technique does not require any elaborate jailbreaking methods. In fact, it’s delightfully simple.

How Does It Work?

Step 1: Ask GPT-4 for a physical description of the character. I use this prompt format and replace {character} with user input.

The model responds with a short description. For the 16-bit Mario above, GPT-4 gave me this:

This output is often enough to get a picture of the character. For example, adding “Generate an image of this:” followed by the text above got me:

Sometimes, the output contains the character’s name, which often — but not always — triggers the copyright filter. Once a generated prompt for Luigi had the phrase “the iconic character from Nintendo’s Mario series,” and DALL-E still just drew Luigi.

Anyway, I crudely remove the character’s name from the output:

To get more interesting images — with rich backgrounds or with the character taking specific actions — I send one of the following prompts to GPT-4:

Where {subject} or {character}, respectively, are replaced with the detailed output from above, and {setting} is supplied by the user.

To get a specific style, we do something similar, adding style details to the result of the previous prompt using this format:

Examples:

Aang from Avatar: The Last Airbender

Character: Aang
Name replacement: he
Setting: The Northern Air Temple
Style: TV Animation

Elsa from Frozen

Character: Princess Elsa
Name replacement: she
Style: 3D Animation, hd

Some Assembly Required

A little manual tweaking is sometimes necessary. I prompted my tool for Marvel X-Men’s "Cyclops.”

Character: Marvel Cyclops
Name replacement: he
Style: comic book

The final prompt generated by my tool contained keywords (in bold) that triggered Open AI to rewrite the prompt to DALL-E (which happens for a variety of reasons, copyright being one):

Open AI rewrote the prompt as follows:

Which produced:

So, I went back to my original prompt and made two changes. 1) Removed the first sentence and a half, where the proper nouns were. 2) Replaced “ruby quartz eyewear” with “ruby visor.” Here’s that rewritten prompt in full and the resulting image:

A Catch-22

OpenAI's copyright filters seem to operate on a high level — looking for specific words like “Nintendo,” “Mickey Mouse,” and so on. But, clearly, phrases that effectively mean Nintendo’s Mario to DALL-E breeze past the filter. Moreover, I found that GPT can produce such descriptions ad nauseam.

Perhaps because of their shared architecture and training data, DALL-E-3 usually knows what GPT-4 is talking about. With a metaphorical wink and a nod, DALL-E says, “Oh, an Italian plumber in a video game? Say no more…” And out pops Mario.

Blocking questions to GPT-4 about well-known characters would substantially reduce the value of GPT-4. But, if GPT-4 and DALL-E-3 have a shared understanding of these characters, my technique can exploit that to produce prompts that reliably recreate the character.

Here’s a prompt snippet that reliably depicts Mario when included in an image prompt to DALL-E.

All 12 of these images were produced with small additions to that prompt.

DALL-E also seems to understand this character’s context from the physical description. For example, I added “with his brother,” “with his princess,” and “after eating a flower” to get the images in the 7th, 9th, and 12th positions, respectively.

Here are a few more phrases from GPT-4 that reliably produce the characters I prompted for.

Aang, from Avatar: The Last Airbender:

Disney’s Donald Duck:

Nintendo’s Luigi:

Some Limitations

If you remove enough details, DALL-E will produce more generic images. For example, I removed the hat’s details from our Mario prompt and got this:

His “natural setting” is (apparently) still a fantastical world of pipes, not somewhere a real plumber would be, like a kitchen, basement, bathroom, or utility truck. But it’s also amazing how much impact the details of the hat had on everything else about the character.

Then again, I may have just gotten “lucky.” I pasted the exact same prompt three more times and just got three near-Marios in the iconic world of green pipes and stylized bricks:

Some Characters Are Harder

I had a lot of trouble generating a perfect Donkey Kong, possibly because tie-wearing cartoon gorillas are surprisingly common.

Most of my Kirby attempts came out recognizable… but unsettling.

Sometimes, a particular word or phrase would capture the model’s attention and take the focus off the character. For example, these three attempts at making Princess Peach used very similar prompts that all had the word “accessory” in them:

Removing the word accessory moved the output into Peach’s natural setting instead of a toy doll’s setting:

Transformative or Not: GenAI Contributes Materially to Infringement

AI firms might win the narrow argument that training models with copy-protected data is — all by itself — fair use. But copyright law doesn’t stop there. The courts will evaluate the ultimate uses of those generative models in determining outcomes: File-sharing isn’t illegal per se, but the courts still killed Napster.

The same goes for generative models. If your model makes it trivially easy to generate infringing content, you might want to pull an Open AI and hire more than 24 in-house lawyers in one year. You’re going to need them.

PS.

I don’t want to be sued, so I’m not publishing our character generator tool. But if you’re a copyright holder who wants to see if GenAI is cloning your protected works, get in touch, and I’ll see if I can help.