I'm building a roguelike ecosystem management game with two of my friends.  The player is a small robot who runs around and picks up biowaste drops while avoiding the critters in the ecosystem who can damage him.  On level up, the player chooses from 3 randomly selected critters to introduce to the ecosystem.  The player also gets to choose from 3 randomly selected relic-level rewards when their critters take down boss robots that show up and try to destroy the ecosystem.

For this game, we need a bunch of card images for the various buffs and relic options.  We could also use some interesting menu art to liven up the UI with some personality.  Art is far from my strong suit, but I have experimented with art generation in the past using neural networks with some positive results.  I decided to experiment with stable diffusion text to image generation to try to create some of the art assets we need.  If you're a game dev or software engineer, you've probably heard of stable diffusion.  It's blowing up all over the place with a bunch of different applications.  There are a lot of different stable diffusion websites and programs you can find on Google and Github.  The first website I tried is also the first Google search result for stable diffusion stablediffusionweb.com.  It's free to use though it can hit you with some errors when it's overwhelmed with incoming requests.  The results from it were fascinating.  The first few prompts I tried were general robot prompts without a particular design goal just to see what it was capable of.  "a robot in the style of favela".

Very encouraging.  This stable diffusion network is already a much better artist than I am.  To target the art a little more towards the game theme where you're a robot dealing with a lot of insects and arthropods I tried "a brave robot surrounded by insects"

The alien insects it created here are mildly horrifying.  Looks like it tried to spell out brave on one of these robots to indicate how brave he is or maybe he's a robot named BRVE.  Is that a robo-dick it gave the last one?  He's obviously brave to let it hang out like that.  That's some next level art, but I'm not sure I can use it for the game.  Let's try targeting a more specific image we need for the game.  One of the relic items you can get in the game is a movement buff.  We're thinking some sneakers would be a good icon cause most gamers recognize shoes as a speed buff.  "cute robot in a pair of sneakers"

It nailed the cute aspect, but we need to improve the prompt we're providing to narrow the images it's giving back.  The art in the game should be stylistically similar.  It would be chaotic to throw realistic 3d images and abstract Picasso art together in the UI.  In exploring other stable diffusion websites out there I found this one creator.nightcafe.studio which also offers free stable diffusion image generation and has some styling presets and control options for iterating on images that make it much easier to work with.  Here are some results trying to generate sneaker icons with anime style with Nightcafe.

Here are some anime style robot images Nightcafe produced that I really liked and are very usable for menu art in the game.  The prompt for these was "cute robot surrounded by spiders".  I selected the anime style preset for them which under the hood adds this to the prompt "Studio Ghibli, Anime Key Visual, by Makoto Shinkai, Deep Color, Intricate, 8k resolution concept art, Natural Lighting, Beautiful Composition".  It didn't always pickup on the spider aspect but some of them you can see the influence with some spider-like critters and webs in the background.

I really like that Nightcafe saves your results and has the option to upscale and iterate on an image.  Under the advanced options you can set a starting image and adjust the noise level or amount it lets the image generation differ from the starting image.  There is a lot of power in being able to iterate on images given a starter image.  It enables you to almost have a conversation with the AI like you would with a human artist by adjusting your prompt with the an image that has aspects you like.  Here is an example of doing that for another icon I needed for the game.  One of the relic card options is health regeneration for the player character or as we named it, "auto-repair" since you're a robot.  I didn't have a clear image in mind for this, but I figured a hammer might work as that's a common repair icon I've seen.  This may be thanks to the years I spent playing Starcraft in my youth.  Blizzard ingrained its icon language in my mind.  "hammer icon" with the same anime style preset.

I had many more results like this which make me think it doesn't know what a hammer is.  The network may just not have been trained on enough hammer images.  So, I tried providing a starting hammer icon image.  The starter icon I chose is a hammer in a gear from game-icons.net which is a website that provides some basic but good icons for free.

Starting icon image

Here are some iterations on it with prompt "hammer anime style" and varied noise levels.

Now we update the hammer icon prompt and add robotic with the anime style preset nightcafe has, which to reiterate, just adds to the prompt. "robotic hammer Studio Ghibli, Anime Key Visual, by Makoto Shinkai, Deep Color, Intricate, 8k resolution concept art, Natural Lighting, Beautiful Composition"

Results from this are very good and some could be usable for the game.  We want more consistency between icon images for the cards.  We tried this starter image that we quickly drew with some aspects we wanted in the icon.  Framing is a major aspect we hoped to capture here and we also wanted to indicate a wrench crossing a hammer which it was having a lot of trouble with from just the word prompt.

We use this with "wrench and hammer" and the anime style preset and then varied the noise levels.  It tended to do best at replicating the wrench and hammer aspects with low noise levels like 20-40%.

The results here were far better than we expected.  We provided a rough child-like crayon drawing to the network and it was able to generate great images from that.  From this we determined a pipeline for a viable solution to generate the card images in the game going forward.  We need maybe 100+ images eventually.

1. Hand draw a very basic pic design with paint or similar program for each relic.
2. Save this starter image with a prompt of what it's supposed to be.
3. For each image we want our script to append its prompt to a preset styling prompt we setup.  We also provide a negative prompt which are aspects we don't want in it.  Run that image through stable diffusion with varying noise levels ranging from 20-50%.  Generate maybe 20 images each and save them for our team to look at and vote on later.
4. Manually review the variants and vote for the best ones.

With this pipeline in mind it would be best to have a local stable diffusion network we can run against.  Nightcafe is wonderful but it does limit the amount of free requests you can make within a short time frame.  I couldn't find documentation for an API with Nightcafe either so automating with it would not be pleasant.  I could run 100+ starter images each 20+ times against Nightcafe over time, but that sounds like a pain.  So, we are using another stable diffusion network we snagged from github that we can run locally stable-diffusion-ui.github.io.

I hope this gives a good idea of what's possible with stable diffusion art generation for use in indie game development.  I highly recommend experimenting with different prompts, different websites, and various stable diffusion github repositories to see what works best for you.  It can add a lot especially for indie dev teams that may not have a full-time artist to create their art assets.  I plan to continue to use stable diffusion for generating art to supplement our assets.  It feels right using machine learning to generate the art for a game where you play the role of a machine in a universe where organic lifeforms have been pushed to extinction.