Creating highly specialized imagery without access to $1,000,000 in scientific equipment
UPDATE: I now have a scanning electron microscope in my garage.
I have been fascinated by electron microscopy since I was a child flipping through Popular Science and stumbling upon an image like this:
I love using things for things they were never intended for. So, what if we used an electron microscope to take landscape images?
My personal scanning electron microscope is currently in transit across the Atlantic Ocean, so until it arrives we will have to get creative. Without an electron microscope handy, we can instead train a Stable Diffusion LoRA on scanning electron microscope images and then use that LoRA to perform style transfer on landscape photographs.
Our process will be as follows:
We will start by getting as close as we can to a typical scanning electron microscope image using the base SDXL model. This will give us the prompting style we will later use during LoRA training and ultimately in image generation.
The base model was trained on scanning electron microscope imagery, so we can get a decent approximation with the following prompting style:
scanning electron microscope image of the head of a fly, smooth surface, grooves, sharp edges, striated surface, wide angle, high resolution, 4k, rule of thirds, masterpiece, monochrome
Not great, but a good enough starting point for our LoRA training.
As always, the success of LoRA training is dependent upon the quality of the data we use for training. High resolution scanning electron microscope imagery is readily available:
However, captions are non-existent. We will need to create highly detailed captions for each of our sample images. In addition, we plan on using this LoRA in an unconventional way; instead of generating typical scanning electron microscopy images (i.e. close-ups of surfaces or small objects), we will be generating landscape photographs. To best achieve this goal, we should “trick” our LoRA during training by convincing it that it is seeing landscape images.
To do this we write deceptive captions for our training images, for example:
wide angle landscape photo of a sparse forest with three tall bare tree trunks and numerous smaller bare tree trunks in between, rocky surface beneath, top down view, black and white, monochrome, scanning electron microscope style
This steers our LoRA in the unusual direction we want: macroscopic compositions in a microscopic context.
With our training data prepared, we move to Kohya’s GUI to perform the LoRA. After considerable experimentation, the following settings work best for our needs:
With 50 training images, 20 repeats, a batch size of 1, and 15 epochs (saving a checkpoint at each epoch) we will train for a total of 15,000 steps (with a checkpoint saved every 1000 steps). This will give us a wide cross-section of checkpoints with varying degrees of training. It is not necessarily the case that more training produces a better result, overtraining is always a risk.
Once we have our 15 epochs, we need to decide which one works best. To do this we generate a grid of test images using the prompting style we developed earlier:
wide angle landscape photo of a hill silhouetted against a black background, sharp edges, striated surface, side view, high resolution, 4k, rule of thirds, masterpiece, black and white, monochrome, scanning electron microscope style
Overtraining artifacts become a problem after epoch 13. Epoch 13 produces the best results.
We can now shift our focus to the next problem - ensuring that our generated images have compositions typical of scenic landscape photography.
We could try to generate landscape images using text prompts only, but instead let’s use famous landscape photographs to guide our image generation.
Let’s start with perhaps the most famous landscape photo of all time, Windows XP’s default wallpaper, bliss.jpg:
We can imagine an electron microscope image of a surface with a topology similar to this, but the clouds in the sky will be hard to deal with. We simplify the image in Photoshop:
We now have an image we can feed through ControlNet to guide our image generation. In this case we will use the Canny preprocessor to isolate the edges of our reference image:
Not bad, but it is missing some characteristics of true electron microscopy imagery: diffuse glow, noise, and that iconic black bar with the measurement details along the bottom. We add all of this in Photoshop:
We’ve achieved our goal.
Let’s try our workflow on photos of other famous locations.
Half Dome, as photographed by Ansel Adams:
Laila Peak:
And we are not limited to photos. Here is Hokusai’s “The Great Wave off Kanagawa” woodcut:
When visiting Napa Valley in California I felt obligated to track down the actual location of bliss.jpg.
Here is what it looked like when I visited in summer of 2016:
