After playing around for a while with the art-style transfer routines, I found a couple of pages and scripts that detailed some ways to improve the quality of the results (based on “Controlling Perceptual Factors in Neural Style Transfer”). Additionally, one of them described a way to mix a fine style with a coarse one to generate style mixes that could be applied to images. This, combined with some image processing and machine learning super-resolution up-scaler algorithms gave me the opportunity to create a new pipeline for more aggressive style transfers.

Development

My art-style transfer approach takes place in several stages:

1. Input image: Processing applied directly to the original image
2. Stylemix: Generating the “stylemix” palette
   1. Color Transfer: Transferring color to the palette
   2. Post-Process: Blurring and filtering done to the generated palette
3. Style transfer: Proper artstyle transfer to the image
   1. NN-Artstyle Transfer: NN transfer
   2. Blend: Blending original image to the NN transfer
   3. Upscaling: Artstyle upscaling
4. Superes: Super-resolution applied to the processed image

which will be described in the following sections of the post.

Input image

Most of the processing steps applied to the original involve color balancing, cropping, resizing, histogram shifting, etc. This is very image-dependent, so it is not included in the main pipeline.

Some general tips for better results are:

• Start with a reasonably large image to have enough detail to work with.
• Try to get an image as close as possible to the style you want transferred.
• Work a bit with the colors before throwing it to the pipeline.
• Avoid unnecessary details in the backgrounds.

Stylemix

As detailed in the paper, the first step is to take the fine and coarse styles and mix them together into a transfer palette. Generally speaking, we want the fine style to be a texture (or technique), whilst the coarse one would probably be the composition (and/or style). To do this, we

Color Transfer

An initial optional step in generating the style-mix is if we want the color of the original image to be transferred upon it. This involves…

Post-Process

A second optional step is to do some post-processing on the stylemix. In my experience, the NN algorithm sometimes generates some glitches in the image (such as color dots and strange image artifacts). This can be slightly alleviated through some bi-linear filtering and sharpening (which also generates “smoother” images).

Style transfer

Once these steps have taken place on the stylemix we can move onto the proper NN style transfer.

NN-Artstyle Transfer

Blend

As good as the styletransfer is, sometimes it helps to have a slight blend of the original image “overlaid” on top of the already transferred one. This is mostly necessary in cases where we want to retain some of the fine details of the original in places where the transfer might be overpowering them.

Upscaling

As noted by Gatys, et al., to get much better results in the upscaling of stylized image, we take the lo-res version of the input image, upscale it with some basic algorithm, and then apply the stylemix ot this pre-upscaled image. This also helps alleviate the artificial noise added to the image in earlier stages.

Superes

Finally, to smooth out the image a bit and to make it larger, we can use a super-resolution algorithm:

Style transfer notes

1. Current’s algorithm resolution “sweet spot” is around 500px. This means that for the best results, images should ideally be re-scaled to this resolution before processing.
2. Some of the glitches on the style-transfers can be somewhat alleviated with doing some blurring. The mixing algorithm can sometimes create some glitches in the images, which manifest themselves as color blends or irregularities in the texture; but doing some bi-linear filtering combined with sharpening can improve the results.
3. The more similar the “coarse” style image to the one we want the style transferred to, the better results. This is somewhat straightforward, if possible, we should use images that are as close as possible in general representation to the image we want the artstyle transferred to.

Splatoon Art Project

Using these scripts, I went ahead and did some transfers on my Splatoon’s team portraits!

Dependencies and Code

• Repository: Github repo
• Dependencies: neural-style, Image Super-Resolution, open-cv, color-transfer

Sources

• “Controlling Perceptual Factors in Neural Style Transfer” by Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Aaron Hertzmann, Eli Shechtman
• ProGamerGov’s Github (neural-style-pt & neural-tools)
• Image Super-Resolution