Neural Style Transfer with
{{errorMessage}}
Pick a content and a style:
What is this?
Neural Style Transfer is the process of taking the content of one image and fusing it with the style of another, for example famous grand masters paintings. The demo seen here is trained on the Image Transformation Network proposed by Johnson et. al. using MXNet and deployed using Amazon Elastic Container Service. The training code used is from the mxnet-fast-neural-style repository.
How does it work?
The Image Transformation Network proposed by Johnson et. al.
The Image Transformation Network for Neural Style Transfer is trained on a per style basis, (i.e) train one model per style image that can be used to transfer the style to any input content image. Figure 1 gives a brief overview of the Image Transformation Network training procedure.
It's all about the losses
The goal is to learn weights for the image transformation network based on the feature reconstruction loss and the style reconstruction loss. The Feature reconstruction loss penalizes the Image Transformation Network for the features of the output image y^ deviating from the features of the input (the content image x), at particular layers of the VGG net. The feature reconstruction loss is presented in Equation 1.
The Style reconstruction loss is the squared Frobenius norm of the difference between the Gram matrices of the output and target style images. It is described in Equation 2. At high level, the style reconstruction loss penalizes the Image transformation network for the features of the generated image deviating from the features of the style image.
Equation 2: Style reconstruction loss
Note: More information about the use of Gram matrices for style reconstruction loss can be found in Demystifying Neural Style Transfer. To train the model end to end, the loss functions are combined as shown in Equation 3. It is the weighted sum of the feature reconstruction loss and the style reconstruction loss and other simpler loss functions such as the per pixel loss and the total variation regularization.
Equation 3: Perceptual losses for neural style transfer
Resources
Acknowledgement
Questions?
Demo built by Thomas Delteil
