pystiche_papers.johnson_alahi_li_2016

Title	Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Authors	Justin Johnson, Alexandre Alahi, and Fei-Fei Li
Citation	[JAL2016]
Reference implementation	Repository / Archive
Variant	Model optimization
Content loss	FeatureReconstructionLoss
Style loss	GramLoss
Regularization	TotalVariationLoss

Behavioral changes

See also

Paper implementations

The following parts are affected:

• content_transform()

• GramLoss

• TotalVariationLoss

• decoder()

• preprocessor()

• postprocessor()

• multi_layer_encoder()

Hyper parameters

See also

Paper implementations

content_loss()

Parameter	impl_params
	True	False
layer	"relu2_2"
score_weight		1e0

style_loss()

Parameter	impl_params
	True	False
layers	("relu1_2", "relu2_2", "relu3_3", "relu4_3")
layer_weights	"sum"
score_weight		5e0

regularization()

Parameter	impl_params
	True	False
score_weight		1e-6

content_transform()

Parameter	impl_params
	True	False
image_size	(256, 256)

style_transform()

Parameter	impl_params
	True	False
edge_size	256
edge	"long"

batch_sampler()

Parameter	impl_params
	True	False
num_batches	40000
batch_size	4

API

pystiche_papers.johnson_alahi_li_2016.content_transform(impl_params=True, hyper_parameters=None)

Content image transformation from [JAL2016].

Parameters

• impl_params (bool) –

  Switch the behavior and hyper-parameters between the reference implementation of the original authors and what is described in the paper. For details see here.

  Additionally, if True, appends the preprocessor() as a last transformation step.

• hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

Return type

Sequential

pystiche_papers.johnson_alahi_li_2016.style_transform(hyper_parameters=None)

Style image transformation from [JAL2016].

Parameters

hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

Return type

Module

pystiche_papers.johnson_alahi_li_2016.images()

Return type

DownloadableImageCollection

pystiche_papers.johnson_alahi_li_2016.dataset(root, impl_params=True, transform=None)

Return type

ImageFolderDataset

pystiche_papers.johnson_alahi_li_2016.batch_sampler(data_source, hyper_parameters=None)

Batch sampler from [JAL2016].

Parameters

• data_source (Sized) – Dataset to sample from.

• hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

Return type

FiniteCycleBatchSampler

pystiche_papers.johnson_alahi_li_2016.image_loader(dataset, hyper_parameters=None, pin_memory=True)

Return type

DataLoader

pystiche_papers.johnson_alahi_li_2016.content_loss(impl_params=True, multi_layer_encoder=None, hyper_parameters=None)

Content loss from [JAL2016].

Parameters

• impl_params (bool) – Switch the behavior and hyper-parameters between the reference implementation of the original authors and what is described in the paper. For details see here.

• multi_layer_encoder (Optional[MultiLayerEncoder]) – Pretrained MultiLayerEncoder. If omitted, the default multi_layer_encoder() is used.

• hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

Return type

FeatureReconstructionLoss

class pystiche_papers.johnson_alahi_li_2016.GramLoss(encoder, impl_params=True, **gram_op_kwargs)

Gram loss from [JAL2016].

Parameters

• encoder (Encoder) – Encoder used to encode the input.

• impl_params (bool) – If True, normalize the Gram matrix additionally by the number of channels.

• **gram_op_kwargs – Additional parameters of a pystiche.loss.GramLoss.

See also

• pystiche.loss.GramLoss

pystiche_papers.johnson_alahi_li_2016.style_loss(impl_params=True, multi_layer_encoder=None, hyper_parameters=None)

Style loss from [JAL2016].

Parameters

• impl_params (bool) – Switch the behavior and hyper-parameters between the reference implementation of the original authors and what is described in the paper. For details see here.

• multi_layer_encoder (Optional[MultiLayerEncoder]) – Pretrained MultiLayerEncoder. If omitted, the default multi_layer_encoder() is used.

• hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

Return type

MultiLayerEncodingLoss

class pystiche_papers.johnson_alahi_li_2016.TotalVariationLoss(**total_variation_op_kwargs)

Total variation loss from [LW2016].

Parameters

**total_variation_op_kwargs – Additional parameters of a pystiche.loss.TotalVariationLoss.

In contrast to pystiche.loss.TotalVariationLoss, the score is calculated with the squared error (SE) instead of the mean squared error (MSE).

See also

• pystiche.loss.TotalVariationLoss

pystiche_papers.johnson_alahi_li_2016.regularization(hyper_parameters=None)

Regularization from [JAL2016].

Parameters

hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

Return type

TotalVariationLoss

pystiche_papers.johnson_alahi_li_2016.perceptual_loss(impl_params=True, multi_layer_encoder=None, hyper_parameters=None)

Perceptual loss comprising content and style loss as well as a regularization.

Parameters

• impl_params (bool) – Switch the behavior and hyper-parameters between the reference implementation of the original authors and what is described in the paper. For details see here.

• multi_layer_encoder (Optional[MultiLayerEncoder]) – Pretrained MultiLayerEncoder. If omitted, the default multi_layer_encoder() is used.

• hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

Return type

PerceptualLoss

pystiche_papers.johnson_alahi_li_2016.encoder(instance_norm=True)

Encoder part of the Transformer from [JAL2016] .

Parameters

instance_norm (bool) – If True, use InstanceNorm2d rather than BatchNorm2d as described in the paper. In addition, the number of channels of the convolution layers is reduced by half.

Return type

SequentialModule

pystiche_papers.johnson_alahi_li_2016.decoder(impl_params=True, instance_norm=True)

Decoder part of the Transformer from [JAL2016].

Parameters

• impl_params (bool) – If True, the output of the is not externally pre-processed before being fed into the perceptual_loss(). Since this step is necessary to get meaningful encodings from the multi_layer_encoder(), the pre-processing transform has to be learned within the output layer of the decoder. To make this possible, 150 * tanh(input) is used as activation in contrast to the (tanh(input) + 1) / 2 given in the paper.

• instance_norm (bool) – If True, use InstanceNorm2d rather than BatchNorm2d as described in the paper. In addition, the number of channels of the convolution layers is reduced by half.

Return type

SequentialModule

class pystiche_papers.johnson_alahi_li_2016.Transformer(impl_params=True, instance_norm=True, init_weights=True)

pystiche_papers.johnson_alahi_li_2016.transformer(style=None, framework='pystiche', impl_params=True, instance_norm=True)

Pretrained transformer from [JAL2016] .

Parameters

• style (Optional[str]) – Style the transformer was trained on. Can be one of styles given by images(). If omitted, the transformer is initialized with random weights according to the procedure used by the original authors.

• framework (str) – Framework that was used to train the the transformer. Can be one of "pystiche" (default) and "luatorch".

• impl_params (bool) – If True, use the parameters used in the reference implementation of the original authors rather than what is described in the paper.

• instance_norm (bool) – If True, use InstanceNorm2d rather than BatchNorm2d as described in the paper.

For framework == "pystiche" all combinations of parameters are available.

The weights for framework == "luatorch" were ported from the reference implementation (impl_params is True) of the original authors. See https://download.pystiche.org/models/LICENSE for licensing details. The following combinations of parameters are available:

style	instance_norm
	True	False
"candy"	x
"composition_vii"		x
"feathers"	x
"la_muse"	x	x
"mosaic"	x
"starry_night"		x
"the_scream"	x
"the_wave"		x
"udnie"	x

Return type

Transformer

pystiche_papers.johnson_alahi_li_2016.training(content_image_loader, style_image, impl_params=True, instance_norm=None, hyper_parameters=None, quiet=False)

Training a transformer for the NST.

Parameters

• content_image_loader (DataLoader) – Content images used as input for the transformer.

• style_image (Union[str, Tensor]) – Style image on which the transformer should be trained. If str, the image is read from images().

• impl_params (bool) – If True, uses the parameters used in the reference implementation of the original authors rather than what is described in the paper. For details see below.

• instance_norm (Optional[bool]) – If True, use InstanceNorm2d rather than BatchNorm2d as described in the paper. If omitted, defaults to impl_params.

• hyper_parameters (Optional[HyperParameters]) – If omitted, hyper_parameters() is used.

• quiet (bool) – If True, not information is logged during the optimization. Defaults to False.

If impl_params is True , an external preprocessing of the images is used.

Return type

Module

pystiche_papers.johnson_alahi_li_2016.stylization(input_image, transformer, impl_params=True, instance_norm=None, framework='pystiche')

Transforms an input image into a stylised version using the transfromer.

Parameters

• input_image (Tensor) – Image to be stylised.

• transformer (Union[Module, str]) – Pretrained transformer for style transfer or the style to load a pretrained transformer with transformer().

• impl_params (bool) – If True, uses the parameters used in the reference implementation of the original authors rather than what is described in the paper. For details see below.

• instance_norm (Optional[bool]) – If True, use InstanceNorm2d rather than BatchNorm2d as described in the paper. If omitted, defaults to impl_params.

• framework (str) – Framework that was used to train the the transformer. Can be one of "pystiche" (default) and "luatorch". This only has an effect, if if a pretrained transformer is loaded.

Return type

Tensor

pystiche_papers.johnson_alahi_li_2016.hyper_parameters()

Hyper parameters from [JAL2016].

Return type

HyperParameters

pystiche_papers.johnson_alahi_li_2016.preprocessor(impl_params=True)

Preprocessor from [JAL2016].

Parameters

impl_params (bool) – If True, the input is preprocessed for models trained with the Caffe framework. If False, the preprocessor performs the identity operation.

See also

• pystiche.enc.CaffePreprocessing

Return type

Module

pystiche_papers.johnson_alahi_li_2016.postprocessor(impl_params=True)

Postprocessor from [JAL2016].

Parameters

impl_params (bool) – If True, the input is postprocessed from models trained with the Caffe framework. If False, the postprocessor performs the identity operation.

See also

• pystiche.enc.CaffePostprocessing

Return type

Module

pystiche_papers.johnson_alahi_li_2016.multi_layer_encoder(impl_params=True)

Multi-layer encoder from [JAL2016].

Parameters

impl_params (bool) – If True, the necessary preprocessing of the images is performed internally.

Return type

VGGMultiLayerEncoder

pystiche_papers.johnson_alahi_li_2016.optimizer(transformer)

Optimizer from [JAL2016].

Parameters

transformer (Module) – Transformer to be optimized.

Return type

Adam