API Reference

nnweaver.activations module

The activations module provides a set of activation functions to be applied to a layer of a neural network.

class nnweaver.activations.Activation

Bases: abc.ABC

Abstract base class for classes that implement an activation function.

classmethod apply(x)

classmethod gradient(x)

class nnweaver.activations.HardSigmoid

Bases: nnweaver.activations.Activation

classmethod apply(x)

Compute the hard-sigmoid activation function on each element of the input array.

Parameters:x – an array. Returns:the activated values of the input array.

classmethod gradient(x)

Compute the gradient of the hard-sigmoid activation function on each element of the input array.

Parameters:x – an array. Returns:the gradients on the values of the input array.

class nnweaver.activations.HardTanH

Bases: nnweaver.activations.Activation

classmethod apply(x)

Compute the hard-tanh activation function on each element of the input array.

Parameters:x – an array. Returns:the activated values of the input array.

classmethod gradient(x)

Compute the gradient of the hard-tanh activation function on each element of the input array.

Parameters:x – an array. Returns:the gradients on the values of the input array.

class nnweaver.activations.Linear

Bases: nnweaver.activations.Activation

classmethod apply(x)

Compute the linear activation function (identity) on each element of the input array.

Parameters:x – an array. Returns:the input array.

classmethod gradient(x)

Compute the gradient of the linear activation function (constant to 1) on each element of the input array.

Parameters:x – an array. Returns:an array of 1s of the same shape of the input array.

class nnweaver.activations.Rectifier

Bases: nnweaver.activations.Activation

classmethod apply(x)

Compute the rectifier activation function on each element of the input array.

Parameters:x – an array. Returns:the activated values of the input array.

classmethod gradient(x)

Compute the gradient of the rectifier activation function on each element of the input array.

Parameters:x – an array. Returns:the gradients on the values of the input array.

class nnweaver.activations.Sigmoid

Bases: nnweaver.activations.Activation

classmethod apply(x)

Compute the sigmoid activation function on each element of the input array.

Parameters:x – an array. Returns:the activated values of the input array.

classmethod gradient(x)

Compute the gradient of the sigmoid activation function on each element of the input array.

Parameters:x – an array. Returns:the gradients on the values of the input array.

class nnweaver.activations.TanH

Bases: nnweaver.activations.Activation

classmethod apply(x)

Compute the tanh activation function on each element of the input array.

Parameters:x – an array. Returns:the activated values of the input array.

classmethod gradient(x)

Compute the gradient of the tanh activation function on each element of the input array.

Parameters:x – an array. Returns:the gradients on the values of the input array.

nnweaver.callbacks module

The nnweaver.callbacks module provides the callback classes that will be used to execute some pre-defined or user-defined actions before, during, or after the training of a neural network.

class nnweaver.callbacks.Callback

Bases: abc.ABC

Abstract base class for classes that provide methods that can be called during the Optimizer.train() execution.

on_epoch_end(epoch, nn, loss_value, metrics_values)

on_training_begin(nn)

on_training_end(nn)

class nnweaver.callbacks.PlotLearningCurve(x_validation=None, y_validation=None, loss=None, interactive=True, max_epochs=None, blocking=True)

Bases: nnweaver.callbacks.Callback

Create a callback that plot the learning curve of a model during and after its training phase. If a data set and a loss function are given, it will also plot the validation results.

Parameters:

• x_validation – the set of validation examples.
• y_validation – the set of validation targets.
• loss – the loss function to be evaluated in the validation step.
• interactive – if True, show the learning curve interactively during the training of the model, otherwise only after it’s ended. Notice that the former may cause a loss of training performance.
• max_epochs – the expected number of epochs. If present, this parameter prevents the plot from rescaling during the training (if interactive is False, this parameter is ignored).
• blocking – if True display the curve window and block until it has been closed.

on_epoch_end(epoch, nn, loss_value, metrics_values=None)

The callback to be executed at the end of every epoch.

Parameters:

• epoch – the epoch number.
• nn – the neural network that is being trained.
• loss_value – the loss value obtained in this epoch.
• metrics_values – ignored.

on_training_begin(nn)

The callback to be executed at the beginning of the training.

Parameters:nn – the neural network that is being trained.

on_training_end(nn)

The callback to be executed at the end of the training.

Parameters:nn – the neural network that is being trained.

class nnweaver.callbacks.WriteFileCallback(filename, x_validation=None, y_validation=None, loss=None, metrics=None)

Bases: nnweaver.callbacks.Callback

Create a callback that, at each epoch, writes the loss found by a training algorithm to a CSV file. If a data set and a loss function are given, it will also write to the file the validation results.

Parameters:

• filename – the output file.
• x_validation – the set of validation examples.
• y_validation – the set of validation targets.
• loss – the loss function to be evaluated in the validation step.
• metrics – a list of metrics to be evaluated in the validation step.

on_epoch_end(epoch, nn, loss_value, metrics_values)

The callback to be executed at the end of every epoch.

Parameters:

• epoch – the epoch number.
• nn – the neural network that is being trained.
• loss_value – the loss value obtained in this epoch.
• metrics_values – the other metric values obtained in this epoch.

on_training_begin(nn)

The callback to be executed at the beginning of the training.

Parameters:nn – the neural network that is being trained.

on_training_end(nn)

The callback to be executed at the end of the training.

Parameters:nn – the neural network that is being trained.

nnweaver.losses module

The losses module provides a set of loss/objective functions whose output should be minimized during the training phase.

class nnweaver.losses.Loss

Bases: abc.ABC

Abstract base class for classes that provide methods to compute the cost function over a single example or a batch of examples.

classmethod batch_mean(y, y_target)

Compute the average loss over a batch of examples.

Parameters:

• y – the bi-dimensional matrix whose rows are the examples and whose columns are the predicted features.
• y_target – the target bi-dimensional matrix. Must have the same number of rows of y,

Returns:

the average loss.

classmethod gradient(y, y_target)

classmethod loss(y, y_target)

class nnweaver.losses.MEE

Bases: nnweaver.losses.Loss

classmethod euclidean_distance(y, y_target)

Compute the euclidean distance between two vectors.

Parameters:

• y – the predicted output (a column vector).
• y_target – the target output (a column vector).

Returns:

the euclidean distance.

classmethod gradient(y, y_target)

Compute the gradient of Mean Euclidean Error (MEE) of the predicted example.

Parameters:

• y – the predicted output (a column vector).
• y_target – the target output (a column vector).

Returns:

the gradient of the MEE (a column vector).

classmethod loss(y, y_target)

Compute the Mean Euclidean Error (MEE) of the predicted example.

Parameters:

• y – the predicted output (a column vector).
• y_target – the target output (a column vector).

Returns:

the MEE.

class nnweaver.losses.MSE

Bases: nnweaver.losses.Loss

classmethod gradient(y, y_target)

Compute the gradient of Mean Squared Error (MSE) of the predicted example.

Parameters:

• y – the predicted output (a column vector).
• y_target – the target output (a column vector).

Returns:

the gradient of the MSE (a column vector).

classmethod loss(y, y_target)

Compute the Mean Squared Error (MSE) of the predicted example.

Parameters:

• y – the predicted output (a column vector).
• y_target – the target output (a column vector).

Returns:

the MSE.

nnweaver.nn module

The nn module contains the NN (neural network) and the Layer classes.

Formally, a neural network models a function \(f(\boldsymbol x)=f^{(L)}(f^{(L-1)}(\dots f^{(1)}(\boldsymbol x)\dots ))\) , where \(\boldsymbol x\) is the input vector, and the \(f^{(i)}\) s are the layers of the network.

A layer specifies a function \(f(\boldsymbol x)=g(\boldsymbol{Wx} + \boldsymbol b)\) , where \(g\) is called activation function, \(\boldsymbol W\) provides the weights of a linear transformation, and \(\boldsymbol b\) is the bias vector.

A neural network can be trained with an optimizer and then used to predict the target values of (previously unseen) examples. See the optimizers module for a list of available training algorithms and the activations module for a list of available activation functions.

class nnweaver.nn.Layer(units, activation=<class 'nnweaver.activations.Linear'>, weights_initializer=<function uniform.<locals>.<lambda>>, bias_initializer=<function uniform.<locals>.<lambda>>)

Bases: object

Create a new layer with a given number of units and activation function.

Parameters:

• weights_initializer – a function that takes a shape and returns a matrix with the specified shape. Already implemented weights initializer are uniform(), he_normal(), glorot_uniform().
• bias_initializer – see the description of the argument weights_initializer.
• units – the number of units in the layer.
• activation – the activation function.

__call__(x)

Compute the output of the layer.

Parameters:x – input to the layer (a column vector). Returns:activated output (a column vector).

build_weights(preceding_units)

Initialize the weights and the bias vector of the layer with random values.

Parameters:preceding_units – number of units in the previous level.

input_sum(x)

Compute the input to the layer, without applying the activation function.

Parameters:x – input to the layer (a column vector). Returns:weighted sum of the input with the bias (a column vector).

reset_weights()

Re-initialize the weights and the bias vector of the layer with random values.

class nnweaver.nn.NN(input_dim)

Bases: object

Create a multi-layer perceptron (MLP, also called feedforward neural network) with the given number of inputs.

Parameters:input_dim – number of dimensions of the input.

add_layer(layer)

Add a layer to the neural network.

Parameters:layer – the Layer to add.

clone()

Clone the neural network.

Returns:the copy of the neural network.

predict(x)

Feed a single input to the neural network.

Parameters:x – the input. Its size must match the input_dim of the neural network. Returns:the output of the neural network (a column vector).

predict_batch(x_batch)

Feed multiple inputs to the neural network.

Parameters:x_batch – a list of examples. Returns:an array (with the same size of x_batch) with the outputs of the network.

reset()

Flash the neural network with a neuralyzer.

nnweaver.nn.glorot_uniform()

Returns a weights initializer with values uniformly distributed over the half-open interval \(\left[-\sqrt{\frac{6}{n_i+n_o}}, \sqrt{\frac{6}{n_i+n_o}} \right)\), where \(n_i\) and \(n_o\) are the number of inputs and outputs of a layer.

The result of this function can be passed as distribution argument to NN.add_layer() .

See “Understanding the difficulty of training deep feedforward neural networks” by Glorot and Bengio (2010).

Returns:a weight initializer.

nnweaver.nn.he_normal()

Returns a weight initializer with a Gaussian distribution whose mean is zero and standard deviation is \(\sqrt{\frac{2}{n_i}}\), where \(n_i\) is the number inputs to a layer.

See “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification” by He et al (2015).

Returns:a weight initializer.

nnweaver.nn.uniform(low, high)

Returns a weights initializer with values uniformly distributed over the half-open interval \([low, high)\).

The result of this function can be passed as distribution argument to NN.add_layer() .

Parameters:

• low – lower boundary of the random values.
• high – upper boundary of the random values.

Returns:

a weight initializer.

nnweaver.optimizers module

The optimizers module provides a set of optimization algorithms that can be used to train neural networks.

Currently, the following optimizers are available:

1. SGD Stochastic Gradient Descent.
2. ProximalBundleMethod Proximal Bundle Method.

class nnweaver.optimizers.GradientBasedOptimizer(loss)

Bases: nnweaver.optimizers.Optimizer

Abstract class for optimizers that use the gradient information.

Parameters:loss – the loss function to optimize.

backward(nn, x, y, inputs, outputs)

Propagate an error backward through the network. See “Deep Learning” by Ian Goodfellow et. al (2016) p. 206.

Parameters:

• nn – the neural network.
• x – the input.
• y – the target output.
• inputs – a list of the input sums at each layer.
• outputs – a list of the activated outputs at each layer.

Returns:

a list of the gradients of the loss function at each layer.

forward(nn, x)

Propagate an input signal through the network.

Parameters:

• nn – the neural network.
• x – the input.

Returns:

two lists of length len(nn.layers) that contains, respectively, the input sum and the output (i.e. the result of applying the activation function on the input sum) at each layer.

train(nn, x, y, **train_args)

class nnweaver.optimizers.Optimizer(loss)

Bases: abc.ABC

Abstract base class for optimizers.

Parameters:loss – the loss function to optimize.

classmethod shuffle(x, y)

Shuffle x and y elements with the same permutation.

Parameters:

• x – a list of examples.
• y – a list with the target output of each example.

Returns:

x, y. Permuted.

train(nn, x, y, **train_args)

class nnweaver.optimizers.ProximalBundleMethod(loss)

Bases: nnweaver.optimizers.GradientBasedOptimizer

Create an optimizer that implement the Proximal Bundle Method algorithm.

Parameters:loss – the loss (cost) function to optimize.

static flatten(list_weights, list_bias)

Returns a column vector with the concatenation of the given neural network parameters.

train(nn, x, y, mu=1.0, m_L=0.1, m_R=0.99, t_bar=0.5, gamma=0.5, accuracy_tolerance=0.0001, max_iterations=10, a_bar=inf, regularizer=None, callbacks=None)

Train the given neural network using the Proximal Bundle Method algorithm.

Parameters:

• nn – the neural network.
• x – a list of examples.
• y – a list with the target output of each example.
• mu – the fixed weight to be given to the stabilizing term throughout all the algorithm. It must be a strictly positive number.
• m_L – line search parameter. Must be 0 < m_L <= 0.5.
• m_R – line search parameter. Must be m_L < m_R < 1.
• t_bar – set to a small value in (0,1) to prevent large steps relative to the decreasing of the loss function.
• gamma – the distance measure parameter. Higher values lead to more localized information of the subgradients. Must be γ >= 0.
• accuracy_tolerance – a value greater than 0 that determines the stopping criterion.
• max_iterations – maximum number of iterations before stopping the training.
• a_bar – the locality radius tolerance. The subgradient information collected outside the ball of radius a_bar around the current solution will be discarded.
• regularizer – a regularizer that will be used in the training.
• callbacks – a list of Callback objects.

static unflatten(nn, flattened)

Plug back the flattened weights/bias into the neural networks.

class nnweaver.optimizers.SGD(loss)

Bases: nnweaver.optimizers.GradientBasedOptimizer

Create an optimizer that implement the Stochastic Gradient Descent (SGD) algorithm.

Parameters:loss – the loss (cost) function to optimize.

static batch_ranges(x, batch_size)

Compute the batch ranges for a given data set.

Parameters:

• x – a list of examples.
• batch_size – the batch size.

Returns:

a list of tuples containing the lower and the upper bounds (indexes) of each batch.

train(nn, x, y, learning_rate=0.05, batch_size=1, epochs=1, momentum=0, metrics=None, callbacks=None, regularizer=None)

Train the given neural network using the Stochastic Gradient Descent (SGD) algorithm.

See “Deep Learning” by Ian Goodfellow et. al (2016) p. 286.

Parameters:

• nn – the neural network.
• x – a list of examples.
• y – a list with the target output of each example.
• learning_rate – the step size of the gradient descend. It can be a constant, or a generator that returns the step size at each next() call.
• batch_size – the batch size.
• epochs – the number of the epochs.
• momentum – value between 0 and 1. Determines how quickly the contribution of previous gradients exponentially decay.
• metrics – a list of metric functions to be evaluated at each epoch.
• callbacks – a list of Callback objects.
• regularizer – a regularizer that will be used in the training.

nnweaver.optimizers.learning_rate_linearly_decayed(initial_rate, final_rate=0, max_iterations=20)

A generator function that, starting from initial_rate, decays the learning rate linearly. After max_iterations it always yields final_rate.

It can be passed as learning_rate argument to SGD.train() .

Formally, it computes \(\epsilon_k = (1 - \alpha)\epsilon_0+ \alpha \epsilon_\tau\), where \(k\) is the iteration number, \(\tau\) is max_iterations, \(\epsilon_0\) is initial_rate, \(\epsilon_\tau\) is final_rate, and \(\alpha=\frac{k}{\tau}\).

Parameters:

• initial_rate – the rate at the first iteration.
• final_rate – the rate to return after the maximum number of iterations.
• max_iterations – the maximum number of iterations.

nnweaver.optimizers.learning_rate_time_based(initial_rate, rate_decay)

A generator function that decays the rate with time.

It can be passed as learning_rate argument to SGD.train() .

Formally, it computes \(\epsilon_k = \frac{\epsilon_{0}}{1+\gamma k}\), where \(k\) is the iteration number, \(\gamma\) is rate_decay, \(\epsilon_0\) is initial_rate.

Parameters:

• initial_rate – the rate at the first iteration.
• rate_decay – the decay factor.

nnweaver.regularizers module

The regularizers provides a set of weight regularization classes to be used during the training of a neural network.

class nnweaver.regularizers.L1L2Regularizer(l1=0, l2=0)

Bases: nnweaver.regularizers.Regularizer

Initialize a regularizer on weights’ norms.

Parameters:

• l1 – the factor of the L1 weight decay term.
• l2 – the factor of the L2 weight decay (aka ridge regression or Tikhonov regularization) term.

__call__(nn)

Apply both L1 and L2 norms on the weights of every layer of the given neural network.

Parameters:nn – a neural network. Returns:the loss penalty.

gradient(layer)

Compute the gradient of the loss penalty for a single layer of a neural network.

Parameters:layer – a layer. Returns:the weight and the bias gradient penalty.

class nnweaver.regularizers.Regularizer

Bases: abc.ABC

Abstract base class for classes that implements functions on the NN’s parameters to control the complexity of the model during the training.

__call__(nn)

Call self as a function.

gradient(layer)

nnweaver.utils module

The nnweaver.utils module provides a set of utility functions.

nnweaver.utils.accuracy(y, y_target)

Compute the (binary) accuracy of a given prediction.

If the dimension of the input arrays is greater than 1 (e.g. y_target is encoded with one_hot_encoding()), the multi-class accuracy is computed.

Parameters:

• y – the predicted output.
• y_target – the target (binary) output.

Returns:

the accuracy of the predicted output, rounded.

nnweaver.utils.one_hot_encoding(x)

Compute the One-Hot-Encoding (OHE) of a list of elements.

Parameters:x – the elements to be encoded. Returns:a matrix with rows equal to the number of the elements in x and max(x) columns.

nnweaver.validation module

The validation module provides a set of cross validation and model selection methods.

nnweaver.validation.grid_search(nn_builder: Callable[dict, nnweaver.nn.NN], optimizer, x, y, train_args, builder_args, cv: Callable[[nnweaver.nn.NN, nnweaver.optimizers.Optimizer, object, object], Dict[str, List[float]]] = None)

Perform an exhaustive search through a space of training and neural network topology parameters.

An iteration of the grid search is one of the possible combinations of the training and builder parameters. The first ones will be fed to optimizer.train(), while the second ones will be fed to nn_builder.

Parameters:

• nn_builder – a function that accepts a parameter builder_args and returns a neural network.
• optimizer – the optimizer to use in the grid search. Its signature must be compatible with the keys in train_args.
• x – a list of examples.
• y – a list with the target output of each example.
• train_args – a dictionary whose keys are: (1) compatible with the arguments of optimizer.train(), and (2) associated with lists that represent the subset of the arguments to explore.
• builder_args – a dictionary whose keys are: (1) compatible with the arguments of nn_builder(), and (2) associated with lists that represent the subset of arguments to explore.
• cv – an optional cross validation method that is called at each iteration.

Returns:

a tuple with: (1) the best model found by the grid search, (2) its loss value, (3) arguments used to train it, and (4) arguments used to build it.

nnweaver.validation.hold_out_validation(nn, optimizer, x, y, train_ratio=0.8, **train_args)

Perform an Hold-Out Validation of the given neural network.

Parameters:

• nn – a neural network.
• optimizer – the optimizer used to train the neural network. Its signature must be compatible with the keys in train_args.
• x – a list of examples.
• y – a list with the target output of each example.
• train_ratio – the ratio between the size of the partition of examples used to train the neural network and the one used to test it.
• train_args – a dictionary whose keys are compatible with the arguments of optimizer.train().

Returns:

the trained model and the value of the loss computed on the test partition.

nnweaver.validation.kfold_cross_validation(nn, optimizer, x, y, k=3, **train_args)

Perform a K-Fold Cross Validation of the given neural network.

It splits the data set into k partitions that will be used to test k different models (trained on the other k - 1 partitions).

Parameters:

• nn – a neural network.
• optimizer – the optimizer used to train the neural network. Its signature must be compatible with the keys in train_args.
• x – a list of examples.
• y – a list with the target output of each example.
• k – the number of partitions (folds) of the data set.
• train_args – a dictionary whose keys are compatible with the arguments of optimizer.train().

Returns:

a dictionary with keys 'validation_scores' and 'train_scores' whose values are arrays of size k.

nnweaver.validation.random_search(nn_builder: Callable[dict, nnweaver.nn.NN], optimizer, x, y, train_args: Dict[str, Union[scipy.stats._distn_infrastructure.rv_frozen, typing.List[typing.Any]]], builder_args: Dict[str, Union[scipy.stats._distn_infrastructure.rv_frozen, typing.List[typing.Any]]], iterations, cv: Callable[[nnweaver.nn.NN, nnweaver.optimizers.Optimizer, object, object], Dict[str, List[float]]] = None)

Perform a random search through a space of training and neural network topology parameters.

An iteration of the random search is one of the possible combinations of the training and builder parameters. The first ones will be fed to optimizer.train(), while the second ones will be fed to nn_builder.

Parameters:

• nn_builder – a function that accepts a parameter builder_args and returns a neural network.
• optimizer – the optimizer to use in the random search. Its signature must be compatible with the keys in train_args.
• x – a list of examples.
• y – a list with the target output of each example.
• train_args – a dictionary whose keys are: (1) compatible with the arguments of optimizer.train(), and (2) associated with lists or (SciPy) random distributions that represent the subset of arguments to explore.
• builder_args – a dictionary whose keys are: (1) compatible with the arguments of nn_builder(), and (2) associated with lists or (SciPy) random distributions that represent the subset of arguments to explore.
• iterations – the number of iterations of the random search.
• cv – an optional cross validation method that is called at each iteration.

Returns:

a tuple with: (1) the best model found by the grid search, (2) its loss value, (3) arguments used to train it, and (4) arguments used to build it.

nnweaver.validation.splits_generator(x, y, groups)

Split a data set into partitions.

Parameters:

• x – a list of examples.
• y – a list with the target output of each example.
• groups – the list of the sizes of each partition (the sum of its elements must be len(x).