dnn_estimators Package

dnn_estimators Package

Package provides deep learning based estimators for the calibration of SDEs. The estimators can be categorized into two types of estimators:

All other modules provided by this package are helpers to enable an easier usage of the actual estimators.

adversarial_regressor Module

class sde_calibration.dnn_estimators.adversarial_regressor.AdversarialRegressor(data, gen_layers, discr_layers, **kwargs)

Bases: sde_calibration.estimators.RegressionEstimator

AdversarialRegression estimator. By performing an Ito-Taylor expansion of the random variable describing the process, the drift and diffusion coefficients can be approximated using conditional expectations. Therefore the time series data is split up in a way s.t. the aforementioned expectation is approximated. The approximation is performed by modelling the conditional expectation by a Generative Adversarial Network (GAN).

Parameters

  • data (pd.DataFrame) – The time series data that should be used for the estimation. The DataFrame needs to have the fields ‘t’ and ‘X’.

  • gen_layers (list) – Setup for the layers of the generator. For more information on the format of the layers list, c.f. NeuralNetwork.

  • discr_layers (list) – Setup for the layers of the discriminator. For more information on the format of the layers list, c.f. NeuralNetwork.

  • **kwargs –

    Additional arguments for the setup of the generator, discriminator and the preprocessor. Possible values are given as specified by the NeuralNetwork and Preprocessor classes.

Return type

None

Note

Note that the parameters provided by **kwargs that should be used in the NeuralNetwork class must have the prefix gen_ and discr_ to indicate the affiliation to the generator and discriminator, respectively.

__call__(x, training=False)

Evaluates the generator model for a given input. The input is first scaled according to the preprocessor that is initialized, the network mode is called and finally the inverse transform is applied to the outputs.

Parameters

  • x (np.ndarray) – Array of points for evaluating the model.

  • training (bool, optional) –

    Specifying whether training mode or prediction mode should be used.

    |default| False

Returns

Predictions of the generator.

Return type

tf.Tensor

compile(mode='drift', optimizers=[<tensorflow.python.keras.optimizer_v2.adam.Adam object>, <tensorflow.python.keras.optimizer_v2.adam.Adam object>], gan_type='vanilla', train_metrics=None, val_metrics=None, logger=None, tf_logging=None, update_steps=[1, 2])

Compiles the GAN estimator to be ready for fitting. This includes setting up the optimizer, metrics, and loggers as well as specifying the mode of estimation.

Parameters

  • mode (str, optional) –

    A string giving the mode of estimation. Possible options are:

  • optimizers ([tf.keras.optimizers.Optimizer, tf.keras.optimizers.Optimizer], optional) –

    Optimizers that should be used to optimize the loss function of the adversarial game.

    |default| [<tensorflow.python.keras.optimizer_v2.adam.Adam object>, <tensorflow.python.keras.optimizer_v2.adam.Adam object>]

  • gan_type (str, optional) –

    Type of the GAN that should be used. Possible options are:

    • vanilla

    • wasserstein

    • least_squares

    • kl

    • reverse_kl

      |default| 'vanilla'

  • train_metrics ([tf.keras.metrics.Metric, tf.keras.metrics.Metric], optional) – Metrics that are used to monitor the training progress. |default| None

  • val_metrics ([tf.keras.metrics.Metric, tf.keras.metrics.Metric, optional) –

    Metrics that are used to monitor the performance on the validation set.

    |default| None

  • logger (logging.Logger, optional) – Logger to be used for logging the training progress. |default| None

  • tf_logging (str, optional) –

    Specifies the path where TensorFlow logging should be performed. This includes logging the

    • weights of the network,

    • the training metric, and

    • the validation metric.

    The stored data can then be used for further visualization using TensorBoard. If None is provided, the aforementioned parameters are not logged.

    |default| None

  • update_steps (list) –

    Number of optimization steps that should be performed for the generator and discriminator, respectively.

    |default| [1, 2]

Return type

None

Raises

  • ValueError – If the provided string for the mode is invalid.

  • ValueError – If the string specifying the type of GAN is not any of the valid options.

Note

For all parameters that require a list to have the respective quantity for the generator as well for the discriminator, the list must contain the parameter for the generator in first place and the corresponding parameter for the discriminator in second place.

See also

For more information on the GAN versions used here and some other material, the following material is provided:

fit(epochs=1000)

Performs the adversarial training for a certain number of iterations.

Parameters

epochs (int, optional) – Number of epochs that should be performed. |default| 1000

Returns

The history of the training comprised in a dictionary.

Return type

dict

Raises

NotCompiledError – If this method is called before the estimator is compiled.

classmethod from_config(data, config)

Creates a new instance of this class from a dictionary containing the configuration of the network and the data required for the estimator.

Parameters

  • data (pd.DataFrame) – DataFrame containing the time series data.

  • config (dict) – Dictionary containing the configuration of the class.

Returns

New instance setup by the given configuration.

Return type

sde_calibration.dnn_estimators.NeuralNetwork

get_config()

Gives the configuration of the generator and discriminator networks that is required to create a new instance with the same setup.

Returns

Dictionary of all relevant parameters given to the constructor.

Return type

dict

get_gan_type()

Gives access to the GAN type of the estimator.

Returns

GAN type being one of the valid options for compilation.

Return type

str

get_num_parameters()

Returns the number of parameters in the model that is used by a certain estimator.

Returns

Number of parameters.

Return type

int

classmethod load(filepath)

Loads a saved model into an instance of the NerualNetwork class.

Parameters

filepath (str) – Path where the model is saved.

Returns

Instance of the neural network class.

Return type

sde_calibration.dnn_estimators.NeuralNetwork

Raises

FileNotFoundError – If either no config or no model can be found in the path specified.

Note

To make it possible to load a model it is necessary that under the specified file path, the files config.pkl and model.h5 exist.

save(filepath)

Saves the current neural network at in the local memory. This includes storing the configuration as well as the model.

Parameters

filepath (str) – Path where the neural network should be stored.

Return type

None

gan_losses Module

class sde_calibration.dnn_estimators.gan_losses.GANLoss(loss_type='vanilla')

Bases: object

Class to handle different GAN losses.

Parameters

loss_type (str, optional) –

Specifies the loss for a certain GAN type. Possible options are:

  • vanilla

  • least_squares

  • wasserstein

  • kl

  • reverse_kl

    |default| 'vanilla'

Return type

None

Raises

ValueError – If the GAN type for the loss is not a valid option.

discriminator(label_real, label_fake, **kwargs)

Evaluates the discriminator loss.

Parameters

  • label_real (tf.Tensor) – The response variables provided by the training set.

  • label_fake (tf.Tensor) – The response variables predicted by the generator.

  • **kwargs –

    Additional arguments that have to be provided if the loss is chosen to be the one of a Wasserstein GAN.

Returns

The value of the loss function.

Return type

tf.Tensor

Raises

KeyError – If the loss is computed for a Wasserstein GAN, the exception is thrown if the additional three arguments z_hat, discr, and penalty_coeff are missing. In case that any other GAN loss is chosen the exception is thrown if any keyword arguments are given.

generator(label_fake)

Evaluates the generator loss.

Parameters

label_fake (tf.Tensor) – The response variables predicted by the generator.

Returns

The value of the loss function.

Return type

tf.Tensor

invariant_density_estimator Module

class sde_calibration.dnn_estimators.invariant_density_estimator.InvariantDensityEstimator(data, layers, penalty_weight=20, **kwargs)

Bases: sde_calibration.dnn_estimators.neural_network.NeuralNetwork, sde_calibration.estimators.Estimator

Class for inferring drift and diffusion coefficients of an SDE based on the estimation of the invariant density \(\pi \colon \mathbb{R} \to \mathbb{R}\). The estimator trains a neural network to learn the invariant density of the underlying process. After this the estimated density is used in the stationary Fokker-Planck equation

\[b(x) \pi(x) = \frac{1}{2} \frac{\mathrm{d}}{\mathrm{d}x} (\sigma^2(x) \pi(x))\]

to infer either the drift coefficient using

\[b(x) = \frac{1}{2 \pi(x)} \frac{\mathrm{d}}{\mathrm{d}x} (\sigma^2(x) \pi(x))\]

or the diffusion coefficient by using

\[\sigma^2(x) = \frac{2}{\pi(x)} \int_{-\infty}^x b(\xi) \pi(\xi) \mathrm{d} \xi \ .\]

This does however mean that one of the two coefficients needs to be known apriori.

Parameters

  • data (pd.DataFrame) – The time series data that should be used for the estimation. The DataFrame needs to have the fields ‘t’ and ‘X’.

  • layers (list) – Setup for the layers the neural network. For more information on the format of the layers list, c.f. NeuralNetwork.

  • penalty_weight (float, optional) –

    Weight for the penalty term enforcing the neural network to behave like a density function and integrate to one.

  • **kwargs –

    Additional arguments for the setup of the neural network and the preprocessor. Possible values are given as specified by the NeuralNetwork and Preprocessor classes.

Return type

None

Raises

ValueError – If the weight of the penalty term is chosen to be smaller than zero.

See also

For more information on the Fokker-Planck equation, c.f.

Warning

Since this estimator is based on the invariant density, it requires the time series data to be stationary. If this cannot be guaranteed, the estimator will not yield any viable results.

__call__(x, training=False)

Evaluates the estimated density at the given points. It automatically applies the necessary in- and output scalings that are used to train the neural network approximating the invariant density.

Parameters

  • x (np.ndarray) – Array of points at which to evaluate the density function.

  • training (bool, optional) – Specifying whether training mode or prediction mode should be used. |default| False

Returns

Array of invariant density function estimates evaluated at the points specified by the input.

Return type

tf.Tensor

diffusion(x, drift_model)

Method to estimate the diffusion term given an array of points \(x\) based on the neural network approximating the invariant density and a given drift model.

Parameters

  • x (np.ndarray) – Array \(x\) specifying the points at which the diffusion term should be approximated.

  • drift_model (Callable) – Drift model of the underlying stochastic process.

Returns

Array of estimates of the diffusion term based on the stationary Fokker-Planck equation.

Return type

tf.Tensor

Raises

TypeError – If the given drift model is not callable.

drift(x, diffusion_model)

Method to estimate the drift term given an array of points \(x\) based on the neural network approximating the invariant density and a given diffusion model.

Parameters

  • x (np.ndarray) – Array \(x\) specifying the points at which the drift term should be approximated.

  • diffusion_model (Callable) – Diffusion model of the underlying stochastic process.

Returns

Array of estimates of the drift term based on the stationary Fokker-Planck equation.

Return type

tf.Tensor

Raises

TypeError – If the given diffusion model is not callable.

fit(epochs=1000, n_quad=10000)

Trains the model for a certain number of iterations.

Parameters

  • epochs (int, optional) – Number of iterations the optimizer should perform. |default| 1000

  • n_quad (int, optional) –

    Number of quadrature points that should be used to approximate the integral in the penalty term to enforce the network to behave like a density function.

    |default| 10000

Returns

The history of the training comprised in a dictionary.

Return type

dict

Raises

NotCompiledError – If this method is called before the estimator is compiled.

metrics Module

class sde_calibration.dnn_estimators.metrics.AverageLogLikelihood(name='average_log_likelihood', **kwargs)

Bases: tensorflow.python.keras.metrics.Metric

Metric that computes the average log likelihood given some log probabilities.

Parameters

  • name (str, optional) – Name of the metric |default| 'average_log_likelihood'

  • **kwargs –

    Additional keyword arguments that can be passed. More information of the keyword arguments can be found on the documentation of the tf.keras.metrics.Metric class.

Return type

None

result()

Computes and returns the metric value tensor.

Result computation is an idempotent operation that simply calculates the metric value using the state variables.

Return type

Tensor

update_state(log_probs)

Accumulates statistics for the metric.

Note: This function is executed as a graph function in graph mode. This means:

  1. Operations on the same resource are executed in textual order. This should make it easier to do things like add the updated value of a variable to another, for example.

  2. You don’t need to worry about collecting the update ops to execute. All update ops added to the graph by this function will be executed.

As a result, code should generally work the same way with graph or eager execution.

Args:

*args: **kwargs: A mini-batch of inputs to the Metric.

Return type

None

class sde_calibration.dnn_estimators.metrics.LogLikelihood(name='log_likelihood', **kwargs)

Bases: tensorflow.python.keras.metrics.Metric

Metric that computes the log likelihood given some log probabilities.

Parameters

  • name (str, optional) – Name of the metric |default| 'log_likelihood'

  • **kwargs –

    Additional keyword arguments that can be passed. More information of the keyword arguments can be found on the documentation of the tf.keras.metrics.Metric class.

Return type

None

result()

Computes and returns the metric value tensor.

Result computation is an idempotent operation that simply calculates the metric value using the state variables.

Return type

Tensor

update_state(log_probs)

Accumulates statistics for the metric.

Note: This function is executed as a graph function in graph mode. This means:

  1. Operations on the same resource are executed in textual order. This should make it easier to do things like add the updated value of a variable to another, for example.

  2. You don’t need to worry about collecting the update ops to execute. All update ops added to the graph by this function will be executed.

As a result, code should generally work the same way with graph or eager execution.

Args:

*args: **kwargs: A mini-batch of inputs to the Metric.

Return type

None

mixture_density_estimator Module

class sde_calibration.dnn_estimators.mixture_density_estimator.MixtureDensityEstimator(data, layers, **kwargs)

Bases: sde_calibration.dnn_estimators.neural_network.NeuralNetwork, sde_calibration.estimators.Estimator

Class for inferring drift and diffusion coefficients of an SDE based on estimting the transition density function \(p(\Delta t_i, X_{i+1} \vert X_i) \equiv p(X_{i+1} \vert X_i)\) by a neural network that learns the parameters of a Gaussian Mixture Model (GMM)

\[p_{\theta}(y \vert x) = \sum\limits_{k = 1}^K \alpha_k(x; \theta) \mathcal{N}(y; \mu_k(x; \theta), \Sigma_k(x; \theta)) \ .\]

Based on the learned transition density it is possible to infer the drift and diffusion coefficients approximately, by

\[b(x) \approx \sum\limits_{k = 1}^K \alpha_k(x; \theta) \frac{\mu_k(x; \theta) - x}{\Delta t_i}\]

and

\[\sigma^2(x) \approx \sum\limits_{k = 1}^K \alpha_k(x; \theta) \frac{(\mu_k(x; \theta) - x)^2 + \Sigma_k(x; \theta)}{\Delta t_i}\]

respectively.

Parameters

  • data (pd.DataFrame) – The time series data that should be used for the estimation. The DataFrame needs to have the fields ‘t’ and ‘X’.

  • layers (list) – Setup for the layers the neural network. For more information on the format of the layers list, c.f. NeuralNetwork.

  • **kwargs –

    Additional arguments for the setup of the neural network and the preprocessor. Possible values are given as specified by the NeuralNetwork and Preprocessor classes.

Return type

None

See also

For more information on mixture density networks, c.f.

__call__(x, training=False)

Evaluates the neural network estimating the coefficients of the GMM. It automatically applies the necessary in- and output scalings that are used to train the neural network approximating the invariant density.

Parameters

  • x (np.ndarray) – Array of points at which to evaluate the density function (nx1).

  • training (bool, optional) – Specifying whether training mode or prediction mode should be used. |default| False

Returns

List containing the weights of the mixture components (nx1), the means (nx1), and the covariance matrices (nx1x1).

Return type

[tf.Tensor, tf.Tensor, tf.Tensor]

diffusion(x)

Method to estimate the diffusion term given an array of points \(x\) based on the MDN approximating the transition density function.

Parameters

x (np.ndarray) – Array \(x\) specifying the points at which the diffusion term should be approximated.

Returns

Array of estimates of the diffusion term.

Return type

tf.Tensor

drift(x)

Method to estimate the drift term given an array of points \(x\) based on the MDN approximating the transition density function.

Parameters

x (np.ndarray) – Array \(x\) specifying the points at which the drift term should be approximated.

Returns

Array of estimates of the drift term.

Return type

tf.Tensor

fit(epochs=1000)

Trains the model for a certain number of iterations.

Parameters

epochs (int, optional) – Number of iterations the optimizer should perform. |default| 1000

Returns

The history of the training comprised in a dictionary.

Return type

dict

Raises

NotCompiledError – If this method is called before the estimator is compiled.

log_prob(x, y, training: bool = False)

Evaluates the logarithm of the estimated transition density function, i.e. \(\log(p(y \vert x))\) at the points specified by the arrays \(x\) and \(y\).

Parameters

  • x (np.ndarray) – Array of points \(x\) on which the density should be conditioned.

  • y (np.ndarray) – Array of points \(y\) used to compute the transition density.

  • training (bool, optional) – Specifying whether training mode or prediction mode should be used. |default| False

Returns

Array of values of the logarithm of the transition density function.

Return type

tf.Tensor

prob(x: numpy.ndarray, y: numpy.ndarray, training: bool = False) → tensorflow.python.framework.ops.Tensor

Evaluates the estimated transition density function \(p(y \vert x)\) at the points specified by the arrays \(x\) and \(y\).

Parameters

  • x (np.ndarray) – Array of points \(x\) on which the density should be conditioned.

  • y (np.ndarray) – Array of points \(y\) used to compute the transition density.

  • training (bool, optional) – Specifying whether training mode or prediction mode should be used. |default| False

Returns

Array of values of the transition density function.

Return type

tf.Tensor

sample(x: numpy.ndarray, num_samples: int, training: bool = False) → tensorflow.python.framework.ops.Tensor

Samples values from the estimated transition density function.

Parameters

  • x (np.ndarray) – Array of points \(x\) on which the transition density is conditioned.

  • num_samples (int) – Number of samples that should be drawn from the estimated transition density function.

  • training (bool, optional) – Specifying whether training mode or prediction mode should be used. |default| False

Returns

The specified number of samples \(y\) for which it holds that \(y \sim p(y \vert x)\)

Return type

tf.Tensor

neural_network Module

class sde_calibration.dnn_estimators.neural_network.NeuralNetwork(layers, hidden_activation='tanh', output_activation='linear', regularization_penalty=0.0, dropout_rate=0.0, name='Model')

Bases: object

Class providing an interface for handling feed forward neural networks.

Parameters

  • layers (list) –

    Setup for the layers of the neural network. The layers can be provided using two different styles. Each style is explained using an examples of usage:

    • [[1, 10], [50, 50], [100], [1]] creates a neural network with two inputs where the first input has dimensionality 1 and the second 10. Then both are separately fed through a dense layer, each containing 50 neurons. After that the results are concatenated fed through an additional fully connected layer with 100 neurons. Finally this is connected to the output having the dimensionality 1.

    • [2, 50, 100, 1] creates a neural network with one input of dimensionality 2. This input is then fed through a fully connected layer with 50 neurons. Afterwards another fully connected layer with 100 neurons follows. Finally this result is connected to the output having the dimensionality 1.

  • hidden_activation (str, optional) –

    Activation function that should be used for the hidden layers. Possible values for the activations are:

    • elu

    • linear

    • relu

    • sigmoid

    • softmax

    • softplus

    • swish

    • tanh

    Moreover, it is possible to pass any instance of the tf.keras.layers.Layer class. A list can be found here.

    |default| 'tanh'

  • output_activation (str, optional) –

    Activation function that should be applied after the output layer. Possible values for the activation are:

    • elu

    • linear

    • relu

    • sigmoid

    • softmax

    • softplus

    • swish

    • tanh

    Moreover, it is possible to pass any instance of the tf.keras.layers.Layer class. A list can be found here.

    |default| 'linear'

  • regularization_penalty (float, optional) –

    Specifies the strength of an \(L^2\)-regularizer that will be used on the weights and biases of the network. If zero is provided no regularization is performed.

  • dropout_rate (float, optional) –

    Dropout rate that should be used during training and prediction. If zero is provided no weights are dropped.

  • name (str, optional) – Name of the tf.keras.models.Model class that is created. |default| 'Model'

Return type

None

Raises

  • ValueError – If the layers list is too short, i.e. if not at least an input, one hidden layer, and one output are specified.

  • ValueError – If splitted layers are used and in one hidden layer more splits are requested than inputs are available.

  • ValueError – If the hidden activation is chosen different from the possible activations.

  • ValueError – If the output activation is chosen different from the possible activations.

  • ValueError – If the dropout rate does not lie in the interval [0, 1).

  • ValueError – If the weight of the regularizer that is provided is smaller than zero.

__call__(x, training=False)

Evaluates the neural networks core model for a given input.

Parameters

  • x (Union[np.ndarray, tf.Tensor]) – Array of points for evaluating the model.

  • training (bool, optional) –

    Specifying whether training mode or prediction mode should be used.

    |default| False

Returns

Predictions of the model.

Return type

tf.Tensor

compile(optimizer=<tensorflow.python.keras.optimizer_v2.adam.Adam object>, train_metric=None, val_metric=None, logger=None, tf_logging=None)

Compiles the neural network to be ready for fitting. This includes setting up the optimizer, metrics, and loggers.

Parameters

  • optimizer (tf.keras.optimizers.Optimizer, optional) – Optimizer that should be used to minimize the loss function. |default| <tensorflow.python.keras.optimizer_v2.adam.Adam object>

  • train_metric (tf.keras.metrics.Metric, optional) – Metric that is used to monitor the training progress. |default| None

  • val_metric (tf.keras.metrics.Metric, optional) –

    Metric that is used to monitor the performance on the validation set.

    |default| None

  • logger (logging.Logger, optional) – Logger to be used for logging the training progress. |default| None

  • tf_logging (str, optional) –

    Specifies the path where TensorFlow logging should be performed. This includes logging the

    • weights of the network,

    • the training metric, and

    • the validation metric.

    The stored data can then be used for further visualization using TensorBoard. If None is provided, the aforementioned parameters are not logged.

    |default| None

Return type

None

fit(train_dataset, validation_dataset=None, epochs=1000)

Trains the model for a certain number of iterations given a dataset.

Parameters

  • train_dataset (tf.data.Dataset) – TensorFlow dataset used for the training.

  • validation_dataset (tf.data.Dataset, optional) –

    TensorFlow dataset used for the validating the training progress.

    |default| None

  • epochs (int, optional) – Number of iterations the optimizer should perform. |default| 1000

Returns

The history of the training comprised in a dictionary.

Return type

dict

Raises

NotCompiledError – If this method is called before the neural network is compiled.

classmethod from_config(config)

Creates a new instance of this class from a dictionary containing the configuration of the network.

Parameters

config (dict) – Dictionary containing the configuration of the class.

Returns

New instance setup by the given configuration.

Return type

sde_calibration.dnn_estimators.NeuralNetwork

get_config()

Gives the configuration of the neural network class that is required to create a new instance with the same setup.

Returns

Dictionary of all relevant parameters given to the constructor.

Return type

dict

get_losses()

Gives access to all the losses that are defined by the model and not obtained via gradients. An example includes regularization losses.

Returns

List of losses, where each entry in the list contains the losses of one layer.

Return type

list

get_num_parameters()

Gives the number of parameters in the neural network.

Returns

Number of parameters.

Return type

int

classmethod load(filepath)

Loads a saved model into an instance of the NerualNetwork class.

Parameters

filepath (str) – Path where the model is saved.

Returns

Instance of the neural network class.

Return type

sde_calibration.dnn_estimators.NeuralNetwork

Raises

FileNotFoundError – If either no config or no model can be found in the path specified.

Note

To make it possible to load a model it is necessary that under the specified file path, the files config.pkl and model.h5 exist.

plot_model(save_path, show_shapes=True)

Converts the neural network internal model to dot format and saves it to a file.

Parameters

  • save_path (str) – File name of the image created.

  • show_shapes (bool, optional) – Specifies whether to display shape information or not. |default| True

Returns

A Jupyter notebook Image object if Jupyter is installed. This enables in-line display of the model plots in notebooks.

save(filepath)

Saves the current neural network at in the local memory. This includes storing the configuration as well as the model.

Parameters

filepath (str) – Path where the neural network should be stored.

Return type

None

neural_regressor Module

class sde_calibration.dnn_estimators.neural_regressor.NeuralRegressor(data, layers, **kwargs)

Bases: sde_calibration.dnn_estimators.neural_network.NeuralNetwork, sde_calibration.estimators.RegressionEstimator

Class combining the functionalities of the NeuralNetwork and the RegressionEstimator classes. It is the baseline for a deep learning regression-based estimator.

Parameters

  • data (pd.DataFrame) – The time series data that should be used for the estimation. The DataFrame needs to have the fields ‘t’ and ‘X’.

  • layers (list) – Setup for the layers the neural network. For more information on the format of the layers list, c.f. NeuralNetwork.

  • **kwargs –

    Additional arguments for the setup of the neural network and the preprocessor. Possible values are given as specified by the NeuralNetwork and Preprocessor classes.

Return type

None

__call__(x, training=False)

Evaluates the neural networks core model for a given input. The input is first scaled according to the preprocessor that is initialized, the network mode is called and finally the inverse transform is applied to the outputs.

Parameters

  • x (np.ndarray) – Array of points for evaluating the model.

  • training (bool, optional) –

    Specifying whether training mode or prediction mode should be used.

    |default| False

Returns

Predictions of the model.

Return type

tf.Tensor

compile(mode='drift', optimizer=<tensorflow.python.keras.optimizer_v2.adam.Adam object>, train_metric=None, val_metric=None, logger=None, tf_logging=None)

Compiles the neural network to be ready for fitting. This includes setting up the optimizer, metrics, and loggers as well as specifying the mode of estimation.

Parameters

  • mode (str, optional) –

    A string giving the mode of estimation. Possible options are:

  • optimizer (tf.keras.optimizers.Optimizer, optional) – Optimizer that should be used to minimize the loss function. |default| <tensorflow.python.keras.optimizer_v2.adam.Adam object>

  • train_metric (tf.keras.metrics.Metric, optional) – Metric that is used to monitor the training progress. |default| None

  • val_metric (tf.keras.metrics.Metric, optional) –

    Metric that is used to monitor the performance on the validation set.

    |default| None

  • logger (logging.Logger, optional) – Logger to be used for logging the training progress. |default| None

  • tf_logging (str, optional) –

    Specifies the path where TensorFlow logging should be performed. This includes logging the

    • weights of the network,

    • the training metric, and

    • the validation metric.

    The stored data can then be used for further visualization using TensorBoard. If None is provided, the aforementioned parameters are not logged.

    |default| None

Return type

None

Raises

ValueError – If the provided string for the mode is invalid.

fit(epochs=1000)

Trains the model for a certain number of iterations.

Parameters

epochs (int, optional) – Number of iterations the optimizer should perform. |default| 1000

Returns

The history of the training comprised in a dictionary.

Return type

dict

Raises

NotCompiledError – If this method is called before the estimator is compiled.

non_parametric_regressor Module

class sde_calibration.dnn_estimators.non_parametric_regressor.NonParametricRegressor(data, layers, n_quad=50, **kwargs)

Bases: sde_calibration.dnn_estimators.neural_regressor.NeuralRegressor

Estimator that uses the power of a neural network as a universal approximator in order to train a model that is that is approximating the function \(m(\cdot)\) satisfying

\[\min\limits_{m \in L^2(\Omega, \mathcal{F}_X, \mathbb{P})} \int\limits_V \mathbb{E}[(Y - m(X))^2 \vert X = x] \mathrm{d} x \ ,\]

where \(X\) and \(Y\) are random variables in \(L^2(\Omega, \mathcal{F}, \mathbb{P})\), and \(V\) is the convex hull of the training dataset. An approximation then yields the loss function

\[L(\theta) = \sum\limits_{k = 1}^K w_k \sum_{i = 1}^n \frac{k_{h}(\xi_k - X_i)}{\sum_{j = 1}^n k_{h}(\xi_k - X_j)} (Y_i - \hat{m}(X_i; \xi_k, \theta))^2 \ ,\]

where \(X_i\) and \(Y_i\) are the observed training data points, \(\{(w_k, \xi_k)\}_{k=1}^K\) is a set of quadrature weights and points, respectively, \(k_h\) is a kernel function with bandwidth \(h\), and \(\hat{m}\) is the approximation by a neural network. This idea is inspired by the idea of local polynomial regression, hence it is called a non-parametric deep learning based approach.

Parameters

  • data (pd.DataFrame) – The time series data that should be used for the estimation. The DataFrame needs to have the fields ‘t’ and ‘X’.

  • layers (list) – Setup for the layers the neural network. For more information on the format of the layers list, c.f. NeuralNetwork.

  • n_quad (int, optional) –

    Number of quadrature points that should be used in the loss function for approximating the integral.

  • **kwargs –

    Additional arguments for the setup of the neural network and the preprocessor. Possible values are given as specified by the NeuralNetwork and Preprocessor classes.

Return type

None

Warning

The larger the number of quadrature points the higher the effort in the training, since in every iteration the integration has to be performed. Therfore, it is advised to not choose the number of quadrature points too large.

fit(epochs=1000)

Trains the model for a certain number of iterations.

Parameters

epochs (int, optional) – Number of iterations the optimizer should perform. |default| 1000

Returns

The history of the training comprised in a dictionary.

Return type

dict

Raises

NotCompiledError – If this method is called before the estimator is compiled.

parametric_regressor Module

class sde_calibration.dnn_estimators.parametric_regressor.ParametricRegressor(data, layers, **kwargs)

Bases: sde_calibration.dnn_estimators.neural_regressor.NeuralRegressor

Class that estimates a conditional expectation that approximates the drift or diffusion coefficient, respectively. It uses a standard mean squared error (MSE) loss. This is similar to the idea of standard linear regression, which is a standard parametric approach. Due to the similarity to the parametric linear regression approach the name of this estimator is chosen to be named as a deep learning based parametric regression approach although by the power of the neural network as a universal approximator it is rather a semi-parametric approach.

Parameters

  • data (pd.DataFrame) – The time series data that should be used for the estimation. The DataFrame needs to have the fields ‘t’ and ‘X’.

  • layers (list) – Setup for the layers the neural network. For more information on the format of the layers list, c.f. NeuralNetwork.

  • **kwargs –

    Additional arguments for the setup of the neural network and the preprocessor. Possible values are given as specified by the NeuralNetwork and Preprocessor classes.

Return type

None

sde_informed_gan Module

class sde_calibration.dnn_estimators.sde_informed_gan.SDEInformedGAN(data, **kwargs)

Bases: sde_calibration.estimators.Estimator

Estimator based a Generative Adversarial Network (GAN). The generator is build of two neural networks, one approximating the drift coefficient and the other one approximating the diffusion coefficient of the SDE describing a diffusion process. The SDE model is build into the generator s.t. the generator finally makes a prediciton of the next element in the time series. Therefore it is implicitly approximating the transition density function \(p(X_{i+1} \vert X_i)\) by learning how to draw samples \(x\), where \(x \sim p(X_{i+1} \vert X_i)\). By learning this distribution the drift and diffusion coefficient are inferred indirectly during the training process.

Parameters

  • data (pd.DataFrame) – The time series data that should be used for the estimation. The DataFrame needs to have the fields ‘t’ and ‘X’.

  • gen_n_separate_layers (int, optional) – Number of split layers in the generator network.

  • gen_n_concatenated_layers (int, optional) – Number of fully connected layers after concatenation of the input and the latent variables for the generator.

  • gen_n_hidden_units (int, optional) – Number of neurons in the hidden layers of the generator networks.

  • gen_n_latent_variables (int, optional) – Number of latent variables used in the generator networks.

  • gen_n_steps (int, optional) – Number of intermediate integration steps to solve the SDE for one timestep.

  • gen_integrator (str, optional) –

    Integration scheme, that should be used to solve the SDE. Possilbe options are:

    • euler

    • milstein

  • gen_diffusion_prior (bool, optional) – Whether the diffusion model should be assumed to be a matrix to be estimated or a neural network.

  • gen_diagonal_diffusion (bool, optional) – Whether the diffusion model is approximated as a diagonal matrix with unknown entries or the matrix should be full with unknown entries. Only applies if gen_diffusion_prior is set to True.

  • gen_prior (Callable, optional) – Prior that is used to sample the latent variables of the generator form.

  • discr_n_separate_layers (int, optional) – Number of split layers in the discriminator network.

  • discr_n_concatenated_layers (int, optional) – Number of fully connected layers after concatenation of the real and fake inputs for the discriminator.

  • discr_n_hidden_units (int, optional) – Number of neurons in the hidden layers of the discriminator network.

  • **kwargs –

    Additional arguments for the setup of the neural network and the preprocessor. Possible values for the preprocessor of the estimator are given by the Preprocessor classes.

Return type

None

Note

For the defaults of the generator parameters c.f. SDEInformedGenerator.

__call__(x, training=False)

Evaluates the generator model for a given input, that is, samples from the transition density are drawn. The input is first scaled according to the preprocessor that is initialized, the network mode is called and finally the inverse transform is applied to the outputs.

Parameters

  • x (np.ndarray) – Array of points for evaluating the model.

  • training (bool, optional) –

    Specifying whether training mode or prediction mode should be used.

    |default| False

Returns

Samples drawn from the transition density estimated by the generator network.

Return type

tf.Tensor

compile(optimizers=[<tensorflow.python.keras.optimizer_v2.adam.Adam object>, <tensorflow.python.keras.optimizer_v2.adam.Adam object>], gan_type='vanilla', train_metrics=None, val_metrics=None, tf_logging=None, update_steps=[1, 2])

Compiles the GAN estimator to be ready for fitting. This includes setting up the optimizers, metrics, and loggers.

Parameters

  • optimizers ([tf.keras.optimizers.Optimizer, tf.keras.optimizers.Optimizer], optional) –

    Optimizers that should be used to optimize the loss function of the adversarial game.

    |default| [<tensorflow.python.keras.optimizer_v2.adam.Adam object>, <tensorflow.python.keras.optimizer_v2.adam.Adam object>]

  • gan_type (str, optional) –

    Type of the GAN that should be used. Possible options are:

    • vanilla

    • wasserstein

    • least_squares

    • kl

    • reverse_kl

      |default| 'vanilla'

  • train_metrics ([tf.keras.metrics.Metric, tf.keras.metrics.Metric], optional) – Metrics that are used to monitor the training progress. |default| None

  • val_metrics ([tf.keras.metrics.Metric, tf.keras.metrics.Metric, optional) –

    Metrics that are used to monitor the performance on the validation set.

    |default| None

  • logger (logging.Logger) – Logger to be used for logging the training progress.

  • tf_logging (str, optional) –

    Specifies the path where TensorFlow logging should be performed. This includes logging the

    • weights of the network,

    • the training metric, and

    • the validation metric.

    The stored data can then be used for further visualization using TensorBoard. If None is provided, the aforementioned parameters are not logged.

    |default| None

  • update_steps (list) –

    Number of optimization steps that should be performed for the generator and discriminator, respectively.

    |default| [1, 2]

Return type

None

Raises

ValueError – If the string specifying the type of GAN is not any of the valid options.

Note

For all parameters that require a list to have the respective quantity for the generator as well for the discriminator, the list must contain the parameter for the generator in first place and the corresponding parameter for the discriminator in second place.

See also

For more information on the GAN versions used here and some other material, the following material is provided:

diffusion(x)

Method extracts the inferred diffusion coefficient from the generator and evaluates it on the points that are given.

Parameters

x (np.ndarray) – Array of inputs at which the estimated diffusion term should be evaluated.

Returns

Array of approximations of the diffusion term estimated by the generator.

Return type

tf.Tensor

drift(x)

Method extracts the inferred drift coefficient from the generator and evaluates it on the points that are given.

Parameters

x (np.ndarray) – Array of inputs at which the estimated drift term should be evaluated.

Returns

Array of approximations of the drift term estimated by the generator.

Return type

tf.Tensor

fit(epochs=2000, n_paths=2, show_sample_progress=True)

Performs the adversarial training for a certain number of iterations. During the training trajectories of the SDE are sampled and stored in the history that is returned.

Parameters

  • epochs (int, optional) – Number of epochs that should be performed. |default| 2000

  • n_paths (int, optional) – Number of trajectories that should be sampled. |default| 2

  • show_sample_progress (bool, optional) –

    Whether the progress of the sampling procedure should be displayed or not.

    |default| True

Returns

The history of the training comprised in a dictionary.

Return type

dict

Raises

NotCompiledError – If this method is called before the estimator is compiled.

get_config()

Gives the configuration of the neural network class that is required to create a new instance with the same setup. This includes the parameters for the generator networks, the discriminator networks as well as the parameters for the preprocessor.

Returns

Dictionary of all relevant parameters given to the constructor.

Return type

dict

get_gan_type()

Gives access to the GAN type of the estimator.

Returns

GAN type being one of the valid options for compilation.

Return type

str

get_num_parameters()

Gives the number of parameters in the whole GAN, i.e the generator networks as well as the discriminator network.

Returns

Number of parameters.

Return type

int

sample_paths(n_paths, X0, T, show_progress=True)

Draw several trajectory of the generator model.

Parameters

  • n_paths (int) – Number of trajectories that should be sampled form the generator model.

  • X0 (float) – Initial value at which to start sampling.

  • T (float) – Final time at which to stop sampling from a trajectory. Note that it is assumed that \(t_0 = 0\).

  • show_progress (bool, optional) –

    Whether the progress in sampling the trajectories should be displayed or not.

    |default| True

Returns

Array that contains the trajectories (in the first component of the array). The different paths are accessed by the second index of the array.

Return type

np.ndarray

save_generator(filepath)

Method to save the generator model into a certain directory. The saved model can later be loaded into a SDEInformedGenerator object by calling the :method:`load <.sde_informed_generator.SDEInformedGenerator.load>` method.

Parameters

filepath (str) – Directory for saving the generator model.

Return type

None

sde_informed_generator Module

class sde_calibration.dnn_estimators.sde_informed_generator.Diffusion(dimensions, diagonal=False, name=None)

Bases: tensorflow.python.module.module.Module

Class that builds upon the TensorFlow module. It implements a diffusion model based on a matrix. The elements of the rectangular matrix are then variables that are trainable, i.e. can be changed by the optimizer in the training process.

Parameters

  • dimensions (int) – Number of dimensions \(d\) that should be used to build the matrix \(\sigma \in \mathbb{R}^{d \times d}\).

  • diagonal (bool, optional) –

    Whether the matrix should be diagonal or not. If True then \(\sigma = \mathrm{diag}(\sigma_1, \ldots, \sigma_d)\).

    |default| False

  • name (str, optional) – Name of the tf.Module. |default| None

Return type

None

__call__(x)

Returns the matrix with batch shape of the given input.

Parameters

x (np.ndarray) – Array \(x\) which gives the number of batches that should be returned to the matrix. In order to be conform to, e.g. the neural networks that are used. The input are the points of evaluation instead of the batch shape directly.

Returns

Return the diffusion matrix in batched form, i.e. if \(x \in \mathbb{R}^{n \times d}\), where \(n\) is the batch size and \(d\) the number of dimensions. Then the output will be a tensor that lies in \(\mathbb{R}^{n \times d \times d}\). That means the diffusion matrix is repeated \(n\) times.

Return type

tf.Tensor

Raises

ValueError – If the number of dimensions of the given array \(x\) does not match the number of dimensions of the diffusion matrix.

classmethod from_config(config)

Creates a new instance of this class from a dictionary containing the configuration of the network and the data required for the estimator.

Parameters

config (dict) – Dictionary containing the configuration of the class.

Returns

New instance setup by the given configuration.

Return type

sde_calibration.dnn_estimators.sde_informed_generator.Diffusion

get_config()

Gives the configuration of the diffusion model that is required to create a new instance with the same setup.

Returns

Dictionary of all relevant parameters given to the constructor.

Return type

dict

classmethod load(filepath)

Loads a saved diffusion model into an instance of the Diffusion class.

Parameters

filepath (str) – Path where the model is saved.

Returns

Instance of the Diffusion class.

Return type

sde_calibration.dnn_estimators.sde_informed_generator.Diffusion

Raises

FileNotFoundError – If either no config or no model can be found in the path specified.

Note

To make it possible to load a model it is necessary that under the specified file path, the file diffusion.pkl exists.

save(filepath)

Saves the diffusion model to a file.

Parameters

filepath (str) – Path where the model should be stored.

Return type

None

class sde_calibration.dnn_estimators.sde_informed_generator.SDEInformedGenerator(n_inputs, n_outputs, n_separate_layers=1, n_concatenated_layers=3, n_hidden_units=50, n_latent_variables=50, n_steps=2, integrator='euler', diffusion_prior=False, diagonal_diffusion=False, prior=<function SDEInformedGenerator.<lambda>>)

Bases: object

Class that build a generator for training in a GAN framework. This generator is aimed to calibrate a diffusion process while simultaneously estimating the transition density function implicitly by learning how to draw samples from it. This model is aware of an SDE model and approximates the drift and diffusion coefficient as a neural network, respectively.

Parameters

  • n_inputs (int) – Number of inputs in the neural networks, i.e. the dimensionality of the input data.

  • n_outputs (int) – Number of outputs in the neural network, i.e. the dimensionality of the output data.

  • n_separate_layers (int, optional) –

    Number of layers that the input and the latent variables should be fed through separate fully connected layers.

  • n_concatenated_layers (int, optional) –

    Number of fully connected layers that should be applied after the concatenation of the input and the latent variables.

  • n_hidden_units (int, optional) –

    Number of neurons in the hidden layers. They are chosen to be the same in all hidden layers.

  • n_latent_variables (int, optional) –

    Number of latent variables used in the generator networks.

  • integrator (str, optional) –

    Integration scheme, that should be used to solve the SDE. Possilbe options are:

  • diffusion_prior (bool, optional) –

    Whether the diffusion model should be assumed to be a matrix to be estimated or a neural network.

    |default| False

  • diagonal_diffusion (bool, optional) –

    Whether the diffusion model is approximated as a diagonal matrix with unknown entries or the matrix should be full with unknown entries. Only applies if diffusion_prior is set to True.

    |default| False

  • prior (Callable, optional) – Prior that is used to sample the latent variables of the form. |default| <function SDEInformedGenerator.<lambda>>

Param

n_steps: Number of intermediate integration steps to step from time \(t_i\) to time \(t_{i+1}\).

Return type

None

Raises

  • ValueError – If the inegrator chosen is not a valid option.

  • TypeError – If the prior function for the latent variables is not callable.

__call__(X: numpy.ndarray, training: bool = False) → tensorflow.python.framework.ops.Tensor

Draws a sample from the generator model given the values that are passed.

Parameters

  • X (np.ndarray) – Array of the values which is conditioned on for drawing the samples.

  • training (bool, optional) –

    Specifying whether training mode or prediction mode should be used.

    |default| False

Returns

Array of samples drawn from the transition density.

Return type

tf.Tensor

diffusion(X)

Method extracts the inferred diffusion coefficient and evaluates it on t he points that are given.

Parameters

x (np.ndarray) – Array of inputs at which the estimated diffusion term should be evaluated.

Returns

Array of approximations of the diffusion term estimated by the generator.

Return type

tf.Tensor

drift(X)

Method extracts the inferred drift coefficient and evaluates it on the points that are given.

Parameters

X (np.ndarray) – Array of inputs at which the estimated drift term should be evaluated.

Returns

Array of approximations of the drift term estimated by the generator.

Return type

tf.Tensor

classmethod from_config(config)

Creates a new instance of this class from a dictionary containing the configuration of the network and the data required for the estimator.

Parameters

config (dict) – Dictionary containing the configuration of the class.

Returns

New instance setup by the given configuration.

Return type

sde_calibration.dnn_estimators.SDEInformedGenerator

get_config()

Gives the configuration of the generator and discriminator networks that is required to create a new instance with the same setup.

Returns

Dictionary of all relevant parameters given to the constructor.

Return type

dict

get_num_parameters()
classmethod load(filepath)

Loads a saved generator_model into an instance of the NerualNetwork class.

Parameters

filepath (str) – Path where the model is saved.

Returns

Instance of the neural network class.

Return type

sde_calibration.dnn_estimators.NeuralNetwork

Raises

FileNotFoundError – If either no config or no model can be found in the path specified.

Note

To make it possible to load a model it is necessary that under the specified file path, the files config.pkl and model.h5 exist.

sample_paths(num_paths, X0, dt, T, display=None)

Draw a trajectory of the generator model.

Parameters

  • num_paths (int) – Number of trajectories that should be sampled form the generator model.

  • X0 (float) – Initial value at which to start sampling.

  • dt (float) – Time step that should be used for sampling the trajectory.

  • T (float) – Final time at which to stop sampling from a trajectory. Note that it is assumed that \(t_0 = 0\).

  • display (Callable, optional) –

    Function that should be used to display the progresss of the sampling process. If None is given, then the sampling occurs silently

    |default| None

Returns

Array that contains the trajectories (in the first component of the array). The different paths are accessed by the second index of the array.

Return type

np.ndarray

save(filepath)

Saves the current generator network in the local memory. This includes storing the configuration as well as the internal models.

Parameters

filepath (str) – Path where the generator should be stored.

Return type

None

Subpackages