---

Hessian Matrix Evaluation Routines.
[Nodes]

---

Detailed Description

The routines presented in this module deal with the computation of the second derivatives of the error, given by

where both indexes run over all parameters . These elements form the Hessian Matrix, which plays a very important role in several aspects of neural computing, like, per example:

• Some parameter optimization algorithms use the information of the error surface's second-order properties.
• Pruning algorithms use the Hessian in order to determine the relevance of the parameters. Although the Hessian Matrix does hold important information about a modelling function, its computation can be very expensive, both in time and memory requirements.

It is important to note that some of these algorithms assume special error functions. Nevertheless, sometimes this assumption does not heavilly contradict the situation encountered in e.g. pruning procedures using another cost function.

Functions
int	gnn_hessian_vector_mult (gnn_line *line, const gsl_vector *v, gsl_vector *vH, double eps)
	Fast multiplication by the Hessian matrix.
int	gnn_hessian_diagonal (gnn_line *line, gsl_vector *H, double eps)
	Finite differences diagonal Hessian matrix approximation.
int	gnn_hessian_levenberg_marquadt_init (gnn_grad *grad, gsl_matrix *H)
	Initializes the Levenberg-Marquadt Hessian approximation.
int	gnn_hessian_levenberg_marquadt (gnn_grad *grad, gsl_matrix *H, size_t k)
	Levenberg-Marquadt Hessian approximation.
int	gnn_hessian_inverse_init (gnn_grad *grad, gsl_matrix *H, double alpha)
	Levenberg-Marquadt Hessian Inverse approximation initialization.
int	gnn_hessian_inverse (gnn_grad *grad, gsl_matrix *H, size_t k)
	Levenberg-Marquadt Hessian Inverse approximation.
int	gnn_hessian_finite_differences (gnn_line *line, gsl_matrix *H, double eps)
	Finite differences Hessian matrix approximation.

---

Function Documentation

int gnn_hessian_diagonal (  gnn_line *    line, gsl_vector *    H, double    eps )

This numerical efficient method computes the diagonal values of the Hessian matrix using the central differentes method, given by where is a small value that gives the required precision, since the error resulting from this approximation is . The same as above written in matrix notation is: where is the Hessian matrix and . Note that this requires a single forward and backward step for evaluation, since the assumption of the diagonal Hessian is equivalent to an uncorrelated parameter space. Parameters: grad  A pointer to a Evaluations. buffer. H  A pointer to a vector of size , where is the number of parameters involved in the gnn_node. The resulting diagonal terms of the Hessian will be stored in this vector. eps  The required precision . Returns: Returns 0 if succeeded. Definition at line 184 of file gnn_hessian.c.

int gnn_hessian_finite_differences (  gnn_line *    line, gsl_matrix *    H, double    eps )

This numerical efficient method computes the values of the Hessian matrix using the central differentes method, given by where is a small value that gives the required precision, since the error resulting from this approximation is . Parameters: line  A pointer to a Evaluations. buffer. H  A pointer to a matrix of size , where is the number of parameters involved in the gnn_node. The resulting Hessian will be stored in this matrix. eps  The required precision . Returns: Returns 0 if succeeded. Definition at line 534 of file gnn_hessian.c.

int gnn_hessian_inverse (  gnn_grad *    grad, gsl_matrix *    H, size_t    k )

This function computes the k-th iteration of the outer-product inverse Hessian approximation, using the k-th pattern. The outer product approximation used in Hessian Matrix Evaluation Routines. can be used to develop an efficient procedure for computing the inverse of the Hessian matrix. Recalling the formula for the outer product approximation where , it is easy to see that this leads to a sequential form for computing the Hessian: Using the following matrix identity where is the identity matrix, and setting , and , then the previous formula can be rewritten in order to obtain an iterative procedure for building the inverse of the Hessian: The procedure can be initialized by , where is a small value (because the procedure actually seeks to approximate ). It is important to note that, as in the case for Hessian Matrix Evaluation Routines., this approximation is only valid for a sum-of-squares error function. Warning: Evaluation??? Shouldn't it be handled completely by a gnn_eval or gnn_grad?? Parameters: grad  A pointer to a Evaluations. buffer. H  A pointer to a matrix of size , where is the number of parameters involved in the gnn_node. The resulting inverse Hessian will be stored in this matrix. k  The index of the pattern to be used. Returns: Returns 0 if succeeded. Definition at line 450 of file gnn_hessian.c.

int gnn_hessian_inverse_init (  gnn_grad *    grad, gsl_matrix *    H, double    alpha )

This function initializes the outer product inverse approximation routine. For further details, please refer to Hessian Matrix Evaluation Routines.. Parameters: grad  A pointer to a Evaluations. buffer. H  A pointer to a matrix of size , where is the number of parameters involved in the gnn_node. The resulting inverse Hessian will be stored in this matrix. alpha  A small value for . Returns: Returns 0 if succeeded. Definition at line 376 of file gnn_hessian.c.

int gnn_hessian_levenberg_marquadt (  gnn_grad *    grad, gsl_matrix *    H, size_t    k )

This function computes the k-th iteration of the Levenberg-Marquadt Hessian approximation, using the k-th pattern. The Levenberg-Marquadt approximation (also known as the outer product approximation) of the Hessian matrix is given by where the sum runs over all pattern outputs , . It is very important to note that this approximation is only valid for an error function of the form that is, for an sum-of-squares error function (and its equivalent forms). The previous formula can be written in matrix form, giving the sequential procedure for building the Hessian: which is the way used by libgnn. Warning: Evaluation??? Shouldn't it be handled completely by a gnn_eval or gnn_grad?? Parameters: grad  A pointer to a Evaluations. buffer. H  A pointer to a matrix of size , where is the number of parameters involved in the gnn_node. The resulting Hessian will be stored in this matrix. k  The index of the pattern to be used. Returns: Returns 0 if succeeded. Definition at line 309 of file gnn_hessian.c.

int gnn_hessian_levenberg_marquadt_init (  gnn_grad *    grad, gsl_matrix *    H )

This function initializes the Hessian for the Levenberg-Marquadt approximation. For further details, see Hessian Matrix Evaluation Routines.. Parameters: grad  A pointer to a Evaluations. buffer. H  A pointer to a matrix of size , where is the number of parameters involved in the gnn_node. The resulting Hessian will be stored in this matrix. Returns: Returns 0 if succeeded. Definition at line 250 of file gnn_hessian.c.

int gnn_hessian_vector_mult (  gnn_line *    line, const gsl_vector *    v, gsl_vector *    vH, double    eps )

Some applications do not require the Hessian explicitally. Instead, what really is needed is a fast method for computing , where is the Hessian matrix (which is assumed to be symmetric, so that ) and is a vector. This numerical method computes this multiplication using central differences: where is a small value that gives the required precision, since the error resulting from this approximation is . Parameters: grad  A pointer to a Evaluations. buffer. v  A pointer to the vector . vH  A pointer to a vector of size , where is the number of parameters involved in the gnn_node. The resulting multiplication will be stored in this vector. eps  The required precision . Returns: Returns 0 if succeeded. Definition at line 97 of file gnn_hessian.c.

---