---

gnn_lmbfgs : Limited Memory BFGS Algorithm.
[gnn_trainer : Trainers for Models.]

---

Detailed Description

This trainer implements the Limited Memory Broyden-Fletcher-Goldfarb-Shanno (LMBFGS) optimization algorithm. It is a so-called Quasi-Newton method. The parameters are updated using the formula:

where is a search direction given by:

and where the following vectors have been defined:

and is the -th gradient . The step size is found using a line-minimization, which can be one of those available in gnn_line_search. The coefficients and are defined as

The difference with the BFGS method lies in the fact that the expensive approximation and storage of the inverse Hessian matrix has been replaced by the unit matrix. The resulting algorithm takes only space, where is the number of parameters of the Gradient Machine. The algorithm is comparable to the conjugate gradients algorithm (refer to gnn_conjugate_gradient : Conjugate Gradient Descent Algorithm.).

Defines
#define	GNN_LMBFGS_TOL   0.0001
	Default precision for line search procedures.
#define	GNN_LMBFGS_STEP   0.1
	Default bracketing step taken by the line search procedures.
#define	GNN_LMBFGS_RESTART   100
	Default restart.
#define	GNN_LMBFGS_ALPHA   gnn_line_search_brent
	Default line search procedure.
Typedefs
typedef double(*	gnn_lmbfgs_beta )(gnn_trainer *trainer)
	Type for evaluation procedures.
typedef _gnn_lmbfgs	gnn_lmbfgs
	Conjugate gradients trainer structure.
Functions
int	gnn_lmbfgs_reset (gnn_trainer *trainer)
	The trainer's "reset" implementation.
int	gnn_lmbfgs_iteration (gnn_lmbfgs *bf, double *A, double *B)
	Computes the A and B coefficients.
void	gnn_lmbfgs_destroy (gnn_trainer *trainer)
	The trainers "destroy" implementation.
gnn_trainer *	gnn_lmbfgs_new (gnn_node *node, gnn_criterion *crit, gnn_dataset *data)
	Creates a new LMBFGS trainer.
int	gnn_lmbfgs_set_tol (gnn_trainer *trainer, double tol)
	Sets the precision tolerance for the line search procedure.
double	gnn_lmbfgs_get_tol (gnn_trainer *trainer)
	Gets the tolerance for the line search procedure.
int	gnn_lmbfgs_set_step (gnn_trainer *trainer, double step)
	Sets the initial step for the interval bracketing procedure.
double	gnn_lmbfgs_get_step (gnn_trainer *trainer)
	Gets the initial step for the interval bracketing procedure.
int	gnn_lmbfgs_set_restart (gnn_trainer *trainer, size_t restart)
	Sets the number of iterations before restarting.
size_t	gnn_lmbfgs_get_restart (gnn_trainer *trainer)
	Gets the number of iterations before reinitializing the direction.
int	gnn_lmbfgs_set_line_search (gnn_trainer *trainer, gnn_line_search_type lsearch)
	Sets the line search procedure.
gnn_line_search_type	gnn_lmbfgs_get_alpha (gnn_trainer *trainer)
	Gets the installed line search procedure.

---

Define Documentation

#define GNN_LMBFGS_ALPHA   gnn_line_search_brent

This is the default line search procedure used by the conjugate gradients algorithm. Definition at line 77 of file gnn_lmbfgs.h.

#define GNN_LMBFGS_RESTART   100

This constant defines the default number of iterations that the conjugate gradients algorithm takes before restarting the algorithm. Definition at line 68 of file gnn_lmbfgs.h.

#define GNN_LMBFGS_STEP   0.1

This constant defines the initial step taken by the bracketing algorithm to search the minimum along the line. Definition at line 59 of file gnn_lmbfgs.h.

#define GNN_LMBFGS_TOL   0.0001

This constant defines the tolerance for the line search procedure used by the conjugate gradients algorithm. Definition at line 50 of file gnn_lmbfgs.h.

---

Typedef Documentation

typedef struct _gnn_lmbfgs gnn_lmbfgs

This type extends the basic gnn_trainer : Trainers for Models. structure for the conjugate gradients trainer. Definition at line 95 of file gnn_lmbfgs.h.

typedef double(* gnn_lmbfgs_beta)(gnn_trainer *trainer)

This type defines the form of the procedures for computing the coefficient used by the conjugate gradients method. Definition at line 86 of file gnn_lmbfgs.h.

---

Function Documentation

void gnn_lmbfgs_destroy (  gnn_trainer *    trainer )  [static]

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. Definition at line 261 of file gnn_lmbfgs.c.

gnn_line_search_type gnn_lmbfgs_get_alpha (  gnn_trainer *    trainer )

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. Returns: Returns a pointer to the installed line-search procedure. Definition at line 566 of file gnn_lmbfgs.c.

size_t gnn_lmbfgs_get_restart (  gnn_trainer *    trainer )

This function returns the number of iterations executed by the LMBFGS trainer before reinitializing the search direction. Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. Returns: Returns the number of iterations. Definition at line 511 of file gnn_lmbfgs.c.

double gnn_lmbfgs_get_step (  gnn_trainer *    trainer )

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. Returns: The trainer's internal step. Definition at line 460 of file gnn_lmbfgs.c.

double gnn_lmbfgs_get_tol (  gnn_trainer *    trainer )

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. Returns: Returns the tolerance's value. Definition at line 412 of file gnn_lmbfgs.c.

int gnn_lmbfgs_iteration (  gnn_lmbfgs *    bf, double *    A, double *    B )  [static]

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. A  A pointer where to store the coefficient. B  A pointer where to store the coefficient. Returns: Returns 0 if suceeded. Definition at line 139 of file gnn_lmbfgs.c.

gnn_trainer* gnn_lmbfgs_new (  gnn_node *    node, gnn_criterion *    crit, gnn_dataset *    data )

This function creates a new LMBFGS trainer (gnn_lmbfgs : Limited Memory BFGS Algorithm.). Parameters: node  A pointer to a gnn_node. crit  A pointer to a gnn_criterion : Basic Criterion Function.. data  A pointer to a gnn_dataset : Datasets for Training.. Returns: Returns a pointer to a new gnn_lmbfgs : Limited Memory BFGS Algorithm. trainer. Definition at line 306 of file gnn_lmbfgs.c.

int gnn_lmbfgs_reset (  gnn_trainer *    trainer )  [static]

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. Returns: Returns 0 if suceeded. Definition at line 107 of file gnn_lmbfgs.c.

int gnn_lmbfgs_set_line_search (  gnn_trainer *    trainer, gnn_line_search_type    lsearch )

This function sets a new line search procedure used by the LMBFGS trainer. gnn_trainer *trainer; trainer = gnn_lmbfgs_new (node, crit, data); // use the Golden-Section line search gnn_lmbfgs_set_line_search (trainer, gnn_line_search_golden); Please refer to (Line Search Procedures for Nodes.) for the available line search procedures. Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. lsearch  A pointer to a line search procedure. Returns: Returns 0 if succeeded. Definition at line 544 of file gnn_lmbfgs.c.

int gnn_lmbfgs_set_restart (  gnn_trainer *    trainer, size_t    restart )

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. restart  A stricly positive integer. Returns: Returns 0 if succeeded. Definition at line 480 of file gnn_lmbfgs.c.

int gnn_lmbfgs_set_step (  gnn_trainer *    trainer, double    step )

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. step  A stricly positive real value. Returns: Returns 0 if succeeded. Definition at line 432 of file gnn_lmbfgs.c.

int gnn_lmbfgs_set_tol (  gnn_trainer *    trainer, double    tol )

Parameters: trainer  A pointer to a gnn_lmbfgs : Limited Memory BFGS Algorithm.. tol  A stricly positive real value. Returns: Returns 0 if succeeded. Definition at line 384 of file gnn_lmbfgs.c.

---