4 Algorithms for unconstrained optimization

Introduction¶

In this chapter, we will study algorithms for solving the unconstrained optimization problem

In many cases, finding an explicit solution to the problem is not feasible (unlike for least squares problems encountered in the previous chapter, for instance).

Solution: Use iterative methods (aka algorithms). We aim to find a minimizer of the problem through an iterative process:

• Start from one initial point $x^{(0)}$.

• At each iteration $k$, compute a new point $x^{(k+1)}$ using an update rule based on the function $f_0$ to be minimized.

• Repeat this process until a stopping criterion is met.