# Black-Box Optimization (BBO)¶

The **black-box** nature of many real-world optimization problems comes from one or more of the following factors,
as shown in e.g. the classical book **<<Introduction to Derivative-Free Optimization>>**:

increasing complexity in mathematical modeling,

higher sophistication of scientific computing,

an abundance of legacy or proprietary codes,

noisy function evaluations.

Some common problem characteristics of BBO are presented below:

unavailability of gradient information in the black-box setting (even if the gradient information actually exists),

without a precise model (e.g., owing to complex simulation),

non-differentiability,

non-linearity,

multi-modality,

ill-condition,

noisiness.

For black-box problems, the only information accessible to the algorithm is *function evaluations*, which can be freely
selected by the algorithm, leading to Zeroth-Order Optimization (ZOO) or Derivative-Free Optimization (DFO).

## No Free Lunch Theorems (NFL)¶

As mathematically proved in [Wolpert&Macready, 1997, TEVC], **“for any
algorithm, any elevated performance over one class of problems is offset by performance over another class.”**

This may in part explain why there exist a large number of optimization algorithms from different research communities
in practice. However, unfortunately **not** all optimizers are well-designed and widely-acceptable. Refer to the Design
Philosophy section for discussions.

## Curse of Dimensionality for Large-Scale BBO (LSBBO)¶

Arguably for all black-box optimizers, they can suffer from the famous “Curse of Dimensionality”, since the essence of nearly all black-box optimizers are based on limited sampling. Refer to e.g., [Nesterov&Spokoiny, 2017, FoCM] for a deep theoretical analysis.

Luckily, for some real-world applications, there may exist some structures to be available. If such a structure can be
efficiently exploited in an automatic fashion (via well-designed optimization strategies), the convergence rate may be
significantly improved, if possible. Therefore, any general-purpose black-box optimizer may still need to keep a *subtle*
balance between exploiting concrete problem structures and exploring the entire design space of the optimizer.

## General-Purpose Optimization Algorithms¶

Note

*“Given the abundance of black-box optimization algorithms and of optimization problems, how can best match
algorithms to problems.”*—[Wolpert&Macready, 1997, TEVC]

“Clearly, evaluating and comparing algorithms on a single problem is not sufficient to determine their quality, as much of their benefit lies in their performance generalizing to large classes of problems. One of the goals of research in optimization is, arguably, to provide practitioners with reliable, powerful and general-purpose algorithms.” As a library for BBO, a natural choice is to first prefer and cover general-purpose optimization algorithms (when compared with highly-customized versions), since for most real-world black-box optimization problems the (possibly useful) problem structure is typically unknown in advance.

The following common criteria/principles may be highly expected to satisfy for general-purpose optimization algorithms:

effectiveness and efficiency,

elegance (beauty),

flexibility (versatility),

robustness (reliability),

scalability,

simplicity.

Arguably, the *beauty* of general-purpose black-box optimizers should come from **theoretical depth** and/or **practical
breadth**, though the aesthetic judgment is somewhat *subjective*. We believe that well-designed optimizers could pass
**Test-of-Time** in the history of black-box optimization. For recent critical discussions, refer to e.g.
“metaphor-based metaheuristics, a call for action: the elephant in the room” and “a critical problem in benchmarking and analysis
of evolutionary computation methods”.

For **benchmarking** of continuous optimizers, refer to e.g.
[Hillstrom, 1977, ACM-TOMS],
[More et al., 1981, ACM-TOMS],
[Hansen et al., 2021, OMS],
[Meunier et al., 2022, TEVC]. As stated in
[More et al., 1981, ACM-TOMS], “not testing the algorithm on a large number of functions can easily lead to the
cynical observer to conclude that the algorithm was tuned to particular functions”.

## POPulation-based OPtimization (POP)¶

Note

*“The essence of an evolutionary approach to solve a problem is to equate possible solutions to individuals
in a population, and to introduce a notion of fitness on the basis of solution quality.”*—[Eiben&Smith, 2015,
Nature]

Population-based (particularly evolutionary) optimizers (POP) usually have the following advantages for black-box problems, when particularly compared to individual-based counterparts:

few

*a priori*assumptions (e.g. with a limited knowledge bias),flexible framework (easy integration with problem-specific knowledge via e.g. memetic algorithms),

robust performance (e.g. w.r.t. noisy perturbation or hyper-parameters),

diverse solutions (e.g. for multi-modal/multi-objective/dynamic optimization),

novelty (e.g. beyond intuitions for design problems).

For details (models, algorithms, theories, and applications) about POP, please refer to e.g. the following *well-written*
reviews or books (just to name a few):

Miikkulainen, R. and Forrest, S., 2021. A biological perspective on evolutionary computation. Nature Machine Intelligence, 3(1), pp.9-15.

Schoenauer, M., 2015. Chapter 28: Evolutionary algorithms. Handbook of Evolutionary Thinking in the Sciences. Springer.

Eiben, A.E. and Smith, J., 2015. From evolutionary computation to the evolution of things. Nature, 521(7553), pp.476-482.

De Jong, K.A., Fogel, D.B. and Schwefel, H.P., 1997. A history of evolutionary computation. Handbook of Evolutionary Computation. Oxford University Press.

Forrest, S., 1993. Genetic algorithms: Principles of natural selection applied to computation. Science, 261(5123), pp.872-878.

For **principled design of continuous stochastic search**, refer to e.g.
[Nikolaus&Auger, 2014];
[Wierstra et al., 2014].

For each algorithm family, we also provide some of *wide-recognized* references on its own API documentations. You can also see this GitHub website for a (still growing) paper list of Evolutionary Computation (EC)
published in many *top-tier* and also EC-focused journals and conferences.

## Limitations of BBO¶

Note

*“If you can obtain clean derivatives (even if it requires considerable effort) and the functions defining
your problem are smooth and free of noise you should not use derivative-free methods..”*—[Conn et al., 2009,
Introduction to Derivative-Free Optimization]

Very importantly, **not all** optimization problems can fit well in black-box optimizers. In fact, its *arbitrary abuse*
in science and engineering has resulted in wide criticism. Although not always, black-box optimizers are often seen as
**“the last choice of search methods”**.