Benchmarking the Pure Random Search on the Bi-objective BBOB-2016 Testbed

(1)

HAL Id: hal-01435455

https://hal.inria.fr/hal-01435455

Submitted on 14 Jan 2017

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Benchmarking the Pure Random Search on the

Bi-objective BBOB-2016 Testbed

Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar,

Tobias Wagner

To cite this version:

Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar, et al..

Benchmark-ing the Pure Random Search on the Bi-objective BBOB-2016 Testbed. GECCO 2016 - Genetic

and Evolutionary Computation Conference, Jul 2016, Denver, CO, United States. pp.1217-1223,

�10.1145/2908961.2931704�. �hal-01435455�

(2)

Benchmarking the Pure Random Search on the

Bi-objective BBOB-2016 Testbed

Anne Auger

?

Dimo Brockhoff

•

_{Nikolaus Hansen}

?

Dejan Tušar

•

Tea Tušar

•

_{Tobias Wagner}

?_{Inria Saclay—Ile-de-France}

TAO team, France LRI, Univ. Paris-Sud

firstname.lastname@inria.fr

•

Inria Lille Nord-Europe DOLPHIN team, France

Univ. Lille, CNRS, UMR 9189 – CRIStAL

firstname.lastname@inria.fr

TU Dortmund University Institute of Machining Technology (ISF), Germany

wagner@isf.de

ABSTRACT

The Comparing Continuous Optimizers platform COCO has become a standard for benchmarking numerical (single-ob-jective) optimization algorithms effortlessly. In 2016, COCO has been extended towards multi-objective optimization by providing a first bi-objective test suite. To provide a base-line, we benchmark a pure random search on this bi-objective bbob-biobj test suite of the COCO platform. For each combination of function, dimension n, and instance of the test suite, 106· n candidate solutions are sampled uniformly within the sampling box [−5, 5]n.

Keywords

Benchmarking, Black-box optimization, Bi-objective opti-mization

1. INTRODUCTION

The pure random search, already discussed in the late 1950s by Brooks [4], is the simplest stochastic search algo-rithm and shall serve as a baseline algoalgo-rithm in any bench-marking experiment. The algorithm samples each candidate solution independently and uniformly at random within a fixed search domain and returns the best solution found. In the case of numerical optimization minx∈Rnf (x) with

f : Rn 7→ Rk

and thus k objective functions, the search domain is typically a hyperrectangle H ⊂ Rn. Here, we assume all variables of equal scaling and the region of in-terest centered around the origin and choose a hypercube H = [−b, b]n

with b ∈ R.

2. ALGORITHM PRESENTATION

For the numerical experiments, we employ the pure ran-dom search as implemented in the Matlab/Octave exam-pleexperiment, provided by the COCO platform [6]. Algo-rithm1gives the pseudocode.

c

The authors, 2016. This is the authors’ version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published at GECCO’16, July 20–24, 2016, Den-ver, CO, USA, http://dx.doi.org/10.1145/2908961.2931704

Algorithm 1 Pseudocode (based on the used

Mat-lab/Octave code) of the pure random search (PRS). The parameters of the algorithm are a benchmark problem problem including its dimension n, the lower bounds lb = −b · ones(1, n) and upper bounds ub = b · ones(1, n) for each variable with b ∈ R, and a budget in number of func-tion evaluafunc-tions.

1: procedure PRS(problem, n, lb, ub, budget) 2: while budget > cocoGetEvaluations(problem) do 3: x = lb + rand(1, n) ∗ (ub − lb);

4: cocoEvaluateFunction(problem, x);

5: end while

6: end procedure

2.1 Parameter Settings

Being a simple algorithm, only two parameters need to be set for running the pure random search: the upper and lower bounds −b, b of the sampling hypercube and the overall runtime—typically linearly increasing with the search space dimension n.

As for the pure random search on the single-objective bbob test suite [2], we set here the sample hypercube to [−5, 5]n and perform 106· n function evaluations—although the re-gion of interest for the bbob-biobj test suite has been sug-gested to be [−100, 100]n because it cannot be guaranteed that the entire Pareto set lies within the smaller hypercube of [−5, 5]n_{, even if the single-objective optima are contained}

in it. The reason for sampling in the smaller hypercube here for the baseline algorithm is the curse of dimensional-ity which results in a much worse performance of the pure random search within [−100, 100]n _{than within [−5, 5]}n _as

can be witnessed in the companion paper [1].

3. CPU TIMING

In order to evaluate the CPU timing of the algorithm, we have run the pure random search within [−5, 5]n, denoted as RS-5, on the entire bbob-biobj test suite for 103_{· n function}

evaluations. The Matlab/Octave code was run with Octave 4.0.0 on a Windows 7 machine with Intel(R) Core(TM) i7-5600U CPU 2.60GHz with 1 processor and 2 cores. The time per function evaluation for dimensions 2, 3, 5, 10, 20, and 40 equaled 6.0 · 10−5, 5.8 · 10−5, 5.5 · 10−5, 5.5 · 10−5, 5.8 · 10−5, and 6.8 · 10−5 seconds respectively.

(3)

4. RESULTS AND DISCUSSION

Results of RS-5 from experiments according to [7], [5] and [3] and on the benchmark functions given in [8] are presented in Figures1,2,3, and 4, and in Table 1. The experiments were performed with COCO [6], version 1.0.1, the plots were produced with version 1.1.

Overall, the performance of the pure random search on the bbob-biobj test suite is rather weak as expected. In particular for the larger dimensions, many problems cannot be solved to even small target precisions in the allocated budget. In 20-D, for example, 38 of the 55 problems can-not be solved by a single instance to a target precision of 0.1 or less. The function-wise empirical cumulative distri-bution functions of the algorithm’s runtimes in Fig. 1, 2, and 3, show well the scaling with the problem dimension. With only a few exceptions, the runtimes increase with the problem dimension as expected. Exceptions to this rule, indicating most likely that the quality of the Pareto set ap-proximation is not equal over dimension, can be seen on problems 10, 40, 42, and 43 (almost equal performance in 20-D and in 40-D), on problem 12 (3-D curve better than 2-D curve), on problems 18 and 44 (5-D results almost as good as for 3-D for smaller budgets), on problem 48 (almost equal performance between 5-D and 10-D) and for problems 50 and 55 (almost same performance for 10-D, 20-D, and 40-D).

The differences between the dimensions tested are large in terms of solved problems (interpreted as function/dimension/ target precision tuples): while in 2-D, 60% or more of the target precisions can be reached within the budget of 106· n function evaluations on 26 of the 55 functions, on no func-tion, the algorithm solves more than 20% of the targets in 40-D. The extremes in terms of the number of solved tar-get precisions are function 11 (sep. Ellipsoid/sep. Ellipsoid, for which >80% of the targets are solved in 2-D and almost 20% in 40-D) on the one hand and function 14 (sep. ellip-soid/sharp ridge) and function 46 (Rastrigin/Rastrigin) in 2-D where only about 50% of the targets could be reached within the computational budget and functions 19, 34, and 54 where, in 40-D only a handful of targets were reached (and this already with the first evaluation).

Interestingly, the performance of the algorithm within the first 106· n function evaluations shows most often an almost linearly increasing empirical cumulative distribution func-tion of the runtimes to reach all 58 specified targets with fewer differences among the functions than in the single-objective case.

5. CONCLUSIONS

Pure random search is the simplest of all stochastic op-timization algorithms. Hence, it should serve as a baseline in all numerical benchmarking exercises as a lower bound on the performance that every reasonable algorithm should achieve. In this paper, we benchmarked the pure random search within the hypercube [−5, 5]n (with n the problem dimension) as a baseline algorithm on the new bi-objective bbob-biobj test suite of the COCO platform.

6. ACKNOWLEDGMENTS

This work was supported by the grant ANR-12-MONU-0009 (NumBBO) of the French National Research Agency.

7. REFERENCES

[1] A. Auger, D. Brockhoff, N. Hansen, D. Tuˇsar, T. Tuˇsar, and T. Wagner. The impact of search volume on the performance of RANDOMSEARCH on the bi-objective BBOB-2016 test suite. In GECCO (Companion), 2016. DOI: 10.1145/2908961.2931709, to appear.

[2] A. Auger and R. Ros. Benchmarking the pure random search on the BBOB-2009 testbed. In F. Rothlauf, editor, GECCO (Companion), pages 2479–2484. ACM, 2009.

[3] D. Brockhoff, T. Tuˇsar, D. Tuˇsar, T. Wagner, N. Hansen, and A. Auger. Biobjective performance assessment with the COCO platform. ArXiv e-prints, arXiv:1605.01746, 2016.

[4] S. H. Brooks. A discussion of random methods for seeking maxima. Operations Research, 6(2):244–251, 1958.

[5] N. Hansen, A. Auger, D. Brockhoff, D. Tuˇsar, and T. Tuˇsar. COCO: Performance assessment. ArXiv e-prints,arXiv:1605.03560, 2016.

[6] N. Hansen, A. Auger, O. Mersmann, T. Tuˇsar, and D. Brockhoff. COCO: A platform for comparing continuous optimizers in a black-box setting. ArXiv e-prints,arXiv:1603.08785, 2016.

[7] N. Hansen, T. Tuˇsar, O. Mersmann, A. Auger, and D. Brockhoff. COCO: The experimental procedure. ArXiv e-prints,arXiv:1603.08776, 2016.

[8] T. Tuˇsar, D. Brockhoff, N. Hansen, and A. Auger. COCO: The bi-objective black-box optimization benchmarking (bbob-biobj) test suite. ArXiv e-prints, arXiv:1604.00359, 2016.

(4)

5-D ∆ f 1 e+ 0 1 e-1 1e-2 1 e-3 1 e-4 1e-5 #s ucc f1 1 43 6 9(6682) 3 .0 e6 (7 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f2 3 .0 (2) 2 63 3 8(31254) 3. 8 e6(5 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f3 1 16 3 11 (17395) 5 .1e6 (8 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f4 1 31 8 4(4861) 1 .4 e6 (2 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f5 1 93 5 3(5847) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f6 1 29 4 1(3608) 4 .2 e6 (8 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f7 1 4 .8 e5 (6709) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f8 3 .8 (6) 1 .5e 6(1 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f9 2 .4 (4) 7 50 3 (606) 2. 4 e6 (3 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f10 1 29 0 22 (27019) 2 .3e7 (2 e 7) ∞ ∞ ∞ 5.0 e6 0 /5 f11 1 23 0 1(5514) 4 .5 e5 (1 e 6) 2 .2e6 (3 e 6)2 .3 e7 (2 e 7) ∞ 5.0 e6 0 /5 f12 1 95 8 9(22148) 2. 1 e6(3 e 6) 3 .6e6 (2 e 6)2 .1 e7 (1 e 7) ∞ 5.0 e6 0 /5 f13 2 .8 (4) 1 06 8 (1102) 1 .3e 5(2 e 5) 4 .4e6 (4 e 6) ∞ ∞ 5.0 e6 0 /5 f14 1 1 .9 e5 (2 e 5) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f15 6 .4 (7) 4 27 2 9(63166) 8. 4 e6(1 e 7) ∞ ∞ ∞ 5.0 e6 0 /5 f16 2 .0 (1) 1 .8e 5(3 e 5) 2 .2e7 (3 e 7) ∞ ∞ ∞ 5.0 e6 0 /5 f17 1 70 2 (2126) 1. 9 e6 (4 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f18 1 11 4 5(2450) 4 .2 e5 (1 e 6) 9 .7e6 (1 e 7) ∞ ∞ 5.0 e6 0 /5 f19 1 .8 (2) 3 28 1 (1534) 5 .5e 5(5 e 5) ∞ ∞ ∞ 5.0 e6 0 /5 f20 1 56 9 4(6774) 1 .6 e6 (3 e 6) 2 .0e7 (3 e 7) ∞ ∞ 5.0 e6 0 /5 f21 1 14 4 50 (16358) 5 .7e6 (1 e 7) 2 .1e7 (2 e 7) ∞ ∞ 5.0 e6 0 /5 f22 1 57 9 3(4162) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f23 1 28 8 8(2777) 3 .8 e6 (6 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f24 1 .4 (0 . 5) 1 .6e5 (1 e 5) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f25 3 .0 (2) 1 .3e 6(3 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f26 3 .6 (6) 3 70 6 (2966) 1 .7e 6(4 e 5) ∞ ∞ ∞ 5.0 e6 0 /5 f27 2 .0 (2) 4 18 1 (1072) 1 .7e 6(2 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f28 1 44 1 5(5346) 7 .4 e5 (9 e 5) 2 .4e7 (2 e 7) ∞ ∞ 5.0 e6 0 /5 f29 1 1 .3 e6 (1 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f30 2 .0 (2) 8 33 4 (10080) 3 .8 e6 (9 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f31 1 5 .1 e5 (1 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f32 3 .8 (4) 8 .8e 5(2 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f33 4 .6 (4) 1 83 8 (3348) 2 4 43 3(3930 3)4. 7 e6(1 e 7) ∞ ∞ 5.0 e6 0 /5 f34 1 .6 (0 . 5) 2 30 9 (1604) 1 .6e 6(3 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f35 1 46 9 51 (73104) 2 .3e7 (1 e 7) ∞ ∞ ∞ 5.0 e6 0 /5 f36 1 58 3 85 (47482) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f37 1 4 .1 e6 (9 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f38 1 3 .0 e6 (3 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f39 1 14 5 94 (15590) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f40 1 1 .4e5 (64766) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f41 3 .8 (4) 9 12 3 (5418) 3 .6e 6(3 e 6) ∞ ∞ ∞ 5.0 e6 0 /5 f42 3 .0 (2) 1 .9e 6(4 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f43 2 .2 (2) 7. 9 e5(4 5617) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f44 1 51 5 9(9974) 2 .9 e6 (4 e 6) 2 .1e7 (1 e 7) ∞ ∞ 5.0 e6 0 /5 f45 3 .8 (3) 6 06 2 1(67165) 2. 4 e7(2 e 7) ∞ ∞ ∞ 5.0 e6 0 /5 f46 1 5 .5 e6 (8 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f47 1 2 .4 e6 (5 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f48 5 .6 (5) 2 .2e 6(2 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f49 2 .0 (1) 1 .5e 6(3 e 6) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f50 1 3 .8 e5 (7 e 5) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f51 1 .2 (0 . 5) 1 .6e5 (2 e 5) ∞ ∞ ∞ ∞ 5.0 e6 0 /5 f52 1 .4 (0 . 5) 1 .3e5 (2 e 5) 2 .1e7 (3 e 7) ∞ ∞ ∞ 5.0 e6 0 /5 f53 2 0(47) 1 67 (100) 1 75 38 (40461)1 .3e6 (1 e 6) ∞ ∞ 5.0 e6 0 /5 f54 1 43 0 0(2270) 9 .9 e5 (5 e 5) ∞ ∞ ∞ 5.0 e6 0 /5 f55 1 48 9 97 (56444) 2 .2e7 (2 e 7) ∞ ∞ ∞ 5.0 e6 0 /5 20 -D ∆ f 1 e+ 0 1 e-1 1 e-2 1 e-3 1e-4 1 e-5 #s ucc f1 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f2 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f3 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f4 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f5 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f6 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f7 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f8 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f9 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f10 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f11 1 5 .7 e5 (1 e 6)8 .6e7 (2 e 8) ∞ ∞ ∞ 2.0 e7 0 /5 f12 1 7 .3 e6 (1 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f13 1 8 .3 e6 (2 e 6) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f14 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f15 1 3 .5 e7 (3 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f16 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f17 1 .8 (1) 8 .0e7 (1 e 8) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f18 1 1 .1 e7 (2 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f19 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f20 1 8 .3 e7 (1 e 8) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f21 1 1 .9 e7 (2 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f22 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f23 1 .2 (0 . 2) ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f24 1 8 .3 e7 (4 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f25 1 .2 (0 . 5)8 .1 e7 (2 e 8) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f26 1 3 .1 e7 (9 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f27 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f28 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f29 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f30 1 3 .8 e7 (9 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f31 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f32 6 13 (1526) ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f33 1 3 .4 e7 (3 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f34 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f35 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f36 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f37 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f38 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f39 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f40 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f41 1 .2 (0 . 5)3 .7 e7 (5 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f42 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f43 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f44 2 .0 (2) 8 .1e7 (1 e 8) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f45 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f46 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f47 1 .2 (0 . 5) ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f48 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f49 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f50 1 .4 (1) ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f51 6 .0 (6) 8 .7e7 (8 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f52 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f53 1 1 .0 e7 (1 e 7) ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f54 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 f55 1 ∞ ∞ ∞ ∞ ∞ 2.0 e7 0 /5 T ab le 1 : A v e rag e r un t ime ( aR T ) to re ac h g iv e n t arg e ts , me as u re d in n u m b e r o f fu nc ti on e v al uati o ns . F o r e a c h fun c ti on , th e aR T an d, in bra c e s as di s p e rs io n me as ur e , the h alf di ffe re n c e b e tw e e n 1 0 and 9 0 %-t il e o f (b o o ts tr app e d) ru n ti me s is s ho w n for th e di ffe re n t targ e t ∆ f -v al ue s as s ho w n in th e t op ro w . #s uc c is th e n um b e r o f tri al s t hat re ac h e d the la st ta rg e t H Vref + 10 − 5 . Th e m e di an n u m b e r o f c on du c te d fu nc ti o n e v alu ati o ns is a ddi ti o nal ly g iv e n in it ali cs , if th e targ e t in th e las t c ol u mn w as ne v e r re ac he d .

(5)

0 1 2 3 4 5 6 7 8

log10 of (# f-evals / dimension)

0.0 0.2 0.4 0.6 0.8 1.0

Proportion of function+target pairs

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f1 5 instances 1.1 1 Sphere/Sphere 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f2 5 instances 1.1 2 Sphere/sep. Ellipsoid 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f3 5 instances 1.1 3 Sphere/Attractive sector 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f4 5 instances 1.1 4 Sphere/Rosenbrock 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f5 5 instances 1.1 5 Sphere/Sharp ridge 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f6 5 instances 1.1 6 Sphere/Different Powers 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f7 5 instances 1.1 7 Sphere/Rastrigin 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f8 5 instances 1.1 8 Sphere/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f9 5 instances 1.1 9 Sphere/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f10 5 instances 1.1 10 Sphere/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f11 5 instances 1.1

11 sep. Ellipsoid/sep. Ellipsoid

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

12 sep. Ellipsoid/Attractive sector

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f13 5 instances 1.1 13 sep. Ellipsoid/Rosenbrock 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

14 sep. Ellipsoid/Sharp ridge

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

15 sep. Ellipsoid/Different Powers

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f16 5 instances 1.1 16 sep. Ellipsoid/Rastrigin

Figure 1: Empirical cumulative distribution of simulated (bootstrapped) runtimes in number of

objective function evaluations divided by dimension (FEvals/DIM) for the 58 targets {−10−4, −10−4.2, −10−4.4, −10−4.6, −10−4.8, −10−5, 0, 10−5, 10−4.9, 10−4.8, . . . , 10−0.1, 100_{} for functions f}

(6)

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f17 5 instances 1.1 17 sep. Ellipsoid/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f18 5 instances 1.1 18 sep. Ellipsoid/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f19 5 instances 1.1 19 sep. Ellipsoid/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

20 Attractive sector/Attractive sector

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f21 5 instances 1.1 21 Attractive sector/Rosenbrock 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

22 Attractive sector/Sharp ridge

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

23 Attractive sector/Different Powers

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 10-D 20-D 5-D 3-D 2-D bbob-biobj - f24 5 instances 1.1 24 Attractive sector/Rastrigin 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f25 5 instances 1.1 25 Attractive sector/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 5-D 10-D 3-D 2-D bbob-biobj - f26 5 instances 1.1 26 Attractive sector/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f27 5 instances 1.1 27 Attractive sector/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f28 5 instances 1.1 28 Rosenbrock/Rosenbrock 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f29 5 instances 1.1 29 Rosenbrock/Sharp ridge 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f30 5 instances 1.1 30 Rosenbrock/Different Powers 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f31 5 instances 1.1 31 Rosenbrock/Rastrigin 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f32 5 instances 1.1 32 Rosenbrock/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f33 5 instances 1.1 33 Rosenbrock/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f34 5 instances 1.1 34 Rosenbrock/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

35 Sharp ridge/Sharp ridge

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

36 Sharp ridge/Different Powers

Figure 2: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number

of objective function evaluations, divided by dimension (FEvals/DIM) for the targets as given in Fig. 1 for functions f17 to f36 and all dimensions.

(7)

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f37 5 instances 1.1 37 Sharp ridge/Rastrigin 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f38 5 instances 1.1 38 Sharp ridge/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f39 5 instances 1.1 39 Sharp ridge/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f40 5 instances 1.1 40 Sharp ridge/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

41 Different Powers/Different Powers

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f42 5 instances 1.1 42 Different Powers/Rastrigin 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f43 5 instances 1.1 43 Different Powers/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f44 5 instances 1.1 44 Different Powers/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f45 5 instances 1.1 45 Different Powers/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f46 5 instances 1.1 46 Rastrigin/Rastrigin 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f47 5 instances 1.1 47 Rastrigin/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 5-D 10-D 3-D 2-D bbob-biobj - f48 5 instances 1.1 48 Rastrigin/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f49 5 instances 1.1 49 Rastrigin/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f50 5 instances 1.1 50 Schaffer F7/Schaffer F7 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f51 5 instances 1.1 51 Schaffer F7/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f52 5 instances 1.1 52 Schaffer F7/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f53 5 instances 1.1 53 Schwefel/Schwefel 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f54 5 instances 1.1 54 Schwefel/Gallagher 101 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

20-D 40-D 10-D 5-D 3-D 2-D bbob-biobj - f55 5 instances 1.1 55 Gallagher 101/Gallagher 101

of objective function evaluations, divided by dimension (FEvals/DIM) for the targets as given in Fig. 1 for functions f37 to f55 and all dimensions.

(8)

separable-separable separable-moderate separable-ill-cond. separable-multimodal

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f1, f2, f11 5 instances 1.1 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f3, f4, f12, f13 5 instances 1.1 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f7, f8, f16, f17 5 instances 1.1

separable-weakstructure moderate-moderate moderate-ill-cond. moderate-multimodal

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

moderate-weakstructure ill-cond.-ill-cond. ill-cond.-multimodal ill-cond.-weakstructure

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

multimodal-multimodal multimodal-weakstructure weakstructure-weakstructure all 55 functions

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f53-f55 5 instances 1.1 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

40-D 20-D 10-D 5-D 3-D 2-D bbob-biobj - f1-f55 5 instances 1.1

of objective function evaluations, divided by dimension (FEvals/DIM) for the 58 targets {−10−4, −10−4.2, −10−4.4

, −10−4.6, −10−4.8, −10−5, 0, 10−5, 10−4.9, 10−4.8, . . . , 10−0.1, 100_{} for all function groups and all dimensions.}