Benchmarking RM-MEDA on the Bi-objective BBOB-2016 Test Suite

(1)

HAL Id: hal-01435449

https://hal.inria.fr/hal-01435449

Submitted on 14 Jan 2017

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Benchmarking RM-MEDA on the Bi-objective

BBOB-2016 Test Suite

Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar,

Tobias Wagner

To cite this version:

Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar, et al.. Benchmarking

RM-MEDA on the Bi-objective BBOB-2016 Test Suite. GECCO 2016 - Genetic and Evolutionary

Com-putation Conference, Jul 2016, Denver, CO, United States. pp.1241-1247, �10.1145/2908961.2931707�.

�hal-01435449�

(2)

Benchmarking RM-MEDA on the Bi-objective BBOB-2016

Test Suite

Anne Auger

?

Dimo Brockhoff

•

_{Nikolaus Hansen}

?

Dejan Tušar

•

Tea Tušar

•

_{Tobias Wagner}

?_{Inria Saclay—Ile-de-France}

TAO team, France LRI, Univ. Paris-Sud

firstname.lastname@inria.fr

•

Inria Lille Nord-Europe DOLPHIN team, France

Univ. Lille, CNRS, UMR 9189 – CRIStAL

firstname.lastname@inria.fr

TU Dortmund University Institute of Machining Technology (ISF), Germany

wagner@isf.de

ABSTRACT

In this paper, we benchmark the Regularity Model-Based Multiobjective Estimation of Distribution Algorithm (RM-MEDA) of Zhang et al. on the bi-objective bbob-biobj test suite of the Comparing Continuous Optimizers (COCO) plat-form. It turns out that, starting from about 200 times di-mension many function evaluations, RM-MEDA shows a linear increase in the solved hypervolume-based target values with time until a stagnation of the performance occurs rather quickly on all problems. The final percentage of solved hy-pervolume targets seems to decrease with the problem di-mension.

Categories and Subject Descriptors

G.1.6 [Numerical Analysis]: Optimization—global opti-mization, unconstrained optimization; F.2.1 [Analysis of Algorithms and Problem Complexity]: Numerical Al-gorithms and Problems

Keywords

Benchmarking, Black-box optimization, Bi-objective opti-mization

1. INTRODUCTION

Multi-objective optimization differs from single-objective optimization most importantly in the type of the desired ap-proximation. In the single-objective case, a single solution with a function value as small as possible is sought. In the multi-objective case, one is interested in finding an approx-imation of the set of Pareto-optimal solutions. The quality of this approximation is given explicitly or implicitly by an indicator function, such as the hypervolume indicator [1].

Despite this difference of aiming to converge either to a set or to a single solution, only few efforts have been pur-sued to specifically develop variation or solution generation

c

The authors, 2016. This is the authors’ version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published at GECCO’16, July 20–24, 2016, Den-ver, CO, USA, http://dx.doi.org/10.1145/2908961.2931707

approaches for the multi-objective case. In most cases, still the established variation operators from single-objective op-timization are used. One proposal of such a specific variation method is the Regularity Model-Based Multiobjective Esti-mation of Distribution Algorithm (RM-MEDA) by Zhang et al. We will benchmark this approach on the bi-objective bbob-biobj test suite [6] of the Comparing Continuous Optimizers platform COCO [4].

Throughout the paper, n will denote the problem dimen-sion.

2. ALGORITHM

The main idea behind RM-MEDA is to approximate the Pareto set of a multi-objective problem by an (n − 1)-dimensional manifold represented by a piecewise linear model. This model is used to sample candidate solutions in the vicinity of the current approximation, where the selected solutions are in turn used to improve the sampling model. The number of segments or clusters is a parameter of the algorithm and the respective submodels are learned through the applica-tion of a linear principle component analysis (LPCA). The sampling of the piecewise linear model is performed by ran-domly picking a segment of the model. The probability of picking a segment is relative to the segments’ volume1. On the segment, a point is chosen uniformly at random and a random perturbation in terms of an isotropic n-dimensional normal distribution is added. The variance or step size of the perturbation depends on the variation of the solutions of the current population assigned to the chosen segment.

For a more detailed algorithm description of RM-MEDA, we refer the interested reader to the original publication [7].

2.1 Parameter Settings

The MATLAB implementation of RM-MEDA2 was run with the default parameters [7]. The code was slightly adjusted to cope with the assumptions regarding the vector and variable representation mady by the COCO platform. A population size N = 100 was set. The generator of the distribution for sampling new solutions was based on a linear principal com-ponent analysis (LPCA) with 5 clusters, 50 training steps, and an extension rate of 0.25. The budget of 105_{n was used} 1_{In the bi-objective case, the lengths of the segments are}

used.

2_{http://cswww.essex.ac.uk/staff/qzhang/code/} RM-MEDA-Matlab v0.1.zip

(3)

to determine the number of generations. No restarts were performed.

3. CPU TIMING

In order to evaluate the CPU timing of the algorithm, we have run the RM-MEDA with restarts on the entire bbob-biobj test suite for 100n function evaluations. The COCO Mat-lab/Octave code was run with Octave 4.0.0 on a Windows 7 machine with Intel(R) Core(TM) i7-5600U CPU 2.60GHz with 1 processor and 2 cores. The time per function evalua-tion for dimensions 2, 3, 5, 10, 20, and 40 equaled 4.7 · 10−4, 5.0 · 10−4, 5.2 · 10−4, 5.8 · 10−4, 7.0 · 10−4, and 8.7 · 10−4 seconds respectively.

4. RESULTS

Results of RM-MEDA from experiments according to [5], [3] and [2] and on the benchmark functions given in [6] are presented in Figures 1, 2, 3, and 4, and in Table1. The experiments were performed with COCO [4], version 1.0.1 and the plots were produced with version 1.1.

On almost all problems, three phases can be distinguished in the ECDF plots of Figures1, 2, and3. The first phase of initialization and learning of the piecewise representation takes about 100..200n function evaluations. In this phase, none or almost none of the target precisions are reached. It coincides with the performance of a pure random search within the region of interest [−100, 100]n (comparison re-sults with pure random search not shown here). The second phase shows a linear convergence in which the performance displayed in the ECDFs does not differ much for changing dimension (ECDFs for different dimensions are almost par-allel). In this phase, the algorithm successfully exploits the sampling model. The last phase is characterized by a stag-nation of the approximation quality. The sampling model cannot further be refined and the number of targets achieved remains constant. The actual percentages of solved prob-lems at the end of the runs depend on the dimension of the problems.

For the linear convergence phase, on some functions, the algorithm seems to be quicker for smaller problem dimen-sions (on the four functions 11, 12, 16, and 23) while on most, the algorithm shows an increased performance with higher dimension (on the 35 functions 1–10, 17, 21, 26, 28– 30, 32–40, 44–50, 52, 54, 55, and on almost all function groups). This observation, however, depends on the relative quality of the reference sets and the corresponding hypervol-ume reference values underlying the performance assessment of COCO. This limitation must be taken into account before making more general statements.

This also holds for the performance of RM-MEDA in the last phase, but to a smaller degree: it is expected that algorithm performance is decreasing when the problem dimension in-creases as this is the case also for RM-MEDA on all function groups and most functions. Exceptions are f26 (Attractive sector/Schwefel), and functions for which pairs of dimen-sions show similar performance. Here, it is most likely that the reference sets have a larger impact on the display than the actual effect of the dimension.

5. CONCLUSION

After a short initialization phase, the Regularity Model-Based Multiobjective Estimation of Distribution Algorithm

(RM-MEDA) of Zhang et al. showed a linear increase in the solved hypervolume-based target values on the bi-objective bbob-biobj test suite of the Comparing Continuous Opti-mizers (COCO) platform. However, after some time, a stag-nation of the performance occurred. The final percentage of solved hypervolume targets seems to decrease with the prob-lem dimension.

6. ACKNOWLEDGMENTS

This work was supported by the grant ANR-12-MONU-0009 (NumBBO) of the French National Research Agency.

7. REFERENCES

[1] A. Auger, J. Bader, D. Brockhoff, and E. Zitzler. Hypervolume-based Multiobjective Optimization: Theoretical Foundations and Practical Implications. Theoretical Computer Science, 425:75–103, 2012. [2] D. Brockhoff, T. Tuˇsar, D. Tuˇsar, T. Wagner,

N. Hansen, and A. Auger. Biobjective performance assessment with the COCO platform. ArXiv e-prints,

arXiv:1605.01746, 2016.

[3] N. Hansen, A. Auger, D. Brockhoff, D. Tuˇsar, and T. Tuˇsar. COCO: Performance assessment. ArXiv e-prints,arXiv:1605.03560, 2016.

[4] N. Hansen, A. Auger, O. Mersmann, T. Tuˇsar, and D. Brockhoff. COCO: A platform for comparing continuous optimizers in a black-box setting. ArXiv e-prints,arXiv:1603.08785, 2016.

[5] N. Hansen, T. Tuˇsar, O. Mersmann, A. Auger, and D. Brockhoff. COCO: The experimental procedure. ArXiv e-prints,arXiv:1603.08776, 2016.

[6] T. Tuˇsar, D. Brockhoff, N. Hansen, and A. Auger. COCO: The bi-objective black-box optimization benchmarking (bbob-biobj) test suite. ArXiv e-prints,

arXiv:1604.00359, 2016.

[7] Q. Zhang, A. Zhou, and Y. Jin. RM-MEDA: A regularity model based multiobjective estimation of distribution algorithm. IEEE Transactions on Evolutionary Computation, 12(1):41–63, 2008.

(4)

5-D ∆ f 1 e+ 0 1 e-1 1e-2 1 e-3 1 e-4 1e-5 #s ucc f1 4 19 (312) 1 8 73 (189) 27 4 0(212) 6 0 19 (217) 2. 1 e5 (26314) ∞ 5. 0e5 0 /5 f2 2 20 (241) 3 8 69 (1299) 5 7 74 (1815) 12 86 0 (6950) ∞ ∞ 5. 0e5 0 /5 f3 2 17 (512) 2 4 23 (968) 35 1 7(1569) 8 61 7(460 6) 4. 5 e5(3 e 5) ∞ 5. 0e5 0 /5 f4 6 28 (182) 2 1 09 (104) 30 5 3(248) 5 2 12 (898) ∞ ∞ 5. 0e5 0 /5 f5 3 .8 (4) 2 19 7 (126) 37 9 2(914) 1. 6 e5 (47390) ∞ ∞ 5. 0e5 0 /5 f6 6 85 (414) 2 1 71 (208) 30 7 1(238) 6 5 72 (1218) 4 .5e5 (3 e 5) ∞ 5. 0e5 0 /5 f7 3 59 (305) 2 6 44 (662) 2 .2 e5 (3 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f8 1 10 8 (121) 2 86 7(8 52) 3. 0 e5(2 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f9 8 23 (426) 1 9 77 (293) 28 5 7(270) 4 6 19 (423) 8 .2 e5(1 e 6) ∞ 5. 0e5 0 /5 f10 6 28 (134) 2 5 19 (428) 1 .3 e5 (4 e 5) 3 .4e 5(1 e 6) ∞ ∞ 5. 0e5 0 /5 f11 1 .6 (0 . 8) 7 23 5( 11517)23 1 68 (20566) 1 .6e5 (3 e 5) ∞ ∞ 5. 0e5 0 /5 f12 9 1(111) 33 7 1(2847) 62 0 3(5462) 1 .5 e5 (4 e 5) 7. 6 e5 (1 e 6) ∞ 5. 0e5 0 /5 f13 5 70 (648) 2 4 24 (698) 34 4 2(951) 6 1 85 (2855) 1 3 04 0 (7560) 4 .4e5 (8 e 5) 3 /5 f14 1 0(10) 4 47 7 (541) 80 68 (1219) 1. 6 e5 (1 e 5) ∞ ∞ 5. 0e5 0 /5 f15 9 33 (1532) 4 61 9(79 7) 7 3 42 (3738) 1. 4 e5 (3512) ∞ ∞ 5. 0e5 0 /5 f16 1 31 (230) 5 0 17 (2023) 4. 5 e5(1 e 6) ∞ ∞ ∞ 5. 0e5 0 /5 f17 1 48 8 (206) 7 64 6(4 549) 4 42 1 8(23987) 2. 4 e6(3 e 6) ∞ ∞ 5. 0e5 0 /5 f18 4 3(52) 3 11 8 (690) 52 55 (1988) 13 4 63 (5350) ∞ ∞ 5. 0e5 0 /5 f19 3 21 (268) 4 5 60 (999) 79 7 6(1776) 7 .6e5 (2 e 6) ∞ ∞ 5. 0e5 0 /5 f20 1 50 (370) 3 1 92 (1956) 5 5 49 (2672) 30 92 3 (27578) 5 .8 e5(7 e 5) 2. 0 e6 (4 e 6) 1 /5 f21 3 55 (400) 2 8 41 (958) 1 .3 e5 (3517) 1 .3e 5(1 e 5) 2. 0 e6 (2 e 6) ∞ 5. 0e5 0 /5 f22 6 .2 (12) 24 0 9(394) 4 6 74 (1633) 1 5 66 8( 8626) 2. 0 e6 (4 e 6) ∞ 5. 0e5 0 /5 f23 1 3(18) 2 42 3 (474) 37 02 (578) 1 10 1 8(9460) 9 .0e 5(7 e 5) ∞ 5. 0e5 0 /5 f24 1 97 (383) 4 0 98 (1030) 3. 3 e5(5 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f25 3 18 (457) 3 4 39 (1448) 9 8 54 5 (1 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f26 4 68 (408) 2 3 76 (706) 33 5 3(822) 1 .3 e5(3 e 5) 9. 4 e5 (1 e 6) ∞ 5. 0e5 0 /5 f27 1 7(9) 2 3 04 (156) 7 .5e5 (1 e 6) ∞ ∞ ∞ 5. 0e5 0 /5 f28 3 78 (296) 1 9 89 (268) 29 7 9(458) 4 5 19 (1206) 3 .8e5 (4 e 5) ∞ 5. 0e5 0 /5 f29 3 18 (307) 2 6 58 (416) 53 8 7(1641) 3 79 93 (10668) ∞ ∞ 5. 0e5 0 /5 f30 9 74 (390) 2 2 10 (366) 33 6 0(370) 6 4 56 (1502) 2 .3e6 (3 e 6) ∞ 5. 0e5 0 /5 f31 7 03 (198) 3 2 71 (1177) 2. 1 e5(2 e 5) 2 .3e 6(2 e 6) ∞ ∞ 5. 0e5 0 /5 f32 1 16 4 (324) 2 96 3(4 77) 1. 8 e5(3 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f33 8 42 (593) 1 7 75 (437) 22 3 6(431) 3 8 10 (1714) 7 .6e5 (1 e 6) ∞ 5. 0e5 0 /5 f34 8 00 (371) 2 5 23 (604) 7 .5 e5 (1 e 6) 2 .0e 6(9 e 5) ∞ ∞ 5. 0e5 0 /5 f35 1 24 8 9(356) 5 2 61 (1218) 2. 3 e5(3 e 5) ∞ ∞ 5. 0e5 0 /5 f36 1 15 (50) 26 4 8(281) 8 54 6 (4864) 2 .0 e6(3 e 6) ∞ ∞ 5. 0e5 0 /5 f37 8 6(114) 41 7 3(1593) 3 .1 e5 (4 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f38 6 41 (294) 4 4 59 (2067) 9. 9 e5(1 e 6) ∞ ∞ ∞ 5. 0e5 0 /5 f39 1 32 (36) 23 6 1(206) 1 .3e 5(1 e 5) 1 .4e 5(1 e 5) 2. 1 e6 (4 e 6) ∞ 5. 0e5 0 /5 f40 1 41 (97) 27 5 7(320) 6 78 2 (4396) 2 .1 e6(2 e 6) ∞ ∞ 5. 0e5 0 /5 f41 8 42 (315) 2 5 66 (362) 38 2 2(824) 8 1 31 (2770) 1 .0e6 (5 e 5) ∞ 5. 0e5 0 /5 f42 1 04 4 (294) 4 26 9(2 458) 9 .3e5 (8 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f43 1 20 5 (754) 3 32 3(7 89) 1. 7 e5(2 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f44 1 13 5 (473) 2 41 1(5 04) 3 3 01 (151) 4 94 8(1 096) 2 .3e5 (75384) ∞ 5. 0e5 0 /5 f45 8 31 (518) 2 9 73 (253) 68 5 6(3190) 7 .8e5 (1 e 6) ∞ ∞ 5. 0e5 0 /5 f46 6 16 (392) 1 3 38 8(111 92) 2. 2 e6 (2 e 6) ∞ ∞ ∞ 5. 0e5 0 /5 f47 9 14 (320) 1 0 02 0(651 7) 2 .1e6 (2 e 6) ∞ ∞ ∞ 5. 0e5 0 /5 f48 1 12 7 (156) 1 90 81 (37880) 3 .7e 5(5 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f49 7 43 (374) 3 8 12 (500) 5 .1 e5 (2 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f50 1 06 6 (262) 2 93 3(8 26) 3. 0 e5(3 e 5) ∞ ∞ ∞ 5. 0e5 0 /5 f51 9 67 (197) 2 5 00 (831) 34 4 25 (32526) ∞ ∞ ∞ 5. 0e5 0 /5 f52 8 87 (179) 2 7 23 (881) 3 .6 e5 (1 e 6) 2 .3e 6(3 e 6) ∞ ∞ 5. 0e5 0 /5 f53 7 86 (865) 1 7 69 (45) 2 14 1 (118) 51 5 3(6247) 1 .4e5 (372) ∞ 5. 0e5 0 /5 f54 8 60 (238) 2 3 74 (357) 1 .3 e5 (3 e 5) 8 .2e 5(1 e 6) 1. 2 e6 (7 e 5) ∞ 5. 0e5 0 /5 f55 7 50 (378) 2 9 96 (218) 3 .4 e5 (5 e 5) 7 .6e 5(2 e 6) 8. 4 e5 (1 e 6) ∞ 5. 0e5 0 /5 20 -D ∆ f 1 e+ 0 1e-1 1 e-2 1e-3 1 e-4 1 e-5 #s ucc f1 6 12 (132) 4 0 67 (57) 6 1 08 (55) 4 45 8 5(6122) ∞ ∞ 2.0 e6 0 /5 f2 9 35 (490) 6 7 55 (776) 10 6 89 (1774) ∞ ∞ ∞ 2.0 e6 0 /5 f3 1 35 (106) 5 2 04 (217) 74 3 4(69) 57 1 54 (39353) ∞ ∞ 2.0 e6 0 /5 f4 1 67 7(20 7) 4 23 9 (142) 61 54 (242) 7 08 95 (1 e 5) ∞ ∞ 2.0 e6 0 /5 f5 1 4 53 6 (167) 86 58 (723) 4 .0e5 (2 e 5) ∞ ∞ 2.0 e6 0 /5 f6 1 47 3(62 8) 5 05 9 (316) 85 64 (1133) 31 06 3 (7742) ∞ ∞ 2.0 e6 0 /5 f7 7 43 (400) 2 2 95 1(2 170) 8 .6e6 (1 e 7) ∞ ∞ ∞ 2.0 e6 0 /5 f8 2 05 9(22 6) 1 17 29 (8618) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f9 1 76 2(37 6) 4 16 6 (116) 61 00 (102) 1 18 55 (527) ∞ ∞ 2.0 e6 0 /5 f10 1 04 6(19 4) 5 55 3 (531) 1. 3 e6(1 e 6) 8 .1e6 (9 e 6) ∞ ∞ 2.0 e6 0 /5 f11 1 2 39 5 2(9283) 6 .4 e5 (5 e 5) ∞ ∞ ∞ 2.0 e6 0 /5 f12 5 .6 (6) 1 1 32 1 (4858) 7 .7e5 (1 e 6) ∞ ∞ ∞ 2.0 e6 0 /5 f13 6 2(76) 5 6 84 (988) 89 0 9(1695) 3 .5 e6 (6 e 6) ∞ ∞ 2.0 e6 0 /5 f14 1 3(15) 8 9 27 (1539) 2 6 70 7( 5312) ∞ ∞ ∞ 2.0 e6 0 /5 f15 1 65 7(10 94)14 22 1 (4316) 2 .5 e5(5 e 5) 8 .7e6 (8 e 6) ∞ ∞ 2.0 e6 0 /5 f16 2 87 (88) 66 5 77 (37917) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f17 1 86 1(30 6) 4 62 14 (69280) 8 .0 e6(5 e 6) ∞ ∞ ∞ 2.0 e6 0 /5 f18 1 7 86 8 (1739) 1 31 5 6(3840) 2 .1 e6 (2 e 6) ∞ ∞ 2.0 e6 0 /5 f19 1 28 1(46 5) 5 .1e5 (3 e 6) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f20 8 0(58) 7 6 45 (3776) 1 7 39 8( 10138) 2. 0 e5 (2 e 5) 2 .2e6 (3 e 6) ∞ 2.0 e6 0 /5 f21 4 90 (221) 6 4 30 (1936) 1 1 30 0 (5847) 1 .3e5 (2 e 5) ∞ ∞ 2.0 e6 0 /5 f22 1 8 82 3 (2016) 2 42 1 4(10872) 8. 2 e5(2 e 6) ∞ ∞ 2.0 e6 0 /5 f23 5 32 (178) 3 0 35 7(2 3130)55 1 52 (33387) 8 .5e5 (1 e 6) ∞ ∞ 2.0 e6 0 /5 f24 1 77 (352) 5. 5 e5(1 e 6) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f25 7 43 (128) 5. 8 e5 (72574) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f26 1 25 5(14 31) 66 77 (1548) 88 5 3(2797) 47 4 28 (45488) 1 .6e6 (3 e 6) ∞ 2.0 e6 0 /5 f27 5 81 (299) 1. 4 e6(2 e 6) 3 .0e6 (4 e 6) 8 .1e6 (1 e 7) ∞ ∞ 2.0 e6 0 /5 f28 9 61 (260) 4 2 98 (216) 63 6 8(794) 5 3 83 2(3 1698) ∞ ∞ 2.0 e6 0 /5 f29 1 02 9(14 7) 5 44 2 (50) 1 34 56 (2084) 8. 6 e6 (1 e 7) ∞ ∞ 2.0 e6 0 /5 f30 2 90 7(20 8) 5 51 3 (776) 95 65 (2364) 3. 2 e6 (7 e 6) ∞ ∞ 2.0 e6 0 /5 f31 1 84 3(22 3) 5 07 72 (60356) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f32 2 67 6(33 4) 1 90 18 (16230) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f33 1 05 1(48 4) 3 86 5 (142) 49 93 (309) 1 46 62 (8327) 9. 9 e6 (8 e 6) ∞ 2.0 e6 0 /5 f34 1 96 5(11 9) 6 28 5 (702) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f35 1 5 10 1 (359) 11 28 2 (3294) 3 .4e 6(3 e 6) ∞ ∞ 2.0 e6 0 /5 f36 7 9(130) 6 30 5(370 ) 1 9 96 7( 6103) ∞ ∞ ∞ 2.0 e6 0 /5 f37 1 48 (140) 3 8 27 7(3 7382) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f38 1 25 5(13 6) 3 95 86 (9720) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f39 8 6(110) 5 26 7(472 ) 1 2 93 7( 8766) 2 .5e6 (2 e 6) ∞ ∞ 2.0 e6 0 /5 f40 1 .6 (1) 5. 1 e5(2 e 6) 3 .0e6 (8 e 6) ∞ ∞ ∞ 2.0 e6 0 /5 f41 3 04 2(11 16) 88 72 (1952) 19 0 26 (10856) 5 .4e5 (4 e 5) ∞ ∞ 2.0 e6 0 /5 f42 2 34 5(78 2) 6 .8e5 (2 e 6) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f43 3 46 1(24 8) 6 70 15 (86923) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f44 2 86 8(78 2) 5 53 7 (1058) 8 72 6 (1437) 1 70 6 7(7840) ∞ ∞ 2.0 e6 0 /5 f45 1 80 6(26 5) 7 92 5 (2045) 3 .0e6 (2 e 6) ∞ ∞ ∞ 2.0 e6 0 /5 f46 7 93 (228) 1. 2 e5(1 e 5) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f47 2 55 0(72 ) 1 .3 e5 (38852) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f48 9 81 (381) 3 0 02 8(2 0545) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f49 1 35 1(75 ) 3. 2 e6(4 e 6) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f50 2 70 4(17 6) 2 09 57 (14228) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f51 2 55 2(37 0) 8 88 5 (4466) 8 .0e6 (6 e 6) ∞ ∞ ∞ 2.0 e6 0 /5 f52 2 15 1(15 6) 5 23 39 (86772) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f53 5 21 (26) 39 4 9(128) 4 9 22 (233) 75 9 8(1198) 62 0 26 (10936) ∞ 2.0 e6 0 /5 f54 2 01 4(12 0) 5 99 1 (598) ∞ ∞ ∞ ∞ 2.0 e6 0 /5 f55 1 08 8(18 3) 1 .4e6 (2 e 6) 3 .0e6 (6 e 6) ∞ ∞ ∞ 2.0 e6 0 /5 T ab le 1 : A v e rag e r un t ime ( aR T ) to re ac h g iv e n t arg e ts , me as u re d in n u m b e r o f fu nc ti on e v al uati o ns . F o r e a c h fun c ti on , th e aR T an d, in bra c e s as di s p e rs io n me as ur e , the h alf di ffe re n c e b e tw e e n 1 0 and 9 0 %-t il e o f (b o o ts tr app e d) ru n ti me s is s ho w n for th e di ffe re n t targ e t ∆ f -v al ue s as s ho w n in th e t op ro w . #s uc c is th e n um b e r o f tri al s t hat re ac h e d the la st ta rg e t H Vref + 10 − 5 . Th e m e di an n u m b e r o f c on du c te d fu nc ti o n e v alu ati o ns is a ddi ti o nal ly g iv e n in it ali cs , if th e targ e t in th e las t c ol u mn w as ne v e r re ac he d .

(5)

0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

Proportion of function+target pairs

20-D 10-D 5-D 3-D 2-D bbob-biobj - f1 5 instances 1.1 1 Sphere/Sphere 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f2 5 instances 1.1 2 Sphere/sep. Ellipsoid 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 3-D 5-D 2-D bbob-biobj - f3 5 instances 1.1 3 Sphere/Attractive sector 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f4 5 instances 1.1 4 Sphere/Rosenbrock 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f5 5 instances 1.1 5 Sphere/Sharp ridge 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f6 5 instances 1.1 6 Sphere/Different Powers 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 20-D 5-D 3-D 2-D bbob-biobj - f7 5 instances 1.1 7 Sphere/Rastrigin 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f8 5 instances 1.1 8 Sphere/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 3-D 5-D 2-D bbob-biobj - f9 5 instances 1.1 9 Sphere/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 2-D 3-D bbob-biobj - f10 5 instances 1.1 10 Sphere/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 20-D 5-D 3-D 2-D bbob-biobj - f11 5 instances 1.1

11 sep. Ellipsoid/sep. Ellipsoid

12 sep. Ellipsoid/Attractive sector

20-D 10-D 5-D 3-D 2-D bbob-biobj - f13 5 instances 1.1 13 sep. Ellipsoid/Rosenbrock 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

14 sep. Ellipsoid/Sharp ridge

15 sep. Ellipsoid/Different Powers

20-D 10-D 5-D 3-D 2-D bbob-biobj - f16 5 instances 1.1 16 sep. Ellipsoid/Rastrigin

Figure 1: Empirical cumulative distribution of simulated (bootstrapped) runtimes in number of

objective function evaluations divided by dimension (FEvals/DIM) for the 58 targets {−10−4, −10−4.2, −10−4.4, −10−4.6, −10−4.8, −10−5, 0, 10−5, 10−4.9, 10−4.8, . . . , 10−0.1, 100_{} for functions f}

(6)

20-D 10-D 5-D 3-D 2-D bbob-biobj - f17 5 instances 1.1 17 sep. Ellipsoid/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f18 5 instances 1.1 18 sep. Ellipsoid/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f19 5 instances 1.1 19 sep. Ellipsoid/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20 Attractive sector/Attractive sector

20-D 5-D 3-D 2-D 10-D bbob-biobj - f21 5 instances 1.1 21 Attractive sector/Rosenbrock 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

22 Attractive sector/Sharp ridge

23 Attractive sector/Different Powers

20-D 10-D 5-D 3-D 2-D bbob-biobj - f24 5 instances 1.1 24 Attractive sector/Rastrigin 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f25 5 instances 1.1 25 Attractive sector/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 5-D 10-D 3-D 2-D bbob-biobj - f26 5 instances 1.1 26 Attractive sector/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 5-D 20-D 3-D 2-D bbob-biobj - f27 5 instances 1.1 27 Attractive sector/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f28 5 instances 1.1 28 Rosenbrock/Rosenbrock 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f29 5 instances 1.1 29 Rosenbrock/Sharp ridge 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f30 5 instances 1.1 30 Rosenbrock/Different Powers 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 20-D 5-D 3-D 2-D bbob-biobj - f31 5 instances 1.1 31 Rosenbrock/Rastrigin 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f32 5 instances 1.1 32 Rosenbrock/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f33 5 instances 1.1 33 Rosenbrock/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f34 5 instances 1.1 34 Rosenbrock/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

35 Sharp ridge/Sharp ridge

36 Sharp ridge/Different Powers

Figure 2: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number

of objective function evaluations, divided by dimension (FEvals/DIM) for the targets as given in Fig. 1 for functions f17 to f36 and all dimensions.

(7)

20-D 10-D 5-D 3-D 2-D bbob-biobj - f37 5 instances 1.1 37 Sharp ridge/Rastrigin 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f38 5 instances 1.1 38 Sharp ridge/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f39 5 instances 1.1 39 Sharp ridge/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f40 5 instances 1.1 40 Sharp ridge/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

41 Different Powers/Different Powers

20-D 10-D 5-D 3-D 2-D bbob-biobj - f42 5 instances 1.1 42 Different Powers/Rastrigin 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f43 5 instances 1.1 43 Different Powers/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f44 5 instances 1.1 44 Different Powers/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 20-D 5-D 3-D 2-D bbob-biobj - f45 5 instances 1.1 45 Different Powers/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 20-D 5-D 3-D 2-D bbob-biobj - f46 5 instances 1.1 46 Rastrigin/Rastrigin 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f47 5 instances 1.1 47 Rastrigin/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 5-D 10-D 3-D 2-D bbob-biobj - f48 5 instances 1.1 48 Rastrigin/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f49 5 instances 1.1 49 Rastrigin/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f50 5 instances 1.1 50 Schaffer F7/Schaffer F7 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f51 5 instances 1.1 51 Schaffer F7/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f52 5 instances 1.1 52 Schaffer F7/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 5-D 20-D 3-D 2-D bbob-biobj - f53 5 instances 1.1 53 Schwefel/Schwefel 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f54 5 instances 1.1 54 Schwefel/Gallagher 101 0 1 2 3 4 5 6 7 8 log10 of (# f-evals / dimension) 0.0 0.2 0.4 0.6 0.8 1.0

10-D 20-D 3-D 5-D 2-D bbob-biobj - f55 5 instances 1.1 55 Gallagher 101/Gallagher 101

of objective function evaluations, divided by dimension (FEvals/DIM) for the targets as given in Fig. 1 for functions f37 to f55 and all dimensions.

(8)

separable-separable separable-moderate separable-ill-cond. separable-multimodal

0 1 2 3 4 5 6 7 8

log10 of (# f-evals / dimension)

0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f1, f2, f11 5 instances 1.1 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f3, f4, f12, f13 5 instances 1.1 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f7, f8, f16, f17 5 instances 1.1

separable-weakstructure moderate-moderate moderate-ill-cond. moderate-multimodal

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

moderate-weakstructure ill-cond.-ill-cond. ill-cond.-multimodal ill-cond.-weakstructure

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

multimodal-multimodal multimodal-weakstructure weakstructure-weakstructure all 55 functions

0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

10-D 20-D 5-D 3-D 2-D bbob-biobj - f53-f55 5 instances 1.1 0 1 2 3 4 5 6 7 8

0.0 0.2 0.4 0.6 0.8 1.0

20-D 10-D 5-D 3-D 2-D bbob-biobj - f1-f55 5 instances 1.1

of objective function evaluations, divided by dimension (FEvals/DIM) for the 58 targets {−10−4, −10−4.2, −10−4.4

, −10−4.6, −10−4.8, −10−5, 0, 10−5, 10−4.9, 10−4.8, . . . , 10−0.1, 100_{} for all function groups and all dimensions.}