Black box optimization, machine learning, and no-free lunch theorems

Intro -- Preface -- Contents -- Learning Enabled Constrained Black-Box Optimization -- 1 Introduction -- 2 Constrained Black-Box Optimization -- 3 The Basic Probabilistic Framework -- 3.1 Gaussian Processes -- 3.2 GP-Based Optimization -- 4 Constrained Bayesian Optimization -- 5 Constrained Bayesian Optimization for Partially Defined Objective Functions -- 6 Software for the Generation of Constrained Test Problems -- 6.1 Emmental-Type GKLS Generator -- 7 Conclusions -- References -- Black-Box Optimization: Methods and Applications -- 1 Introduction -- 2 Overview of BBO Methods -- 2.1 Direct Search Methods -- 2.1.1 Simplex Search -- 2.1.2 Coordinate Search -- 2.1.3 Generalized Pattern Search -- 2.1.4 Mesh Adaptive Direct Search -- 2.2 Model-Based Methods -- 2.2.1 Model-Based Trust Region -- 2.2.2 Projection-Based Methods -- 2.3 Heuristic Methods -- 2.3.1 DIRECT -- 2.3.2 Multilevel Coordinate Search -- 2.3.3 Hit-and-Run algorithms -- 2.3.4 Simulated Annealing -- 2.3.5 Genetic Algorithm -- 2.3.6 Particle Swarm Optimization -- 2.3.7 Surrogate Management Framework -- 2.3.8 Branch and Fit -- 2.4 Hybrid Methods -- 2.5 Extension to Constrained Problems -- 2.5.1 Penalty Method -- 2.5.2 Augmented Lagrangian -- 2.5.3 Filter Method -- 2.5.4 Surrogate Modeling -- 3 BBO Solvers -- 4 Recent Applications -- 4.1 Automatic Machine Learning -- 4.2 Optimization Solvers -- 4.3 Fluid Mechanics -- 4.4 Oilfield Development and Operations -- 4.5 Chemical and Biochemical Engineering -- 5 Open Problems and Future Research Directions -- References -- Tuning Algorithms for Stochastic Black-Box Optimization: State of the Art and Future Perspectives -- 1 Introduction -- 2 Tuning: Strategies -- 2.1 Key Topics -- 2.2 Stochastic Optimization Algorithms -- 2.3 Algorithm Tuning -- 2.4 Example: Grefenstette's Study of Control Parameters for Genetic Algorithms.

2.5 No Free Lunch Theorems -- 2.6 Tuning for Deterministic Algorithms -- 3 Test Sets -- 3.1 Test Functions -- 3.2 Application Domains -- 3.2.1 Tuning in Industry -- 3.2.2 Energy -- 3.2.3 Water Industry -- 3.2.4 Steel Industry -- 3.2.5 Automotive -- 3.2.6 Information Technology -- 4 Statistical Considerations -- 4.1 Experimental Setup -- 4.2 Design of Experiments -- 4.3 Measuring Performance -- 4.4 Reporting Results -- 5 Parallelization -- 5.1 Overview -- 5.2 Simplistic Approaches -- 5.3 Parallelization in Surrogate Model-Based Optimization -- 5.3.1 Uncertainty-Based Methods -- 5.3.2 Surrogate-Assisted Algorithms -- 6 Tuning Approaches -- 6.1 Overview -- 6.2 Manual Tuning -- 6.3 Automatic Tuning -- 6.4 Interactive Tuning -- 6.5 Internal Tuning -- 7 Tuning Software -- 7.1 Overview -- 7.2 IRACE -- 7.3 SPOT -- 7.4 SMAC -- 7.5 ParamILS -- 7.6 GGA -- 7.7 Usability and Availability of Tuning Software -- 7.8 Example: SPOT -- 8 Research Directions and Open Problems -- 9 Summary and Outlook -- References -- Quality-Diversity Optimization: A Novel Branch of Stochastic Optimization -- 1 Introduction -- 2 Problem Formulation -- 2.1 Collections of Solutions -- 2.2 How Do We Measure the Performance of a QD Algorithm? -- 3 Optimizing a Collection of Solutions -- 3.1 MAP-Elites -- 3.2 A Unified Framework -- 3.2.1 Containers -- 3.2.2 Selection Operators -- 3.2.3 Population Scores -- 3.3 Considerations of Quality-Diversity Optimization -- 4 Origins and Related Work -- 4.1 Searching for Diverse Behaviors -- 4.2 Connections to Multimodal Optimization -- 4.3 Connections to Multitask Optimization -- 5 Current Topics -- 5.1 Expensive Objective Functions -- 5.2 High-Dimensional Feature Space -- 5.3 Learning the Behavior Descriptor -- 5.4 Improving Variation Operators -- 5.5 Noisy Functions -- 6 Conclusion -- References.

Multi-Objective Evolutionary Algorithms: Past, Present, and Future -- 1 Introduction -- 2 Basic Concepts -- 3 The Past -- 3.1 Non-Elitist Non-Pareto Approaches -- 3.1.1 Linear Aggregating Functions -- 3.1.2 Vector Evaluated Genetic Algorithm (VEGA) -- 3.1.3 Lexicographic Ordering -- 3.1.4 Target-Vector Approaches -- 3.2 Non-Elitist Pareto-Based Approaches -- 3.2.1 Multi-Objective Genetic Algorithm (MOGA) -- 3.2.2 Nondominated Sorting Genetic Algorithm (NSGA) -- 3.2.3 Niched-Pareto Genetic Algorithm (NPGA) -- 3.3 Elitist Pareto-Based Approaches -- 3.3.1 The Strength Pareto Evolutionary Algorithm (SPEA) -- 3.3.2 The Pareto Archived Evolution Strategy (PAES) -- 3.3.3 The Nondominated Sorting Genetic Algorithm-II (NSGA-II) -- 4 The Present -- 4.1 Some Applications -- 5 The Future -- 6 Conclusions -- References -- Black-Box and Data-Driven Computation -- 1 Introduction -- 2 Black Box and Oracle -- 3 Reduction -- 4 Data-Driven Computation -- References -- Mathematically Rigorous Global Optimization and FuzzyOptimization -- 1 Introduction -- 2 Interval Analysis: Fundamentals and Philosophy -- 2.1 Overview -- 2.2 Interval Logic -- 2.3 Extensions -- 2.4 History and References -- 3 Fuzzy Sets: Fundamentals and Philosophy -- 3.1 Fuzzy Logic -- 3.2 A Brief History -- 4 The Branch and Bound Framework: Some Definitions and Details -- 5 Interval Technology: Some Details -- 5.1 Interval Newton Methods -- 5.2 Constraint Propagation -- 5.3 Relaxations -- 5.4 Interval Arithmetic Software -- 6 Fuzzy Technology: A Few Details -- 7 Conclusions -- References -- Optimization Under Uncertainty Explains Empirical Success of Deep Learning Heuristics -- 1 Formulation of the Problem -- 2 Why Rectified Linear Neurons Are Efficient: A Theoretical Explanation -- 3 Why Sigmoid Activation Functions -- 4 Selection of Poolings -- 5 Why Softmax -- 6 Which Averaging Should We Choose.

7 Proofs -- References -- Variable Neighborhood Programming as a Tool of Machine Learning -- 1 Introduction -- 2 Variable Neighborhood Search -- 3 Variable Neighborhood Programming -- 3.1 Solution Presentation -- 3.2 Neighborhood Structures -- 3.3 Elementary Tree Transformation in Automatic Programming -- 3.3.1 ETT in the Tree of an Undirected Graph -- 3.3.2 ETT in AP Tree -- 3.3.3 Bound on Cardinality of AP-ETT(T) -- 4 VNP for Symbolic Regression -- 4.1 Test Instances and Parameter Values -- 4.2 Comparison of BVNP with Other Methods -- 5 Life Expectancy Estimation as a Symbolic Regression Problem Solved by VNP: Case Study on Russian Districts -- 5.1 Life Expectancy Estimation as a Machine Learning Problem -- 5.2 VNP for Estimating Life Expectancy Problem -- 5.3 Case Study at Russian Districts -- 5.3.1 One-Attribute Analysis -- 5.3.2 Results and Discussion on 3-Attribute Data -- 5.4 Conclusions -- 6 Preventive Maintenance in Railway Planning as a Machine Learning Problem -- 6.1 Literature Review and Motivation -- 6.2 Reduced VNP for Solving the Preventive Maintenance Planning of Railway Infrastructure -- 6.2.1 Learning for Stage 1: Prediction -- 6.2.2 Learning for Stage 2: Classification -- 6.3 Computation Results -- 6.3.1 Prediction -- 6.3.2 Classification -- 6.4 Conclusions and Future Work -- 7 Conclusions -- References -- Non-lattice Covering and Quantization of High Dimensional Sets -- 1 Introduction -- 2 Weak Covering -- 2.1 Comparison of Designs from the View Point of Weak Covering -- 2.2 Reduction to the Probability of Covering a Point by One Ball -- 2.3 Designs of Theoretical Interest -- 3 Approximation of Cd(Zn,r) for Design 1 -- 3.1 Normal Approximation for PU,δ,α,r -- 3.2 Refined Approximation for PU,δ,α,r -- 3.3 Approximation for Cd(Zn,r) for Design 1 -- 4 Approximating Cd(Zn,r) for Design 2a -- 4.1 Normal Approximation for PU,δ,0,r.

4.2 Refined Approximation for PU,δ,0,r -- 4.3 Approximation for Cd(Zn,r) -- 5 Approximating Cd(Zn,r) for Design 2b -- 5.1 Establishing a Connection Between Sampling with and Without Replacement: General Case -- 5.2 Approximation of Cd(Zn,r) for Design 2b. -- 6 Numerical Study -- 6.1 Assessing Accuracy of Approximations of Cd(Zn,r) and Studying Their Dependence on δ -- 6.2 Comparison Across α -- 7 Quantization in a Cube -- 7.1 Quantization Error and Its Relation to Weak Covering -- 7.2 Quantization Error for Design 1 -- 7.3 Quantization Error for Design 2a -- 7.4 Quantization Error for Design 2b -- 7.5 Accuracy of Approximations for Quantization Error and the δ-Effect -- 8 Comparative Numerical Studies of Covering Properties for Several Designs -- 8.1 Covering Comparisons -- 8.2 Quantization Comparisons -- 9 Covering and Quantization in the d-Simplex -- 9.1 Characteristics of Interest -- 9.2 Numerical Investigation of the δ-Effect for d-Simplex -- 10 Appendix: An Auxiliary Lemma -- References -- Finding Effective SAT Partitionings Via Black-Box Optimization -- 1 Introduction -- 2 Preliminaries -- 2.1 Boolean Satisfiability Problem (SAT) -- 2.2 SAT-Based Cryptanalysis -- 3 Decomposition Sets and Backdoors in SAT with Application to Inversion of Discrete Functions -- 3.1 On Interconnection Between Plain Partitionings and Cryptographic Attacks -- 3.2 Using Monte Carlo Method to Estimate Runtime of SAT-Based Guess-and-Determine Attacks -- 4 Practical Aspects of Evaluating Effectiveness of SAT Partitionings -- 4.1 Narrowing Search Space to SUPBS -- 4.2 Applications of Incremental SAT Solving -- 4.3 Finding Partitionings via Incremental SAT -- 5 Employed Optimization Algorithms -- 6 Experimental Results -- 6.1 Considered Problems -- 6.2 Implementations of Objective Functions -- 6.3 Finding Effective SAT Partitionings.