Data-driven evolutionary optimization: integrating evolutionary computation, machine learning and data science

Intro -- Foreword -- Preface -- Contents -- Acronyms -- Symbols -- 1 Introduction to Optimization -- 1.1 Definition of Optimization -- 1.1.1 Mathematical Formulation -- 1.1.2 Convex Optimization -- 1.1.3 Quasi-convex Function -- 1.1.4 Global and Local Optima -- 1.2 Types of Optimization Problems -- 1.2.1 Continuous Versus Discrete Optimization -- 1.2.2 Unconstrained Versus Constrained Optimization -- 1.2.3 Single Versus Multi-objective Optimization -- 1.2.4 Deterministic Versus Stochastic Optimization -- 1.2.5 Black-Box and Data-Driven Optimization -- 1.3 Multi-objective Optimization -- 1.3.1 Mathematical Formulation -- 1.3.2 Pareto Optimality -- 1.3.3 Preference Modeling -- 1.3.4 Preference Articulation -- 1.4 Handling Uncertainty in Optimization -- 1.4.1 Noise in Evaluations -- 1.4.2 Robust Optimization -- 1.4.3 Multi-scenario Optimization -- 1.4.4 Dynamic Optimization -- 1.4.5 Robust Optimization Over Time -- 1.5 Comparison of Optimization Algorithms -- 1.5.1 Algorithmic Efficiency -- 1.5.2 Performance Indicators -- 1.5.3 Reliability Assessment -- 1.5.4 Statistical Tests -- 1.5.5 Benchmark Problems -- 1.6 Summary -- References -- 2 Classical Optimization Algorithms -- 2.1 Unconstrained Optimization -- 2.1.1 The Gradient Based Method -- 2.1.2 Newton's Method -- 2.1.3 Quasi-Newton Method -- 2.2 Constrained Optimization -- 2.2.1 Penalty and Barriers -- 2.2.2 Lagrangian Multipliers -- 2.3 Derivative-Free Search Methods -- 2.3.1 Line Search and Pattern Search -- 2.3.2 Nelder-Mead Simplex Method -- 2.3.3 Model-Based Derivative-Free Search Methods -- 2.4 Deterministic Global Optimization -- 2.4.1 Lipschitzian-Based Methods -- 2.4.2 DIRECT -- 2.5 Summary -- References -- 3 Evolutionary and Swarm Optimization -- 3.1 Introduction -- 3.2 Genetic Algorithms -- 3.2.1 Definitions -- 3.2.2 Representation -- 3.2.3 Crossover and Mutation.

3.2.4 Environmental Selection -- 3.3 Real-Coded Genetic Algorithms -- 3.3.1 Real-Valued Representation -- 3.3.2 Blended Crossover -- 3.3.3 Simulated Binary Crossover and Polynomial Mutation -- 3.4 Evolution Strategies -- 3.4.1 (1+1)-ES -- 3.4.2 Evolution Strategies with One Global Step Size -- 3.4.3 Evolution Strategies with Individual Step Sizes -- 3.4.4 Reproduction and Environmental Selection -- 3.4.5 Covariance Matrix Adaptation Evolution Strategy -- 3.5 Genetic Programming -- 3.5.1 Tree-Based Genetic Programming -- 3.5.2 Initialization -- 3.5.3 Crossover and Mutation -- 3.6 Ant Colony Optimization -- 3.6.1 Overall Framework -- 3.6.2 Extensions -- 3.7 Differential Evolution -- 3.7.1 Initialization -- 3.7.2 Differential Mutation -- 3.7.3 Differential Crossover -- 3.7.4 Environmental Selection -- 3.8 Particle Swarm Optimization -- 3.8.1 Canonical Particle Swarm Optimization -- 3.8.2 Competitive Swarm Optimizer -- 3.8.3 Social Learning Particle Swarm Optimizer -- 3.9 Memetic Algorithms -- 3.9.1 Basic Concepts -- 3.9.2 Lamarckian Versus Baldwinian Approaches -- 3.9.3 Multi-objective Memetic Algorithms -- 3.9.4 Baldwin Effect Versus Hiding Effect -- 3.10 Estimation of Distribution Algorithms -- 3.10.1 A Simple EDA -- 3.10.2 EDAs for Discrete Optimization -- 3.10.3 EDAs for Continuous Optimization -- 3.10.4 Multi-objective EDAs -- 3.11 Parameter Adaptation and Algorithm Selection -- 3.11.1 Automated Parameter Tuning -- 3.11.2 Hyper-heuristics -- 3.11.3 Fitness Landscape Analysis -- 3.11.4 Automated Recommendation Systems -- 3.12 Summary -- References -- 4 Introduction to Machine Learning -- 4.1 Machine Learning Problems -- 4.1.1 Clustering -- 4.1.2 Dimension Reduction -- 4.1.3 Regression -- 4.1.4 Classification -- 4.2 Machine Learning Models -- 4.2.1 Polynomials -- 4.2.2 Multi-layer Perceptrons -- 4.2.3 Radial-Basis-Function Networks.

4.2.4 Support Vector Machines -- 4.2.5 Gaussian Processes -- 4.2.6 Decision Trees -- 4.2.7 Fuzzy Rule Systems -- 4.2.8 Ensembles -- 4.3 Learning Algorithms -- 4.3.1 Supervised Learning -- 4.3.2 Unsupervised Learning -- 4.3.3 Reinforcement Learning -- 4.3.4 Advanced Learning Algorithms -- 4.4 Multi-objective Machine Learning -- 4.4.1 Single- and Multi-objective Learning -- 4.4.2 Multi-objective Clustering, Feature Selection and Extraction -- 4.4.3 Multi-objective Ensemble Generation -- 4.5 Deep Learning Models -- 4.5.1 Convolutional Neural Networks -- 4.5.2 Long Short-Term Memory Networks -- 4.5.3 Autoassociative Neural Networks and Autoencoder -- 4.5.4 Generative Adversarial Networks -- 4.6 Synergies Between Evolution and Learning -- 4.6.1 Evolutionary Learning -- 4.6.2 Learning for Evolutionary Optimization -- 4.7 Summary -- References -- 5 Data-Driven Surrogate-Assisted Evolutionary Optimization -- 5.1 Introduction -- 5.2 Offline and Online Data-Driven Optimization -- 5.2.1 Offline Data-Driven Optimization -- 5.2.2 Online Data-Driven Optimization -- 5.3 Online Surrogate Management Methods -- 5.3.1 Population-Based Model Management -- 5.3.2 Generation-Based Model Management -- 5.3.3 Individual-Based Model Management -- 5.3.4 Trust Region Method for Memetic Algorithms -- 5.4 Bayesian Model Management -- 5.4.1 Acquisition Functions -- 5.4.2 Evolutionary Bayesian Optimization -- 5.4.3 Bayesian Evolutionary Optimization -- 5.5 Bayesian Constrained Optimization -- 5.5.1 Acquisition Function for Constrained Optimization -- 5.5.2 Two-Stage Acquisition Functions -- 5.6 Surrogate-Assisted Robust Optimization -- 5.6.1 Bi-objective Formulation of Robust Optimization -- 5.6.2 Surrogate Construction -- 5.7 Performance Indicators for Surrogates -- 5.7.1 Accuracy -- 5.7.2 Selection-based Performance Indicator -- 5.7.3 Rank Correlation -- 5.7.4 Fitness Correlation.

5.8 Summary -- References -- 6 Multi-surrogate-Assisted Single-objective Optimization -- 6.1 Introduction -- 6.2 Local and Global Surrogates Assisted Optimization -- 6.2.1 Ensemble Surrogate Model -- 6.2.2 Multi-surrogate for Single-objective Memetic Optimization -- 6.2.3 Multi-surrogate for Multi-objective Memetic Optimization -- 6.2.4 Trust Region Method Assisted Local Search -- 6.2.5 Experimental Results -- 6.3 Two-Layer Surrogate-Assisted Particle Swarm Optimization -- 6.3.1 Global Surrogate Model -- 6.3.2 Local Surrogate Model -- 6.3.3 Fitness Estimation -- 6.3.4 Surrogate Management -- 6.3.5 Experimental Results and Discussions -- 6.4 Committee Surrogate Assisted Particle Swarm Optimization -- 6.4.1 Committee of Surrogate Models -- 6.4.2 Infill Sampling Criteria -- 6.4.3 Overall Framework -- 6.4.4 Experimental Results on Benchmark Problems -- 6.5 Hierarchical Surrogate-Assisted Multi-scenario Optimization -- 6.5.1 Multi-scenario Airfoil Optimization -- 6.5.2 Hierarchical Surrogates for Multi-scenario Optimization -- 6.6 Adaptive Surrogate Selection -- 6.6.1 Basic Idea -- 6.6.2 Probabilistic Model for Surrogate Selection -- 6.7 Summary -- References -- 7 Surrogate-Assisted Multi-objective Evolutionary Optimization -- 7.1 Evolutionary Multi-objective Optimization -- 7.1.1 Hypothesis and Methodologies -- 7.1.2 Decomposition Approaches -- 7.1.3 Dominance Based Approaches -- 7.1.4 Performance Indicator Based Approaches -- 7.2 Gaussian Process Assisted Randomized Weighted Aggregation -- 7.2.1 Challenges for Surrogate-Assisted Multi-objective Optimization -- 7.2.2 Efficient Global Optimization Algorithm -- 7.2.3 Extension to Multi-objective Optimization -- 7.3 Gaussian Process Assisted Decomposition-Based Multi-objective Optimization -- 7.3.1 MOEA/D -- 7.3.2 Main Framework -- 7.3.3 Local Surrogate Models -- 7.3.4 Surrogate Management.

7.3.5 Discussions -- 7.4 High-Dimensional Multi-objective Bayesian Optimization -- 7.4.1 Main Challenges -- 7.4.2 Heterogeneous Ensemble Construction -- 7.4.3 Pareto Approach to Multi-objective Bayesian Optimization -- 7.4.4 Overall Framework -- 7.5 Summary -- References -- 8 Surrogate-Assisted Many-Objective Evolutionary Optimization -- 8.1 New Challenges in Many-Objective Optimization -- 8.1.1 Introduction -- 8.1.2 Diversity Versus Preferences -- 8.1.3 Search for Knee Solutions -- 8.1.4 Solving Problems with Irregular Pareto Fronts -- 8.2 Evolutionary Many-Objective Optimization Algorithms -- 8.2.1 Reference Vector Guided Many-Objective Optimization -- 8.2.2 A Knee-Driven Many-Objective Optimization Algorithm -- 8.2.3 A Two-Archive Algorithm for Many-Objective Optimization -- 8.2.4 Corner Sort for Many-Objective Optimization -- 8.3 Gaussian Process Assisted Reference Vector Guided Many-Objective Optimization -- 8.3.1 Surrogate Management -- 8.3.2 Archive Maintenance -- 8.4 Classification Surrogate Assisted Many-Objective Optimization -- 8.4.1 Main Framework -- 8.4.2 Radial Projection Based Selection -- 8.4.3 Reference Set Based Dominance Relationship Prediction -- 8.4.4 Surrogate Management -- 8.4.5 Surrogate-Assisted Environmental Selection -- 8.5 Dropout Neural Network Assisted Many-Objective Optimization -- 8.5.1 AR-MOEA -- 8.5.2 Efficient Deep Dropout Neural Networks -- 8.5.3 Model Management -- 8.5.4 Overall Framework of EDN-ARMOEA -- 8.5.5 Operational Optimization in Crude Oil Distillation Units -- 8.6 Summary -- References -- 9 Knowledge Transfer in Data-Driven Evolutionary Optimization -- 9.1 Introduction -- 9.2 Co-Training for Surrogate-Assisted Interactive Optimization -- 9.2.1 Overall Framework -- 9.2.2 Surrogate for Interval Prediction -- 9.2.3 Fitness Estimation -- 9.2.4 An Improved CSSL -- 9.2.5 Surrogate Management.