Machine intelligence integrates computation, data, models, and algorithms to solve problems that are too complex for humans. During the last three decades, machine intelligence has been a highly researched topic and widely used for solving complex real-world engineering problems. For instance, many engineering design problems can be formulated as optimization. Yet the curse of dimensionality with a large number of design variables makes the solution-searching process difficult. Similarly, to predict multiscale and multiphysics phenomena and control complex systems, models that are involved with many parameters will cause not only computational inefficiency but also inaccuracy due to the lack of data. Therefore, simplification and approximation strategies such as reduced-order modeling, surrogates, and linearization are commonly used. As a result, model-form and parameter uncertainties are inherent in the computational models for machine intelligence. Furthermore, the stochastic nature of complex systems, where random noise in the data is propagated to the models through parameter calibration and model identification, makes these analyses more challenging. The use of stochastic algorithms during the problem-solving process adds another layer of complexity for uncertainty quantification.
This special issue is dedicated to the most recent development of machine intelligence methods to solve complex engineering problems with the consideration of uncertainties. The selected collection of 14 articles shed some light on the latest trends in scientific computing and uncertainty quantification for complex problems. A wide range of topics such as model order reduction, inverse problem, sparse regression, physics-informed neural networks, and probabilistic surrogate modeling are covered, with applications in manufacturing, materials, wind farm, nuclear forensics, and transportation.
Model-form uncertainty is the major source of error in data-driven modeling, especially when model complexity is reduced to meet the limit of computational time. In the article by Azzi et al. entitled “Acceleration of a Physics-Based Machine Learning Approach for Modeling and Quantifying Model-Form Uncertainties and Performing Model Updating,” projection-based model order reduction is used for nonlinear structural dynamics models. The stochastic reduced-order bases are constructed on the compact Stiefel manifold. The dimension of the random generating matrix or the number of hyperparameters is reduced using an autoencoder. An energy-conserving sampling and weighting method is applied to sample elements in order to reduce the cost of force calculations further and, at the same time, preserve numerical stability. To enhance the model with the experimental observations, a fusion strategy is taken to identify the model’s hyperparameters by minimizing the combined loss function that includes both the model prediction bias and standard deviation associated with observations. The developed nonparametric probabilistic method is demonstrated with the structural response of a supersonic jet nozzle to a thermal load as well as the crash analysis of a vehicle front bumper. The results show that the stochastic reduced-order models can give accurate predictions in the range while achieving a wall-clock speedup factor of over 500 and a CPU speedup factor of nearly 28,000.
The selection of proper models to fit noisy data is important to reduce model-form uncertainty. “A Priori Denoising Strategies for Sparse Identification of Nonlinear Dynamical Systems: A Comparative Study” by Cortiella et al. presents a comparison of several filtering strategies and model selection techniques to improve the accuracy and robustness of sparse regression methods to recover governing equations of nonlinear dynamical systems from noisy state measurements. This analysis is very useful because measurement noise affects the stability of the recovery process yielding incorrect sparsity patterns and inaccurate estimation of coefficients of the governing equations. Several local and global noise-filtering techniques are compared for a priori state measurement denoising and estimation of state time derivatives without stringent assumptions on the noise statistics. The strategies presented in the paper can be used as guidelines for other identification algorithms that may require measurement smoothing or numerical differentiation to enhance their performance.
The consideration of underlying manifold in data during model training enables both model order reduction and sample generation. “A Probabilistic Learning Approach Applied to the Optimization of Wake Steering in Wind Farms” by Almeida et al. is a paper devoted to the wake steering control in wind farms to reduce energy losses due to wake effects and increase the energy production in a wind farm. The uncertainties in the wind conditions must be taken into account in the design. For this purpose, probabilistic learning on manifold, a recently advanced tool in supervised machine learning, is used for solving the wake steering optimization problem under uncertainties. The expected power generation is estimated in wind speed and direction with good accuracy, and an important computational cost reduction is obtained for two wind farm layouts. The analysis shows the potential gain with the application of wake steering control.
Similarly, the training data may follow certain mathematical relations or contain intrinsic features. The paper by Ries et al. entitled “A Framework for Inverse Prediction Using Functional Response Data” deals with inverse prediction in the case for which data coming from expensive physical experiments are collected in a functional form, instead of the traditional scalar form that often results in a loss of information. The proposed functional data analysis maximizes the information in the data to obtain an inverse model with a better prediction quality. Different methods for representing functional data through basis functions are introduced. The functional inverse prediction framework is a general approach, which allows for obtaining inverse probabilistic predictions in the presence of uncertainties. This methodology has the capability of flexibility. Applications to weather data and nuclear forensics are given, showing how functional models can improve the accuracy and precision of predictions.
Model calibration can be based on different sources of information. “Multi-Level Bayesian Calibration of a Multi-Component Dynamic System Model” by Kapusuzoglu et al. addresses the important problem of calibrating the values of input parameters of physical models from output measurements in the presence of uncertainties. The state-of-the-art is extended here by considering multiple coupled physical models. Some of the parameters to be calibrated are shared between models, whereas the remaining parameters only affect single models. Moreover, the data are collected as a function of time, but the different measurements may be taken at different times. This effort is accomplished here through the construction of a Bayesian network, which leads to a probabilistic description of the parameters given the measurements and the uncertainty. Both offline and online calibrations are considered. The former uses the entire data at the end of the collection phase, whereas the latter performs as data are collected. Application to the deformation and creep of gas turbine blades shows the strength of the approach. While the offline calibration leads to sharper posterior distributions, the online strategy could be applied to time-varying parameters.
Data from different sources may have varied levels of accuracy, uncertainty, and costs. How to collect and utilize data for modeling in a cost-effective way is the main theme of multi-fidelity modeling. The article written by Pidaparthi and Missoum, entitled “A Multi-Fidelity Approach for Reliability Assessment Based on the Probability of Classification Inconsistency,” introduces a new classification method based on the probabilistic support vector machine for reliability with both low- and high-fidelity sample data. While low-fidelity samples in the non-controversial region are used in the initial model construction, new samples are taken adaptively in the region where the most inconsistency between the low- and high-fidelity surrogate models occurs. The probability of inconsistency for reliability assessment according to the surrogate models is calculated to guide the sampling process. Several analytical functions are used to test the new approach. The results show that the proposed multi-fidelity approach based on the support vector machine is more efficient than the traditional single-fidelity approach based on regression with fewer limit-state function calls.
Surrogate modeling is a common approach used for predicting the uncertainty level in systems. The paper entitled “Uncertainty Quantification and Optimal Robust Design for Machining Operations” by Wan et al. focuses on balancing the conflicting desires for high material removal rate and stability from chatter when considering uncertainties in the machining process. Central to this effort is the computationally efficient determination of the domain of parameters (spindle angular velocity and material removal rate here) that corresponds, with a fixed probability, to stable machining. Starting from the mathematical model of the machining process, stochastic kriging is first performed and iteratively refined through active learning to replace computationally expensive Monte Carlo simulations. The estimation of the stability boundary corresponding to the specified probability is then performed by combing a local Gaussian process (GP) modeling and conditional value-at-risk concepts. Validation of the methodology to a representative machining model clearly demonstrates its computational efficiency.
Machine learning models are usually used as surrogates to represent and predict complex relationships. In the paper by Zhao et al., entitled “Surrogate Modeling of Nonlinear Dynamic Systems: A Comparative Study,” the performances of four surrogate models to capture nonlinear dynamics behavior, including GP, long short-term memory (LSTM) network, convolutional neural network with LSTM, and convolutional neural network with bidirectional LSTM, are compared. Examples such as Duffing oscillator and Bouc-Wen model are used for training. The results show that relatively simple GP models exhibit particularly good robustness for different cases. Deep learning models have better performances only when a good amount of training data is available. It is always good practice to start with simpler choices when selecting machine learning models. Deep learning methods do not necessarily provide a universal solution to all problems.
Utilizing additional information and prior knowledge about the quantities of interest can improve the accuracy of surrogate models. In the article entitled “Monotonic Gaussian Process for Physics-Constrained Machine Learning With Materials Science Applications,” Tran et al. applied and compared regular and monotonic GP to three datasets, focusing on the monotonicity expected in materials science applications. The first example considered in this study is fatigue life prediction under multiaxial loading, in which the results demonstrate that monotonic GP significantly outperforms regular GP. The two other examples are Potts kinetic Monte Carlo for grain growth and strain-rate-dependent stress–strain relation, which have similar results. The results of these two cases show that imposing monotonicity on regular GP slightly decreases the accuracy and performance. Moreover, the monotonic behavior seems to fade beyond the training domain, and GP may violate the monotonicity constraints. Compared to regular GP, the posterior variances of monotonic GP are lower, while their posterior means are similar.
Physics-informed machine learning is the latest approach to integrate physics-based models with data-driven machine learning methods for more accurate predictions when there is limited training data. In the article entitled “Physics-Constrained Bayesian Neural Network for Bias and Variance Reduction,” Malashkhia et al. proposed a new physics-constrained Bayesian neural network (PCBNN) framework. The proposed framework intends to reduce both variance and bias during training and improve prediction accuracy and precision. PCBNN parameters, i.e., neuron weights and biases, are given as probabilistic distributions to estimate prediction uncertainty. Unlike traditional Bayesian neural networks, PCBNN allows us to quantify model-form and parameter uncertainties in physics-informed neural networks. In physics-informed neural networks, model-form uncertainty arises from the choices of neural network architectures and physical models, whereas parameter uncertainty is caused by the lack of good-quality training data and the training algorithms themselves. The proposed adaptive optimization and minimax schemes adjust penalties and scale losses for better convergence during the training. Three examples of heat transfer and phase transition are used to test the proposed framework and loss terms. The results show that bias and variance can be reduced simultaneously.
One of the ultimate purposes of uncertainty quantification is to improve the resilience of systems. In the article by Ma et al. entitled “Characterizations and Optimization for Resilient Manufacturing Systems With Considerations of Process Uncertainties,” the resilience of manufacturing systems under the disruption of supply chains is studied. Based on a discrete-time simulation model of multi-stage production systems and a discrete-event model of disruptions, the means of machine processing rates are optimized to maximize the production. The generalized polynomial chaos expansion is applied to quantify the variability associated with the processing rates. As a result, the optimization can be done efficiently, which enables adaptive production planning where adjustments are made after the onset of disruptive events. The case study results show that the adaptive production planning can significantly reduce the downtime ratio and thus improve the resilience.
Feature recognition by machine learning in a noisy real-world environment is challenging. The study by Deng et al., entitled “Spatial Transform Depthwise Over-Parameterized Convolution Recurrent Neural Network for License Plate Recognition in Complex Environment,” aims to improve non-segmented license plate character recognition in complex environments and proposes a deep learning framework based on a depth-wise over-parameterized convolution recurrent neural network (CRNN). The developed framework contains four modules: license plate rectification module, feature extraction module, sequence annotation module, and regularized sequence decoding module. Results indicate that the proposed approach can resolve the over-fitting issue and boost recognition accuracy compared with the common CRNN with conditional entropy regularization, and it outperforms current state-of-art methods.
The selection of surrogates is important in reducing model-form uncertainty. “A Nested Weighted Tchebycheff Multi-Objective Bayesian Optimization Approach for Flexibility of Unknown Utopia Estimation in Expensive Black-Box Design Problems” by Biswas et al. presents a nested weighted Tchebycheff multi-objective Bayesian optimization approach. The weighted Tchebycheff method is a way to combine multiple objectives into a single one to perform searching more efficiently. In this work, the selection of surrogate models for Bayesian optimization is embedded as a nested loop, which enables more efficient sampling. Different regression models, such as multiple linear regression, support vector machine regression, and Gaussian process, were tested in a case study.
How to train deep neural networks for accurate predictions is also an important research topic. In the article entitled “A Quantitative Insight Into the Role of Skip Connections in Deep Neural Networks of Low Complexity: A Case Study Directed at Fluid Flow Modeling,” Choubineh et al. studied the skip connection strategy in deep forward neural networks in order to tackle the vanishing gradient issue. The vanishing gradient is the phenomenon that earlier layers of networks are not updated in the back propagation training. The skip connection strategy is to build shortcuts between layers. In the case study, convolutional neural networks are constructed to predict different basis functions for the generalized multiscale finite element method. The results show that skip connection can improve the model performance.
The guest editors would like to thank all authors for their excellent work and the reviewers for their dedication to help finish the review process with constructive comments to authors. We also thank Prof. S. K. Gupta, the former Editor-in-Chief of JCISE, for his support of this special issue. We would like to express our appreciation to Ms. Amy Suski for editorial assistance. Without your time and efforts, this special issue would not have been possible.
