0
Research Papers

Dynamics Model Validation Using Time-Domain Metrics

[+] Author and Article Information
Dan Ao

Department of Civil and Environmental
Engineering,
Vanderbilt University,
279 Jacobs Hall,
VU Mailbox: PMB 351831,
Nashville, TN 37235
e-mail: dan.ao@vanderbilt.edu

Zhen Hu

Department of Civil and Environmental
Engineering,
Vanderbilt University,
279 Jacobs Hall,
VU Mailbox: PMB 351831,
Nashville, TN 37235
e-mail: zhen.hu@vanderbilt.edu

Sankaran Mahadevan

Professor
Department of Civil and Environmental
Engineering,
Vanderbilt University,
272 Jacobs Hall,
VU Mailbox: PMB 351831,
Nashville, TN 37235
e-mail: sankaran.mahadevan@vanderbilt.edu

1Corresponding author.

Manuscript received November 30, 2016; final manuscript received February 26, 2017; published online March 24, 2017. Assoc. Editor: David Moorcroft.

J. Verif. Valid. Uncert 2(1), 011004 (Mar 24, 2017) (15 pages) Paper No: VVUQ-16-1032; doi: 10.1115/1.4036182 History: Received November 30, 2016; Revised February 26, 2017

Validation of dynamics model prediction is challenging due to the involvement of various sources of uncertainty and variations among validation experiments and over time. This paper investigates quantitative approaches for the validation of dynamics models using fully characterized experiments, in which both inputs and outputs of the models and experiments are measured and reported. Existing validation methods for dynamics models use feature-based metrics to give an overall measure of agreement over the entire time history, but do not capture the model's performance at specific time instants or durations; this is important for systems that operate in different regimes in different stages of the time history. Therefore, three new validation metrics are proposed by extending the model reliability metric (a distance-based probabilistic metric) to dynamics problems. The proposed three time-domain model reliability metrics consider instantaneous reliability, first-passage reliability, and accumulated reliability. These three reliability metrics that perform time-domain comparison overcome the limitations of current feature-based validation metrics and provide quantitative assessment regarding the agreement between the simulation model and experiment over time from three different perspectives. The selection of validation metrics from a decision-making point of view is also discussed. Two engineering examples, including a simply supported beam under stochastic loading and the Sandia National Laboratories structural dynamics challenge problem, are used to illustrate the proposed time-domain validation metrics.

Copyright © 2017 by ASME
Your Session has timed out. Please sign back in to continue.

References

AIAA, 1998, “Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,” American Institute of Aeronautics and Astronautics, Reston, VA, Standard No. AIAA G-077-1998(2002).
Ling, Y. , and Mahadevan, S. , 2013, “ Quantitative Model Validation Techniques: New Insights,” Reliab. Eng. Syst. Saf., 111, pp. 217–231. [CrossRef]
Rebba, R. , Mahadevan, S. , and Huang, S. , 2006, “ Validation and Error Estimation of Computational Models,” Reliab. Eng. Syst. Saf., 91(10), pp. 1390–1397. [CrossRef]
Kleijnen, J. P. , 1995, “ Verification and Validation of Simulation Models,” Eur. J. Oper. Res., 82(1), pp. 145–162. [CrossRef]
Drignei, D. , Mourelatos, Z. P. , Kokkolaras, M. , and Pandey, V. , 2014, “ Reallocation of Testing Resources in Validating Optimal Designs Using Local Domains,” Struct. Multidiscip. Optim., 50(5), pp. 825–838. [CrossRef]
Ferson, S. , Oberkampf, W. L. , and Ginzburg, L. , 2008, “ Model Validation and Predictive Capability for the Thermal Challenge Problem,” Comput. Methods Appl. Mech. Eng., 197(29), pp. 2408–2430. [CrossRef]
Rebba, R. , and Mahadevan, S. , 2008, “ Computational Methods for Model Reliability Assessment,” Reliab. Eng. Syst. Saf., 93(8), pp. 1197–1207. [CrossRef]
Hu, Z. , and Du, X. , 2012, “ Reliability Analysis for Hydrokinetic Turbine Blades,” Renewable Energy, 48, pp. 251–262. [CrossRef]
Singh, A. , Mourelatos, Z. , and Nikolaidis, E. , 2011, “ Time-Dependent Reliability of Random Dynamic Systems Using Time-Series Modeling and Importance Sampling,” SAE Int. J. Mater. Manuf., 4(1), pp. 929–946. [CrossRef]
Zhang, J. , and Du, X. , 2011, “ Time-Dependent Reliability Analysis for Function Generator Mechanisms,” ASME J. Mech. Des., 133(3), p. 031005. [CrossRef]
Wang, Z. , Mourelatos, Z. P. , Li, J. , Singh, A. , and Baseski, I. , 2013, “ Time-Dependent Reliability of Dynamic Systems Using Subset Simulation With Splitting Over a Series of Correlated Time Intervals,” ASME Paper No. DETC2013-12257.
McFarland, J. , and Mahadevan, S. , 2008, “ Error and Variability Characterization in Structural Dynamics Modeling,” Comput. Methods Appl. Mech. Eng., 197(29), pp. 2621–2631. [CrossRef]
Sundermeyer, J. , Betts, J. F. , and Walker, M. A. , 2016, “ Simulation Time History Validation Via Multivariate Hypothesis Testing of Fourier Coefficients,” ASME Paper No. VVS2016-8844.
Marzougui, D. , Samaha, R. R. , Cui, C. , Kan, C. , and Opiela, K. S. , 2012, “ Extended Validation of the Finite Element Model for the 2010 Toyota Yaris Passenger Sedan,” National Crash Analysis Center, George Washington University, Washington, DC, Report No. NCAC 2012-W-005.
Sarin, H. , Kokkolaras, M. , Hulbert, G. , Papalambros, P. , Barbat, S. , and Yang, R.-J. , 2008, “ A Comprehensive Metric for Comparing Time Histories in Validation of Simulation Models With Emphasis on Vehicle Safety Applications,” ASME Paper No. DETC2008-49669.
Xi, Z. , Pan, H. , Fu, Y. , and Yang, R.-J. , 2015, “ Validation Metric for Dynamic System Responses Under Uncertainty,” SAE Int. J. Mater. Manuf., 8(2), pp. 309–314. [CrossRef]
Wold, S. , Esbensen, K. , and Geladi, P. , 1987, “ Principal Component Analysis,” Chemom. Intell. Lab. Syst., 2(1–3), pp. 37–52. [CrossRef]
Hu, Z. , and Mahadevan, S. , 2015, “ Time-Dependent System Reliability Analysis Using Random Field Discretization,” ASME J. Mech. Des., 137(10), p. 101404. [CrossRef]
Wang, Z. , Fu, Y. , Yang, R.-J. , Barbat, S. , and Chen, W. , 2016, “ Validating Dynamic Engineering Models Under Uncertainty,” ASME J. Mech. Des., 138(11), p. 111402. [CrossRef]
Jiang, X. , and Mahadevan, S. , 2008, “ Bayesian Wavelet Method for Multivariate Model Assessment of Dynamic Systems,” J. Sound Vib., 312(4), pp. 694–712. [CrossRef]
Jiang, X. , and Mahadevan, S. , 2011, “ Wavelet Spectrum Analysis Approach to Model Validation of Dynamic Systems,” Mech. Syst. Signal Process., 25(2), pp. 575–590. [CrossRef]
DeSimio, M. , Miller, I. , Derriso, M. , Brown, K. , and Baker, M. , 2003, “ Structural Health Monitoring Experiments With a Canonical Element of an Aerospace Vehicle,” IEEE Aerospace Conference, Big Sky, MT, Mar. 8–15, pp. 3105–3111.
Sohn, H. , Farrar, C. R. , Hunter, N. F. , and Worden, K. , 2001, “ Structural Health Monitoring Using Statistical Pattern Recognition Techniques,” ASME J. Dyn. Syst. Meas. Control, 123(4), pp. 706–711. [CrossRef]
Nichols, J. , Nichols, C. , Todd, M. , Seaver, M. , Trickey, S. , and Virgin, L. , 2004, “ Use of Data-Driven Phase Space Models in Assessing the Strength of a Bolted Connection in a Composite Beam,” Smart Mater. Struct., 13(2), p. 241. [CrossRef]
Qian, Y. , and Mita, A. , 2008, “ Acceleration-Based Damage Indicators for Building Structures Using Neural Network Emulators,” Struct. Control Health Monit., 15(6), pp. 901–920. [CrossRef]
Sankararaman, S. , and Mahadevan, S. , 2013, “ Bayesian Methodology for Diagnosis Uncertainty Quantification and Health Monitoring,” Struct. Control Health Monit., 20(1), pp. 88–106. [CrossRef]
Tobe, R. J. , 2010, “ Structural Health Monitoring of a Thermal Protection System for Fastener Failure With a Validated Model,” Ph.D. thesis, Wright State University, Dayton, OH.
Guratzsch, R. F. , 2007, “ Sensor Placement Optimization Under Uncertainty for Structural Health Monitoring Systems of Hot Aerospace Structures,” Ph.D. dissertation, Vanderbilt University, Nashville, TN.
Therrien, C. W. , 1989, Decision Estimation and Classification: An Introduction to Pattern Recognition and Related Topics, Wiley, New York.
Farrar, C. R. , and Sohn, H. , 2000, “ Pattern Recognition for Structural Health Monitoring,” Workshop on Mitigation of Earthquake Disaster by Advanced Technologies, Las Vegas, NV, Nov. 30–Dec. 1, Paper No. LA-UR-00-5565.
Yang, J. , and Chang, F.-K. , 2006, “ Detection of Bolt Loosening in C–C Composite Thermal Protection Panels—I: Diagnostic Principle,” Smart Mater. Struct., 15(2), p. 581. [CrossRef]
Jiang, X. , Mahadevan, S. , and Guratzsch, R. , 2009, “ Bayesian Wavelet Methodology for Damage Detection of Thermal Protection System Panels,” AIAA J., 47(4), pp. 942–952. [CrossRef]
Sankararaman, S. , and Mahadevan, S. , 2011, “ Model Validation Under Epistemic Uncertainty,” Reliab. Eng. Syst. Saf., 96(9), pp. 1232–1241. [CrossRef]
Mullins, J. , Ling, Y. , Mahadevan, S. , Sun, L. , and Strachan, A. , 2016, “ Separation of Aleatory and Epistemic Uncertainty in Probabilistic Model Validation,” Reliab. Eng. Syst. Saf., 147, pp. 49–59. [CrossRef]
Sankararaman, S. , and Mahadevan, S. , 2015, “ Integration of Model Verification, Validation, and Calibration for Uncertainty Quantification in Engineering Systems,” Reliab. Eng. Syst. Saf., 138, pp. 194–209. [CrossRef]
Mullins, J. , and Mahadevan, S. , 2016, “ Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction,” J. Verif. Valid. Uncertainty Quantif., 1(1), p. 011006. [CrossRef]
Haldar, A. , and Mahadevan, S. , 2000, Probability, Reliability, and Statistical Methods in Engineering Design, Wiley, New York.
Mahadevan, S. , and Rebba, R. , 2005, “ Validation of Reliability Computational Models Using Bayes Networks,” Reliab. Eng. Syst. Saf., 87(2), pp. 223–232. [CrossRef]
Rebba, R. , and Mahadevan, S. , 2006, “ Validation of Models With Multivariate Output,” Reliab. Eng. Syst. Saf., 91(8), pp. 861–871. [CrossRef]
De Maesschalck, R. , Jouan-Rimbaud, D. , and Massart, D. L. , 2000, “ The Mahalanobis Distance,” Chemom. Intell. Lab. Syst., 50(1), pp. 1–18. [CrossRef]
Halder, A. , and Bhattacharya, R. , 2011, “ Model Validation: A Probabilistic Formulation,” 50th IEEE Conference on Decision and Control and European Control Conference (CDC-ECC), Orlando, FL, Dec. 12–15, pp. 1692–1697.
Ferson, S. , and Oberkampf, W. L. , 2009, “ Validation of Imprecise Probability Models,” Int. J. Reliab. Saf., 3(1–3), pp. 3–22. [CrossRef]
Mullins, J. , and Mahadevan, S. , 2014, “ Variable-Fidelity Model Selection for Stochastic Simulation,” Reliab. Eng. Syst. Saf., 131, pp. 40–52. [CrossRef]
Ling, Y. , Mullins, J. , and Mahadevan, S. , 2014, “ Selection of Model Discrepancy Priors in Bayesian Calibration,” J. Comput. Phys., 276, pp. 665–680. [CrossRef]
Bullock, T. , McClune, M. , Achimowicz, J. , Iragui-Madoz, V. , Duckrow, R. , and Spencer, S. , 1995, “ EEG Coherence Has Structure in the Millimeter Domain: Subdural and Hippocampal Recordings From Epileptic Patients,” Electroencephalogr. Clin. Neurophysiol., 95(3), pp. 161–177. [CrossRef] [PubMed]
Grinsted, A. , Moore, J. C. , and Jevrejeva, S. , 2004, “ Application of the Cross Wavelet Transform and Wavelet Coherence to Geophysical Time Series,” Nonlinear Processes Geophys., 11(5/6), pp. 561–566. [CrossRef]
Aach, J. , and Church, G. M. , 2001, “ Aligning Gene Expression Time Series With Time Warping Algorithms,” Bioinformatics, 17(6), pp. 495–508. [CrossRef] [PubMed]
Hu, Z. , and Du, X. , 2013, “ Time-Dependent Reliability Analysis With Joint Upcrossing Rates,” Struct. Multidiscip. Optim., 48(5), pp. 893–907. [CrossRef]
Hu, Z. , and Mahadevan, S. , 2016, “ A Single-Loop Kriging Surrogate Modeling for Time-Dependent Reliability Analysis,” ASME J. Mech. Des., 138(6), p. 061406. [CrossRef]
Du, X. , and Chen, W. , 2004, “ Sequential Optimization and Reliability Assessment Method for Efficient Probabilistic Design,” ASME J. Mech. Des., 126(2), pp. 225–233. [CrossRef]
Dey, A. , and Mahadevan, S. , 1998, “ Ductile Structural System Reliability Analysis Using Adaptive Importance Sampling,” Struct. Saf., 20(2), pp. 137–154. [CrossRef]
Red-Horse, J. , and Paez, T. , 2008, “ Sandia National Laboratories Validation Workshop: Structural Dynamics Application,” Comput. Methods Appl. Mech. Eng., 197(29), pp. 2578–2584. [CrossRef]
Li, C. , and Mahadevan, S. , 2016, “ Role of Calibration, Validation, and Relevance in Multi-Level Uncertainty Integration,” Reliab. Eng. Syst. Saf., 148, pp. 32–43. [CrossRef]

Figures

Grahic Jump Location
Fig. 1

Illustration of difference between experiment and model at one validation sites: (a) experimental observation and model prediction and (b) difference

Grahic Jump Location
Fig. 2

Illustration of first-passage failure

Grahic Jump Location
Fig. 3

Three steps of the first-passage reliability metric computation at one validation site

Grahic Jump Location
Fig. 4

Simply supported beam under stochastic loads

Grahic Jump Location
Fig. 5

Simulation and experimental output: (a) good model and (b) bad model

Grahic Jump Location
Fig. 6

Validation results using time-instantaneous reliability metric for good model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 7

Validation results using the first-passage reliability metric for good model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 8

Validation results using the accumulated reliability metric for good model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 10

Validation results using the first-passage reliability metric for bad model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 11

Validation results using the accumulated reliability metric for bad model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 9

Validation results using time-instantaneous reliability metric for bad model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 12

Validation results using the first-passage reliability metric under different thresholds

Grahic Jump Location
Fig. 13

Validation results using the accumulated reliability metric under different thresholds

Grahic Jump Location
Fig. 14

Mass–spring–dampers on a beam

Grahic Jump Location
Fig. 15

Simulation and experiment data of the good model and bad model: (a) good model and (b) bad model

Grahic Jump Location
Fig. 16

Validation results using time-instantaneous reliability metric for good model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 17

Validation results using the first-passage reliability metric for good model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 18

Validation results using the accumulated reliability metric for good model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 19

Validation results using time-instantaneous reliability metric for bad model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 20

Validation results using the first-passage reliability metric for bad model: (a) λ = 8% and (b) λ = 16%

Grahic Jump Location
Fig. 21

Validation results using the accumulated reliability metric for bad model: (a) λ = 8% and (b) λ = 16%

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In