0
Research Papers

The 2014 Sandia Verification and Validation Challenge: Problem Statement OPEN ACCESS

[+] Author and Article Information
Kenneth T. Hu

Mem. ASME
V&V, UQ, and Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828
e-mail: khu@sandia.gov

George E. Orient

Mem. ASME
V&V, UQ, and Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828
e-mail: georien@sandia.gov

1Correysponding author.

Manuscript received February 10, 2015; final manuscript received January 13, 2016; published online February 19, 2016. Assoc. Editor: Dawn Bardot.

J. Verif. Valid. Uncert 1(1), 011001 (Feb 19, 2016) (10 pages) Paper No: VVUQ-15-1012; doi: 10.1115/1.4032498 History: Received February 10, 2015; Revised January 13, 2016

This paper describes the challenge problem associated with the 2014 Sandia Verification and Validation (V&V) Challenge Workshop. The problem was developed to highlight core issues in V&V of engineering models. It is intended as an analog to projects currently underway at the Sandia National Laboratories—in other words, a realistic case study in applying V&V methods and integrating information from experimental data and simulations to support decisions. The problem statement includes the data, model, and directions for participants in the challenge. In addition, the workings of the provided code and the “truth model” used to create the data are revealed. The code, data, and truth model are available in this paper.

FIGURES IN THIS ARTICLE
<>

Sandia National Laboratories has organized a series of workshops designed to highlight specific concepts in the fields of model V&V. Each workshop was preceded by the release of one or more challenge problems. The participants from the V&V community were invited to solve the problems, present their approaches at the workshop, and then publish the work [13]. This special edition was created to document the 2014 Sandia V&V Challenge Workshop, its challenge problem, and the participants' responses. This paper, in particular, describes a case study in utilizing information from experiments and models to support a decision. The case study also serves as the challenge problem for the 2014 Sandia V&V Challenge Workshop.

The case study is described here at the level required to appreciate the following response and discussion papers. The full problem statement—as provided to the participants—is also available as a Sandia technical report [4]. This section describes the workshop motivation and how that influences the key features of the challenge problem. The remainder of the paper describes the challenge, the supplied data, and the model and code. This material is available under the “Supplemental Data” tab for this paper on the ASME Digital Collection. Further discussion of workshop goals, commentary on the problem, and a summary of the response papers are included in the introduction and summary of this special edition [3,5]. Finally, the Appendix contains details about the provided code, as well as the truth model which was used to generate the data for this challenge problem. The truth model was not revealed prior to this paper.

As defined by ASME [6] and AIAA [7], verification is “the process of determining that a model implementation accurately represents the developer's conceptual description of the model and the solution to the model,” and validation is “the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model… measured with respect to experimental data” [7]. A ubiquitous claim in literature is that V&V will build and quantify credibility, but it remains unclear how this occurs [8]. It is still an open question how the results from a V&V analysis—especially unfavorable or borderline results—aid in the determination of simulation credibility, and ultimately how that contributes to decision support. The term credibility is also used in this paper without a formal, simulation-specific definition. The general definition of credibility is the quality of being trusted and believed in. It is left to the challenge participants to expand upon this definition and describe how credibility is achieved.

The V&V field is still developing and the role of V&V within engineering projects continues to evolve. V&V concepts and methods are intertwined with those of modeling and simulation, the various scientific disciplines being modeled, systems engineering, experimental testing, and others. Each engineering organization must balance these fields to meet project objectives. The large scope and vague boundaries of the V&V field make it difficult to define a “complete” challenge problem that stresses only V&V concepts. The workshop organizers wished to pose a problem that mimicked a real-world engineering project and included many aspects of V&V, but remained feasible for the participants to complete.

This challenge first reveals a scenario where simulation plays a critical role in supporting an engineering and business decision. Second, the challenge provides data, models, and code to enable simulations and many potential analyses, including:

  1. (1)characterization of parametric uncertainty, environmental variability, and measurement noise
  2. (2)incorporation of multiple sources of data, each with a different quantity and quality
  3. (3)calibration of model parameters to match the experimental data
  4. (4)treatment of epistemic and aleatoric sources of uncertainty
  5. (5)estimation of numerical uncertainty/solution verification
  6. (6)sensitivity analysis
  7. (7)uncertainty quantification and aggregation of many sources of uncertainty
  8. (8)validation of models against the experimental data
  9. (9)assessment of relevancy of multiple sources of information
  10. (10)credibility assessment

These features give the participants the opportunity to select and apply methods (mathematical algorithms, qualitative evaluations, expert judgment, etc.) on a realistic project. The novel challenge here, which was not present in previous challenge problems, is that many sources of uncertainty must be combined or aggregated together to understand the uncertainty in the prediction. This requires balancing of resources—and coordination between V&V methods, such that the analysis results can be integrated in a meaningful way.

The possible scope is considerable and must be controlled in some way. The challenge includes a third feature: constraints designed to limit the time required for modeling, simulation, and data analysis and provide a common notation and endpoint and the probability of failure.

The model and code are provided in this paper. Any additional model or code development is prohibited for this challenge. The code is a simplified proxy for a finite-element code, which substantially reduces runtime. Providing this information removes a large source of variability among the participants, but also precludes the participants from performing code verification, or intrusive solution verification or uncertainty quantification methods. This is because the mathematical equations that the code claims to solve are not the equations that are actually being solved. This proxy code forces the participants to take a “black-box” approach to models and codes.

All data for this challenge are also provided in a format that is immediately applicable to the problem. There are several data sources and the data are limited in quantity, noisy, and inconsistent—with some data sources providing conflicting information. An important question is how the participants will utilize each dataset.

The criterion for the ultimate decision (and the required prediction) is specified. This eliminates a significant and difficult aspect of real engineering work. Questions like: “Which quantities of interest are meaningful? Which criteria are important? and How credible must the prediction be?” must be answered before models and simulation can impact decision making. These questions lie on the very edge of the V&V field and are not addressed by this challenge problem. The significant impact of this constraint is that the connection from predictions, uncertainty, and credibility to the ultimate decision is artificially fixed. By sidestepping questions about the role of simulation evidence and V&V in decision making, the participants can focus on the intermediate issue of how credibility is influenced by V&V choices, assumptions, and methods.

Despite the constraints and provided capabilities, the organizers expect that the participants will not complete the entirety of the possible tasks. The participants must prioritize their efforts among the possible analyses and then integrate the results to argue that their predictions are credible.

Mystery Liquid Co. maintains a fleet of 450 nominally identical tanks which are used to store Mystery Liquid. The tanks are located all over the world and operate in all climates. The tanks range from 4 to 12 years old, out of a planned lifetime of 20 years. None has ever experienced a physical failure, but the consequences of a failure would be significant.

The tanks are cylinders with two half-sphere end caps, as in Fig. 1. They are supported by rings around the circumference, located at the junction of the cylinder and end caps. The locations on the tank surface are described by axial distance from centerline, x, and circumferential angle from straight down, φ. The liquid contents rise to a height, H, and angle along the tank wall, α. Standard operating procedures limit the liquid level to below a certain fraction of the tank's height, and the remaining space is filled with pressurized gas. The weight of the contents plus the pressurization causes measureable deformation of the tank walls.

Each year, several tanks are inspected at random while under controlled loading. The safety tests measure displacement at various locations around the tank. This year, one tank exceeded the maximum safety specification, based on a measurement at the bottom, centerline of the tank: x=0andφ=0. The specification had been established from the design and historical performance data and had never before been violated during an inspection. The displacements are measured in the test for convenience, but it is believed that stress is the better predictor of physical failure. The location of maximum stress is not necessarily the same location as the maximum displacement.

The tank in question, tank 0, did not physically fail. There was nothing obvious about tank 0 or the testing conditions to differentiate it from other tanks. The climate, age, condition, etc. of the tank are unremarkable. Nevertheless, tank 0 and its two neighboring tanks were taken out of service and underwent testing in a lab. In addition, four tanks—in four different locations—underwent multiple tests while still in service.

Mystery Liquid Co. has commissioned a modeling project to complement these experimental tests. The historical safety margins have been violated, but it is not known how the safety test relates to physical failure. A better understanding of the margin to failure is required. The project goal is to estimate the probability of tank failure (Pfail), which will be used to decide whether the tanks must be retired/retrofitted.

The challenge problem consists of three parts:

  • Prediction: The ultimate product of this study will be the calculation of Pfail. The result should include a prediction of Pfail, plus an uncertainty estimate on that prediction.

  • Credibility assessment: In addition to the predictions, we need to know the credibility of the Pfail estimates.

  • V&V strategy: The key for providing a good credibility assessment is a logical and clearly defined strategy to gather evidence to demonstrate that the predictions are accurate.

All data for this challenge are also provided in a format that is immediately applicable to the problem. No additional experimental data can be generated.

Prediction.

Modeling and simulation will be the primary source of information used to make a decision. The specific model predictions of interest will be Pfail at the test conditions, and the operating conditions at which Pfail is equal to a threshold value.

In the first scenario, the environmental state is specified at the nominal test conditions: P=73.5psig, χ=1, and H=50in. The participants should compute Pfail and related uncertainty at this input condition. These calculations will require both model predictions and some physical failure criterion. To simplify this exercise and to ensure some level of consistency, “failure” is strictly defined based on stress. More explicitly, failure occurs when the von Mises stress exceeds the yield stress at any point on the tank's surface.

In the second scenario, Pfail is set at a threshold, P(Fail)<103, and the participants must determine the loading levels which will violate the threshold. Standard operating procedures put limits on the pressure, composition, and liquid height: P=[15,75]psig, χ=[0.1,1], and H<55in. We can assume these limits are strictly followed, but the measured operating conditions are not completely accurate, introducing another source of uncertainty.

The questions to be answered include: (1) What is the range of safe operating condition measurements, such that P(Fail)<103 ? (2) are current operating procedures enough to ensure safety? and (3) can the operating regime be safely increased, or should it be further limited?

V&V Strategy.

The V&V strategy is the overall approach for making predictions, estimating uncertainty, and assessing credibility of those predictions. The approach can be described as a collection of analyses to incorporate the experimental data and the simulation results.

The requirement for this part of the challenge problem is:

Develop and communicate a strategy for how data and models will be used to make the requested predictions, estimate the uncertainty of the prediction, and assess the credibility of those predictions.

Credibility Assessment.

At the end of a real project, the ultimate goal would be to make a decision regarding viability of the tanks. Such a decision would require knowledge of many external factors, like company finances, economics, and consequences of tank failure. This is too broad a scope for the challenge problem. Instead, the participants are asked to comment—qualitatively or quantitatively—on the credibility of their predictions. Some guiding questions include:

  • How do you communicate the results, uncertainty, and credibility?

  • How does each V&V task contribute to the credibility of the predictions of interest?

  • Does the V&V strategy as a whole add credibility?

  • What is the impact of extrapolation from the validation domain?

  • Would you feel comfortable making decisions based on your analysis?

  • How would you improve the analysis?

Quantities of Interest.

We will use quantities of interest (QoIs) to refer to: model predictions of a specific quantity, quantities derived from model predictions, quantities measured experimentally, or quantities derived from measurements.

The experimental and modeling studies must be coordinated, so that the experiments produce QoIs that will be useful for the modeling activity. In order to reduce the scope of this challenge, several QoI decisions have been made and cannot be modified, namely:

  • The first QoI is the displacement normal to the tank surface, at various, specified locations. This is directly measured during tests and is simulated in the model. It is directly available from the provided code. Displacements were used because they are easy to understand, visualize, and compute. This is not intended to be a completely realistic scenario.

  • The second QoI is the von Mises stress at arbitrary locations on the tank walls. The material is observed to fail very quickly after reaching its yield stress. Therefore, the decision has been made to equate tank failure to the event where the von Mises stress exceeds the yield stress. This is the only available failure criterion and must be used to estimate Pfail.

Note that displacement data are available from the tanks, but no stresses are ever measured. This means that the prediction of interest is based on a quantity that is never observed. However, there is a strong relationship between these two QoIs.

The data from the five experimental studies are available prior to the start of the modeling project. In addition, legacy data are available from the tank manufacturer. No further experimental data will be available before the conclusion of the project. The available data are summarized below:

  1. (1)legacy data from the manufacturerNominal material properties and tank dimensions
  2. (2)coupon tests in a controlled, lab environmentMeasured material properties and tank wall thickness
  3. (3)liquid characterization tests in a controlled lab environmentSpecific weight and composition measurements on mystery liquid
  4. (4)full tank tests in a controlled lab environmentNo pressure or liquid loading on the tank—measured dimensions (length and radius)
  5. (5)full tank tests in a controlled lab environmentPressure loading only, measured displacements at four locations
  6. (6)full tank tests in a production environmentPressure and liquid loading, measured displacement at 20 locations

“Full tank” refers to the fact that the tank is intact and fully functional. For these studies, a total of three tanks (tanks 0, 1, and 2) were removed from the field. Tank 0, the tank that exceeded its safety specification, was cut up for testing (dataset 2). The two intact tanks (tanks 1 and 2) were used for full-system testing (datasets 4 and 5). Additional tests were performed in the field on tanks 3–6 while they remained in service (dataset 6). Each dataset is described below.

Legacy Data from the Manufacturer.

The manufacturer gave specs when the tanks were delivered, however, there is no other data about manufacturing tolerances, evidence that the specs were met, or data on what changes may have occurred in the years since delivery. The information is supplied in Table 1.

Lab Tests: Material Characterization.

The failed tank (tank 0) was cut up and used for lab tests. At ten locations around the tank, two test coupons were cut away. One sample was used in a uniaxial tension test to estimate Young's modulus and yield stress. The second sample had its thickness measured before being machined into the test article used to estimate Poisson's ratio. The raw data (measurements from the tests) are not available; only the processed material property estimates are given in Table 2. The lab did not provide any uncertainty data or details about their measurements or procedures. The coupons were carefully marked, so we know that the ordering of the data in each file is consistent. For example: data point 1 in each file listed came from coupons taken from the same spot on tank 0. Unfortunately, the original tank locations of the coupons were not recorded. However, we can tell that the coupons came from a variety of locations.

Lab Tests: Specific Weight Measurements on Mystery Liquid.

In addition to the material tests, we also have lab data about the Mystery Liquid's specific weight as a function of composition. Raw data are unavailable, but an equation of state (EoS) has been fitted—see Fig. 2.

Lab Tests: Full Tank—No Loading.

The two intact tanks (tanks 1 and 2) were taken to a laboratory for testing. The tank dimensions—length and radius, were measured very accurately. The measurements are not repeats in the same locations; each measurement is from a different location over the tanks. Unfortunately, the locations were not recorded. Data are given in Table 3—note that all the measurements in this table are independent.

Lab Test: Full Tank—Pressure Loading.

Tanks 1 and 2 also underwent controlled, full-system tests. This involved tests at three pressures with two independent repeats, for a total of six tests on each tank. The data are shown in Tables 4 and 5.

  • Measurements

    1. oDisplacement measurements are taken at four locations, see Fig. 3.
    2. oIt is presumed that these are extremely accurate, within ±3% or 0.002in., whichever is greater.
    3. oPressure is measured by a gauge on the tank and should be within ±5% of the absolute pressure (gauge pressure + atmospheric pressure).

  • Measurement devices

    1. oEach tank has its own pressure gauge. These gauges are made by the same supplier and have the same calibration process.
    2. oThe displacements were measured by four contact sensors. They are considered interchangeable—the data from each location are not associated with a specific sensor.

Field Test: Full Tank—Pressure Plus Liquid Loading.

Four tanks (tanks 3–6) were tested in the field during operations. Each tank is nominally the same, but has different (unmeasured) material properties and dimensions. Three tests were done on each tank, and each test had different experimental conditions which were measured but not controlled. Data are given in Tables 69.

  • Measurements

    1. oDisplacement measurements are taken at 20 locations, see Fig. 4.
    2. oIt is presumed that these are extremely accurate, within ±3% or 0.002in., whichever is greater.
    3. oThe pressure is measured by a gauge on the tank and should be within ±5% of the gauge. This is gauge pressure.
    4. oLiquid height can be measured, but due to orientation of the tank it varies slightly with axial position. The tanks are leveled so the height difference is 2in. at the supports, x=±30.
    5. oχ is measured but with significant uncertainty. The measurement devices are only rated to be within ±0.05 mass fraction. For example, χ=0.5 measured → χ=[0.450.55] actual.
    6. oSpecific weight is not measured, but can be inferred from χ.

  • Measurement devices

    1. oThe four tanks each had their own set of pressure and liquid height gauges, but all devices had the same supplier and calibration process.
    2. oThe measurement of composition takes place offline, using liquid samples removed from the tank. Each test used a different machine since the tanks were physically in different sites.
    3. oχ is measured by a probe located near the bottom of each tank. These are permanently attached, so the location does not change. There is a moderate amount of mixing while the tanks are in use, so χ measurements are believed to be representative of the tank contents.
    4. oThe displacements were measured by contact sensors. Again the devices used in each test were different.

Separate from the experimental study, mathematical models have been developed to simulate the behavior of the materials and tanks.

Material Models.

The two items of interest are the tank walls and the mystery liquid. It is assumed the tank wall will behave as a linear elastic material under the loading scenarios of interest. Outside the linear elastic regime, the material is expected to fail. The mystery liquid is better understood and an empirical EoS was supplied from the lab to relate liquid composition, χ, and specific weight, γDisplay Formula

(1)γ=7χ1+0.25(χ0.3)28χ+5

This is an excellent empirical fit for all χ, supported by many tests. The error is less than ±2% of the measured value over the entire range of χ, as illustrated in Fig. 2.

Tank Model.

A mathematical model was developed for an idealized tank subjected to pressure and liquid loading. The pressure only loading is a special case. The model is based on a simplified geometry. Real tanks are made of a cylinder with two half-sphere end caps and supports at the ends of the cylinder. The model geometry only includes the cylinder portion with simple supports at the edges. The model computes: normal displacement, w, and stress, σ, at any tank wall position defined by x,φ (see Fig. 1 for notation and geometry illustration). The inputs are given in Table 10.

The math model is discretized into a finite-element model, with four different meshes to represent the simplified geometry. The meshes are composed of uniform, four-noded shell elements. A code is supplied to solve this model and is described below. The meshes and the computational cost of evaluating the code are given in Table 11.

Note that the meshes are not necessarily within the asymptotic regime for which Richardson extrapolation is valid. Prior verification testing has indicated that the code is correctly implemented and should provide a theoretical first-order convergence with mesh refinement. However, the current problem is more complicated than any prior verification tests.

The participants are encouraged to view and use the code as a black-box implementation of the finite-element model which accepts inputs and returns displacements and stresses. Although the code is described as a finite-element solver, this is not actually true; the details are revealed in the Appendix. The code accepts the same arguments as the math model plus a mesh size (see Table 10) and computes displacements and stresses at the requested locations. The solution depends on the mesh size, which allows for analysis of numerical errors. Code verification was not encouraged because the code is not actually solving the mathematical model described above, and code verification is not part of the challenge problem. The code is freely available as a test problem of dakota [9] and is under the Supplemental Data tab for this paper on the ASME digital collection.

The 2014 Sandia V&V Challenge Problem has been defined and described. It provides a platform to apply a wide range of V&V analyses, within a limited decision making context. The organizers intend for the participants to apply many different approaches to this challenge problem, in order to demonstrate the state-of-the-art methods, integrate many sources of data and uncertainty, and relate V&V evidence to simulation credibility. The special edition introduction [3] further describes the workshop's goals, audience, and constraints which all influenced the resulting challenge problem.

Although the system here is quite simple, it must be remembered that this system and the whole challenge problem are a proxy for a real engineering system and project. Instead of a simple tank with static pressure and liquid loading, imagine a complex mechanical device with dynamic, unpredictable loads. In this case, the system behavior and even the physics that drive the behavior are not completely understood. The geometry cannot be modeled or meshed with perfect fidelity, and the numerical accuracy of a code cannot be fully verified. The participants are encouraged to view the challenge problem in the same light.

Another note to the participants and readers: there is no correct answer to this challenge, and this is not a competition. The problem is designed to be accessible to a wide range of participants. The expectation is that the participants will focus on different aspects of the problem, subject to their own limits (on time, resources, areas of expertise, etc.). The responses cannot be “ranked” sensibly, and there is no intention of doing so. Instead, we hope to document a variety of approaches, understand their unique advantages, and examine how V&V can impact credibility.

This challenge problem and the resulting workshop were made possible with support from the Sandia National Laboratories and ASME. We wish to thank several people for their contributions and support: Brian Carnes, Vicente Romero, Laura Swiler, and Walter Witkowski at the Sandia National Laboratories, and Ryan Crane and the V&V committees at ASME. Sandia National Laboratories is a multiprogram laboratory managed and operated by the Sandia Corporation, a wholly owned subsidiary of the Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under Contract No. DE-AC04-94AL85000.

  • d =

    tank wall displacement, normal to the surface

  • E =

    Young's modulus

  • H =

    liquid height

  • L =

    length

  • M =

    mesh ID

  • P =

    gauge pressure

  • R =

    radius

  • T =

    wall thickness

  • x =

    axial location

  • γ =

    liquid-specific weight

  • ν =

    Poisson's ratio

  • σ =

    Von Mises stress

  • φ =

    circumferential angle

  • χ =

    liquid composition (mass fraction)

  • Ω, X =

    latent variables used to add dependence to material parameters

Appendix: The Provided Code and the Truth Model

The data, code, and documentation for this challenge problem are provided under the Supplemental Data tab for this paper on the ASME digital collection. Also included are some helpful matlab [10] and Linux scripts to show how to run the tank model in a dakota [11] study, as well as the truth model used to generate the data. The provided code and truth model are revealed below.

The Provided Code—A Series Solution.

The code provided to the participants was not actually a finite-element solver. Instead, the code computes a series solution with some additional manipulation of the inputs and series length to achieve desired behaviors. This series solution is based on the infinitesimal displacement formulation of linear elastic thin shell theory with simple support assumed at the ends of the cylinder [11]; the radial displacement is zero and no circumferential bending moment is transmitted to the supports. This solution only accounts for the middle, cylindrical portion of the tank and neglects the endcaps.

Since the problem is formulated in the linear elastic regime, the response to pressure and liquid weight can be superimposed to obtain the solution for the entire problem. The solution for the static liquid load is expressed in terms of a double series solution. Since this solution exhibits an interesting deformation pattern that combines bending and shear, it is a useful code verification case to verify small deformation shell element formulation in finite-element codes. It should be emphasized that this problem starts exhibiting finite deformation effects for surprisingly low levels of loading, so care must be exercised when verifying codes that do not provide infinitesimal deformation shell formulation.

To make the model's behavior more complex, the inputs to the model are manipulated to give the appearance of nonlinearities and parameter interactions. The values inputted to the code undergo a nonlinear transform before the series solution is computed. In addition, the lengths of the double series were modified based on the selection of the mesh size input, m, to achieve some asymptomatically convergent behavior. No modifications are made to the outputs.

The Truth Model.

The truth model refers to the modeling process that generated data for this challenge problem. The process required several steps and codes: matlab [10] to define and sample the finite-element model's inputs, dakota [9] to drive the simulations, cubit [12] to generate the mesh, sierra sm adagio [13] to solve the model, and matlab to add the measurement noise. Each of these is discussed below. Some relevant files are included in the Supplemental Data for use with matlab, dakota, cubit, and abaqus [14]. sierra sm is a proprietary code, so the model was recreated in abaqus for this distribution.

The data were generated in a very idealized way—a “true” set of material parameter values for each tank, a deterministic model, and additive noise. The creation of a highly realistic data generating mechanism was not the primary focus of this challenge problem. The goal was to provide a limited quantity of noisy data with a known model form error (due to incomplete geometric representation). Since such limited data were provided, the idealized data generating process was obscured by the noise and should not have been detected by the participants.

The first component was a finite-element mesh, created in cubit. The mesh fully represents the tank geometry, with a cylindrical section and two half-sphere endcaps. A quarter model was generated, taking advantage of two symmetry planes in the geometry and loading. The mesh was partitioned at the liquid surface, to aid accurate application of the hydrostatic load. Figure 5 shows the mesh and the deformation under pressure and liquid loading. The liquid level can be identified by the horizontal mesh partition and the corresponding contours, which indicate maximum displacement at the height of the liquid, on the centerline of the tank. The mesh used uniform, four-noded shell elements with roughly a 1:1 aspect ratio. Mesh sizes corresponded to the edge length of the shell elements, which were approximately 1 in. A cubit journal file describes the exact specification of the mesh and is available with the rest of the problem statement information on the Supplemental Data tab on the ASME digital collection. The mesh was regenerated for every simulation; the user can specify the tank dimensions and liquid level.

The second step was the finite-element solution, which used the sierra sm adagio code. The tanks have simple supports, as illustrated in Fig. 1, resulting in zero bending moment at the junction between cylinder and half-sphere. The pressure and the hydrostatic loads were applied to each applicable element. An implicit solution scheme was used, with residual criteria that were selected to effectively eliminate numerical error in the solution. No mesh convergence study was performed to ensure that the mesh size was adequate to eliminate discretization error; the purpose of this model was to generate data—it was deemed “correct” by definition. sierra sm adagio is not openly distributed so an abaqus model was created as a surrogate and is distributed in this paper. Excellent agreement was observed between sierra and abaqus.

The sierra model has a large number of inputs, but the data generating process only varied the parameters described previously in this problem statement. The inputs were defined by distributions and sampled using matlab. The matlab script used to generate all the provided data is included in this paper. The distributions are shown in Table 12.

The L, R, and T variables were sampled independently to create tank 0 material data and to define a “truth” value for tanks 1–6. The E,ν,andσy samples were generated by sampling Ω,X, then sampling from the conditional distributions of E|Ω, ν|Ω, and σy|Ω,X. Again, tank 0 data were sampled from the same distributions that were used to define single truth values for tanks 1–6.

In addition to varying tank parameters, the modeled loading conditions were randomized because the tests were not perfectly controlled. The modeled pressures, heights, and composition were given Gaussian noise with zero mean and standard deviations specified in Table 13. There was no variation in the relationship between χ and γ.

To generate data for each tank, simulated tests were run on the truth model. A single set of the material parameters was chosen for each tank, which were constant for all the tests. Different noise values were added to the nominal loading condition parameters for each test. The sierra model was run with the resulting values to compute displacements at the measurement locations. Model responses on the cylinder section were translated from Cartesian to cylindrical coordinates, with displacements “measured” relative to the unloaded surface in the normal direction. Finally, proportional noise was added to the simulated displacements, dmdmod*(1+N(0,0.052)), where dm is the resulting test measurement and dmod was the simulated displacement.

Helton, J. C. , and Oberkampf, W. L. , 2004, “ Alternative Representations of Epistemic Uncertainty,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 1–10. [CrossRef]
Hills, R. G. , Pilch, M. , Dowding, K. J. , Red-Horse, J. R. , Paez, T. L. , Babuska, I. , and Tempone, R. , 2008, “ Validation Challenge Workshop,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2375–2380. [CrossRef]
Hu, K. T. , 2016, “ The 2014 Sandia V&V Challenge Workshop,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 010301. [CrossRef]
Hu, K. T. , 2013, “ 2014 V&V Challenge: Problem Statement,” Sandia National Laboratories, Report No. SAND2013-10486P.
Schroeder, B. B. , Hu, K. T. , Mullins, J. G. , and Winokur, J. G. , 2016, “ Summary of the 2014 Sandia V&V Challenge Workshop,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 015501.
“ASME V&V 10-2006, 2006, Guide for Verification and Validation in Computational Solid Mechanics, ASME, New York.
1998, “ Guide for the Verification and Validation of Computational Fluid Dynamics Simulations” AIAA Paper No. G-077-1998.
Oberkampf, W. L. , and Trucano, T. G. , 2002, “ Verification and Validation in Computational Fluid Dynamics,” Prog. Aerosp. Sci., 38(3), pp. 209–272. [CrossRef]
Adams, B. M. , Ebeida, M. S. , Jakeman, J. D. , Bohnhoff, W. J. , Eddy, J. P. , Vigil, D. M. , Bauman, L. E. , et al. . 2013, “ Dakota: A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis,” Sandia National Laboratories, Report No. SAND2010-2183.
MATLAB Release, 2014, The MathWorks, Inc., Natick, MA.
Timoshenko, S. , and Woinowsky-Krieger, S. , 1987, Theory of Plates and Shells, McGraw-Hill, New York.
“  cubit,” https://cubit.sandia.gov/, Last accessed Dec. 2, 2015.
SIERRA Solid Mechanics Team, 2011, “ Sierra/SolidMechanics 4.22 User's Guide,” Sandia National Laboratories, Report No. SAND2011-7597.
ABAQUS, 2015, abaqus Documentation, Version 6.12, Dassault Systèmes, Providence, RI.
Copyright © 2016 by ASME
View article in PDF format.

References

Helton, J. C. , and Oberkampf, W. L. , 2004, “ Alternative Representations of Epistemic Uncertainty,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 1–10. [CrossRef]
Hills, R. G. , Pilch, M. , Dowding, K. J. , Red-Horse, J. R. , Paez, T. L. , Babuska, I. , and Tempone, R. , 2008, “ Validation Challenge Workshop,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2375–2380. [CrossRef]
Hu, K. T. , 2016, “ The 2014 Sandia V&V Challenge Workshop,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 010301. [CrossRef]
Hu, K. T. , 2013, “ 2014 V&V Challenge: Problem Statement,” Sandia National Laboratories, Report No. SAND2013-10486P.
Schroeder, B. B. , Hu, K. T. , Mullins, J. G. , and Winokur, J. G. , 2016, “ Summary of the 2014 Sandia V&V Challenge Workshop,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 015501.
“ASME V&V 10-2006, 2006, Guide for Verification and Validation in Computational Solid Mechanics, ASME, New York.
1998, “ Guide for the Verification and Validation of Computational Fluid Dynamics Simulations” AIAA Paper No. G-077-1998.
Oberkampf, W. L. , and Trucano, T. G. , 2002, “ Verification and Validation in Computational Fluid Dynamics,” Prog. Aerosp. Sci., 38(3), pp. 209–272. [CrossRef]
Adams, B. M. , Ebeida, M. S. , Jakeman, J. D. , Bohnhoff, W. J. , Eddy, J. P. , Vigil, D. M. , Bauman, L. E. , et al. . 2013, “ Dakota: A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis,” Sandia National Laboratories, Report No. SAND2010-2183.
MATLAB Release, 2014, The MathWorks, Inc., Natick, MA.
Timoshenko, S. , and Woinowsky-Krieger, S. , 1987, Theory of Plates and Shells, McGraw-Hill, New York.
“  cubit,” https://cubit.sandia.gov/, Last accessed Dec. 2, 2015.
SIERRA Solid Mechanics Team, 2011, “ Sierra/SolidMechanics 4.22 User's Guide,” Sandia National Laboratories, Report No. SAND2011-7597.
ABAQUS, 2015, abaqus Documentation, Version 6.12, Dassault Systèmes, Providence, RI.

Figures

Grahic Jump Location
Fig. 1

Side view and axial view of tanks—the dotted line indicates the liquid height

Grahic Jump Location
Fig. 4

Dataset 6 measurement locations

Grahic Jump Location
Fig. 3

Dataset 5 measurement locations

Grahic Jump Location
Fig. 2

Relationship between composition and specific weight for Mystery Liquid

Grahic Jump Location
Fig. 5

The truth model's mesh showing quarter symmetry plus a visualization of the deformed tank under load. The largest displacement is along the liquid surface where the mesh is partitioned.

Tables

Table Grahic Jump Location
Table 3 Dataset 4—dimensions from tanks 1 and 2. Note that all the measurements in this table are independent.
Table Grahic Jump Location
Table 6 Dataset 6—normal displacements from pressure and liquid loading on tank 3
Table Grahic Jump Location
Table 7 Dataset 6—normal displacements from pressure and liquid loading on tank 4
Table Grahic Jump Location
Table 8 Dataset 6—normal displacements from pressure and liquid loading on tank 5
Table Grahic Jump Location
Table 9 Dataset 6—normal displacements from pressure and liquid loading on tank 6
Table Grahic Jump Location
Table 2 Dataset 2—material characterization from coupon tests from tank 0
Table Grahic Jump Location
Table 1 Dataset 1—information from tank manufacturer
Table Grahic Jump Location
Table 4 Dataset 5—normal displacements from pressure loading on tank 1
Table Grahic Jump Location
Table 5 Dataset 5—normal displacements from pressure loading on tank 2
Table Grahic Jump Location
Table 10 Inputs to the math model/arguments to the provided tank code
Table Grahic Jump Location
Table 13 Standard deviations for the noise terms added to the finite-element model results to simulate test measurements
Table Footer NotePm, Hm, and χm are the measured values.
Table Grahic Jump Location
Table 11 Characteristic mesh size and computation cost for each mesh
Table Grahic Jump Location
Table 12 Input distributions used to generate tank data

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In