Abstract Field Programmable Gate Arrays (FPGA) are integrated circuits (ICs) which can implement virtually any digital function and can be configured by a designer after manufacturing. This is beneficial when dedicated application-specific runs are not time or cost-effective; however, this flexibility comes at the cost of a substantially higher interconnect overhead. Three-dimensional (3D) integration can offer significant improvements in the FPGA architecture by stacking multiple device layers and interconnecting them in the third or vertical dimension, through a substrate, where path lengths are greatly reduced. This will allow for a higher density of devices and improvements in power consumption, signal integrity, and delay. Further, it facilities heterogeneous integration where additional functionalities can be incorporated into the same package as the FPGA, such as sensors, memories, and RF/analog or photonic chips, etc. Traditionally, devices have always been laid out in a planar format. 3D integration is an architecture wherein multiple layers of planar devices are stacked and interconnected using through silicon vias (TSVs) in the vertical direction. This work will specifically detail the development of a processing and fabrication route for a three-dimensional asynchronous field programmable gate array (3D-AFPGA) design based on an extension of preexisting 2D-FPGA tile designs. Since thermal management of 3D-AFPGA is important, numerical simulations performed to predict the temperature distribution and avoid the maximum junction temperature. The numerical thermal modeling for predicting the equivalent thermal conductivity in every layer and three-dimensional temperature fields in the 3D-AFPGA are developed and discussed.

Abstract Electronic cooling represents a major portion of a Data Centers energy consumption, thus efficient thermal management dramatically impacts energy savings. This work proposes reducing the energy consumption associated with server air-cooling by vectoring (tilting) the main upward tile flow using adjacent synthetic jets. The particular fluid dynamics generated by synthetic jets allows controlling the angle at which the tile flow emanates, directing the cooling air toward areas with higher cooling demand. Three-dimensional simulations were performed using k – ε standard turbulence model with the commercial software Ansys Fluent. In order to quantify and localize the inefficiencies of the system, we estimated the Exergy Destruction distribution in the cold aisle and servers. In previous studies, this technique proved successful in finding optimum operation conditions in Data Center cooling. As opposed to a base case without flow control, the adjacent synthetic jets directed the incoming fluid to areas with higher cooling demand, thus saving energy by avoiding over-provisioning air into servers operating under normal demand. The decrease in the overall Exergy Destruction demonstrated that vectoring improves the system’s global energy efficiency.

In raised floor data centers, tiles with high open area ratio or complex understructure are used to fulfill the demand of today’s high-density computing. Using more open tiles reduces pressure drop across the raised floor with the potential advantages of increased airflow and lower noise. However, it introduces the disadvantage of increased non-uniformity of airflow distribution. In addition, there are various tile designs available on the market with different opening shapes or understructures. Furthermore, a physical separation of cold and hot aisles (containment) has been introduced to minimize the mixing of cold and hot air. In this study, three types of floor tiles with different open area, opening geometry, and understructure are considered. Experimentally validated detail models of tiles were implemented in CFD simulations to address the impact of tile design on the cooling of IT equipment in both open and enclosed aisle configurations. Also, impacts of under-cabinet leakage on the IT equipment inlet temperature in the provisioned and under-provisioned scenarios are studied. Finally, a predictive equation for the critical under-provisioning point that can lead to a no-flow condition in IT equipment with weaker airflow systems is presented.

This paper presents an experimentally validated room-level computational fluid dynamics (CFD) model for raised-floor data center configurations employing active tiles. Active tiles are perforated floor tiles with integrated fans, which increase the local volume flowrate by redistributing the cold air supplied by the computer room air conditioning (CRAC) unit to the under-floor plenum. In a previous study [1], experiments were conducted to explore the potential of active tiles for economically and efficiently eliminating hot spots in data center. Our results indicated that active tiles, as the actuators closest to the racks, can significantly and quickly impact the local distribution of cooling resources. They could therefore be used in an appropriate control framework to rapidly mitigate hot spots, and maintain local conditions in an energy-efficient manner. The numerical model of the data center room operates in an under-floor supply and ceiling return cooling configuration and consists of one cold aisle with 12 racks arranged on both sides and three CRAC units sited around the periphery of the room. The commercial computational fluid dynamics (CFD) software package Future Facilities 6SigmaDCX [2], which is specifically designed for data center simulation, is used to develop the model. First, a baseline model using only passive tiles was developed and experimental data were used to verify and calibrate plenum leakage for the room. Then a CFD model incorporating active tiles was developed for two configurations: (a) a single active tile and 9 passive tiles in the cold aisle; and (b) an aisle populated with 10 ( i.e. , all) active tiles. The active tiles are modeled as a combination of a grill, fan elements and flow blockages to closely mimic the actual active tile used in the experimental studies. The fan curve for the active tile fans is included in the model to account for changes in flow rate through the tiles in response to changes in plenum pressure. The model with active tiles is validated by comparing the flow rate through the floor tiles, relative plenum pressure and rack inlet temperatures for selected racks with the experimental measurements. The predictions from the CFD model are found to be in good agreement with the experimental data, with an average discrepancy between the measured and computed values for total flow rate and rack inlet temperature less than 4% and 1.7 °C, respectively. These validated models were then used to simulate steady state and transient scenarios following cooling failure. This physics-based and experimentally validated room-level model can be used to predict temperature and flow distributions in a data center using active tiles. These predictions can then be used to identify the optimal number and locations of active tiles to mitigate hot spots, without adversely affecting other parts of the data center.

The most common approach to air cooling of data centers involves the pressurization of the plenum beneath the raised floor and delivery of air flow to racks via perforated floor tiles. This cooling approach is thermodynamically inefficient due in large part to the pressure losses through the tiles. Furthermore, it is difficult to control flow at the aisle and rack level since the flow source is centralized rather than distributed. Distributed cooling systems are more closely coupled to the heat generating racks. In overhead cooling systems, one can distribute flow to distinct aisles by placing the air mover and water cooled heat exchanger directly above an aisle. Two arrangements are possible: (i.) placing the air mover and heat exchanger above the cold aisle and forcing downward flow of cooled air into the cold aisle (Overhead Downward Flow (ODF)), or (ii.) placing the air mover and heat exchanger above the hot aisle and forcing heated air upwards from the hot aisle through the water cooled heat exchanger (Overhead Upward Flow (OUF)). This study focuses on the steady and transient behavior of overhead cooling systems in both ODF and OUF configurations and compares their cooling effectiveness and energy efficiency. The flow and heat transfer inside the servers and heat exchangers are modeled using physics based approaches that result in differential equation based mathematical descriptions. These models are programmed in the MATLAB™ language and embedded within a CFD computational environment (using the commercial code FLUENT™) that computes the steady or instantaneous airflow distribution. The complete computational model is able to simulate the complete flow and thermal field in the airside, the instantaneous temperatures within and pressure drops through the servers, and the instantaneous temperatures within and pressure drops through the overhead cooling system. Instantaneous overall energy consumption (1st Law) and exergy destruction (2nd Law) were used to quantify overall energy efficiency and to identify inefficiencies within the two systems. The server cooling effectiveness, based on an effectiveness-NTU model for the servers, was used to assess the cooling effectiveness of the two overhead cooling approaches.

Fast Fluid Dynamics (FFD), which has its origins in video game and movie animation applications, promises faster solve times than traditional RANS (Reynolds-Averaged Navier Stokes) CFD, is relatively easy to code, and is particularly suited to parallelization. Further, FFD is capable of modeling all relevant airflow physics including momentum, buoyancy and frictional effects which are not included in a standard Potential Flow Model (PFM). The present study is a first attempt to formally evaluate FFD for data center applications in which perforated tile airflow is predicted utilizing two-dimensional plenum models. Comparisons are made to RANS CFD and Potential Flow Modeling (PFM) over a variety of data center configurations based on five basic data center layouts, most of which are based on actual data centers. Results are compared to experimental measurements for one scenario.

As the number of data centers is exponentially growing globally, pragmatic characterization schemes are considered to be a necessity for measuring and modeling the load capacity and flow pattern of the facility. This paper contains experimental and numerical characterization of a new data center laboratory using practical measurements methods, including tiles and CRAH flow measurements. Then a full physics based CFD model is built to simulate/predict the measured data. A rapid flow curve method is used showing high accuracy and low computational expense. Detailed descriptions of the data center structure, dimensions, layout (Appendix, A-1) and flow devices are given. Also, modeling parameters are mentioned in details to provide a baseline for any investigative parametric or sensitivity studies. Four experimental room level flow constraint scenarios are applied at which measurements were taken, (Appendix, A-2). The model is then built and calibrated then used to predict measurements. Measurements of the cooling unit were performed using hot wire anemometry with a traverse duct installed at the top of the CRAH. The tiles measurements were carried out using a flow hood with back pressure compensation. A detailed CFD model is constructed to predict the four experimental cases. For modeling the interdependency between the flow and pressure in flow devices flow curve approach is used. This is a rapid modeling technique that relies on experimentally measured (for IT) or approximated (for CRAH) flow curves. Applying the operational flow curves boundary conditions at the vents of the flow device results in a very accurate simulation model. It is also shown that the flow curves can be used to predict the real-time flow rate of servers at known RPM. This greatly simplifies flow rate measurements of IT in the data center.

The electronic hardware miniaturization trend continues unabated. The reduced feature spacing expose the high voltage power supply circuits to arcing. The power MOSFET gate to drain and the drain to source lead gaps are narrow enough for zinc whiskers emanating from under the raised floor zinc plated tiles to arc across the MOSFET leads. Arcing can also occur because of paper cellulose fibers and dust in high relative humidity environments. The physics of arcing will be presented. Paschen’s law of arcing will be described; it will be shown how the law led to a novel way of testing power supplies for propensity to arcing. Examples of application of the test, called the partial vacuum test, to power supplies will be described. Testing the integrity of conformal coating is one useful application of the partial vacuum test. A novel zinc whisker spray test, developed to determine the spacing required to avoid arcing between features at high electric potential between them, will be described and its results will be presented that verify the UL feature spacing guidelines.

The perpetual increase of data processing has led to an ever increasing need for power and in turn to greater cooling challenges. High density (HD) IT loads have necessitated more aggressive and direct approaches of cooling as opposed to the legacy approach by the utilization of row-based cooling. In-row cooler systems are placed between the racks aligned with row orientation; they offer cool air to the IT equipment more directly and effectively. Following a horizontal airflow pattern and typically occupying 50% of a rack’s width; in-row cooling can be the main source of cooling in the data center or can work jointly with perimeter cooling. Another important development is the use of containment systems since they reduce mixing of hot and cold air in the facility. Both in-row technology and containment can be combined to form a very effective cooling solution for HD data centers. This current study numerically investigates the behavior of in-row coolers in cold aisle containment (CAC) vs. perimeter cooling scheme. Also, we address the steady state performance for both systems, this includes manufacturer’s specifications such as heat exchanger performance and cooling coil capacity. A brief failure scenario is then run, and duration of ride through time in the case of row-based cooling system failure is compared to raised floor perimeter cooling with containment. Non-raised floor cooling schemes will reduce the air volumetric storage of the whole facility (in this small data center cell it is about a 20% reduction). Also, the varying thermal inertia between the typical in-row and perimeter cooling units is of decisive importance. The CFD model is validated using a new data center laboratory at Binghamton University with perimeter cooling. This data center consists of one main Liebert cooling unit, 46 perforated tiles with 22% open area, 40 racks distributed on three main cold aisles C and D. A computational slice is taken of the data center to generalize results. Cold aisle C consists of 16 rack and 18 perforated tiles with containment installed. In-row coolers are then added to the CFD model. Fixed IT load is maintained throughout the simulation and steady state comparisons are built between the legacy and row-based cooling schemes. An empirically obtained flow curve method is used to capture the flow-pressure correlation for flow devices. Performance scenarios were parametrically analyzed for the following cases: (a) Perimeter cooling in CAC, (b) In-row cooling in CAC. Results showed that in-row coolers increased the efficiency of supply air flow utilization since the floor leakage was eliminated, and higher pressure build up in CAC were observed. This reduced the rack recirculation when compared to the perimeter cooled case. However, the heat exchanger size demonstrated the limitation of the in-row to maintain controlled set point at increased air flow conditions. For the pump failure scenario, experimental data provided by Emerson labs were used to capture the thermal inertia effect of the cooling coils for in-row and perimeter unit, perimeter cooled system proved to have longer ride through time.

Because of the rapid growth in the number of data centers combined with the high density heat dissipation in the IT and telecommunications equipment, energy efficient thermal management of data centers has become a key research focus in the electronics packaging community. Traditional legacy data centers still rely largely on chilled air flow delivered to the IT equipment racks through perforated tiles from the raised floor plenum. When there is large variation in the amount of heat dissipated by the racks in a given aisle, the standard air cooling approach requires over-provisioning. Localized hybrid air-water cooling is one approach to more effectively control the cooling when there is wide variation in the amount of dissipation in neighboring racks. In a closed hybrid air-water cooled server cabinet, the generated heat is removed by a self-contained system that does not interact with the room level air cooling system. In this study, a comprehensive procedure for CFD validation in a close coupled hybrid cooled enclosed cabinet is described. The commercial enclosure has been characterized experimentally in an earlier study, where the effectiveness values were applied as boundary conditions to the compact heat exchanger model. Here, the previously obtained experimental data are used to validate the results from computational modeling. Two cases with different air flow rates are compared. Very good agreement is achieved, with the maximum overall average error less than 4%. Due to relatively high pressure inside the cabinet, it is possible that air leakage from the cabinet may be responsible for the discrepancy between the model and experimental results. A sensitivity study was applied to the validated model to investigate the effect leakage had on the cabinet’s performance.

Existing tools for analyzing raised-floor-plenum airflow distribution focus on the prediction of tile-by-tile airflow rates and therefore necessarily require the input of a specific floor layout. Such CFD-based tools are also typically expensive, potentially slow and are of limited availability. The present study describes a design tool which takes a somewhat more qualitative approach in which overall tile airflow uniformity and the breakdown of cooling airflow (e.g., through the perforated tiles, raised-floor leakage, and cable cutouts) are estimated from only high-level design information. The design tool’s predictions are based on both CFD simulations performed “off line” and an analytical flow network model and can be used at the concept stage in a construction project or as a learning tool to quickly demonstrate the effect of design tradeoffs on plenum airflow distribution. The applicability of the tool to a real facility is confirmed through measured data and the tool is used to investigate the effect of various design choices on perforated tile airflow uniformity and distribution.

Calibrating a CFD model against measured data is the first step to successfully utilizing this technology for change-management and the optimization of an existing data center. To date, there has been very little published on this calibration process; more focus has been placed on the use of CFD at the design stage and the development of modeling techniques and solvers. Further, few studies which feature comprehensive comparisons of CFD-predicted and measured data have been published for real data centers, and many that have, demonstrated only modest agreement at best. This study provides another such comparison — for a 7,400 ft 2 (687 m 2 ), 138-rack, raised-floor facility. The goals of the study are to benchmark the level of agreement that can be practically obtained and also to investigate the level of modeling detail required. Additionally, specific practical advice covering both CFD modeling and experimental measurements is provided. A plenum-only CFD model is compared to measured tile airflow rates and a room-model, which uses measured tile flow rates as boundary conditions, is compared to temperatures measured at each rack inlet. The level of agreement is among the best published to date and demonstrates that a CFD model can be adequately calibrated against measured data and is of value for ongoing data center operation.

Many modern data centers utilize containment systems as a practical and inexpensive thermal management solution. In this paper we investigate the effect of various containment leakage ratios on the flow rates and the temperatures in a representative data center. The modular data center is an actual contained cold aisle system in an experimental laboratory. The numerical model is validated using experimental measurements presented in a previous study for uncontained and contained cold aisle data center configurations [1]. The model takes into account the fan curves in the CRAC and IT-equipment. A commercial CFD code is used to conduct this study. The k-ε model and the Boussinesq approximation are used to model the turbulent airflow and the buoyancy effect, respectively. An approximate overall leakage model in the cold aisle is used in this study rather than a detailed model where the simulated leakage sources include floor tiles, containment surfaces, and racks. Several parametric studies are conducted by varying the leakage ratios ranging between 0% (perfectly sealed containment system) to 100% (uncontained cold aisle). The results are used to investigate the effect of leakage ratios on the flow rates of the CRAC unit and the IT-equipment. Ultimately the effect on the inlet temperatures is also investigated. Leakage in containment system is found to have a critical effect on the containment systems performance. A threshold leakage ratio is found after which the containment system does not have benefits over the conventional uncontained cold aisle system. This approach can be used to characterize the leakage in contained systems.

This study presents the results of a detailed parametric study for a data center that is air cooled using a set of four CRAC units in a cold/hot aisle raised floor configuration. The fans of the CRAC units and the servers are calibrated using their practical characteristics fan curves. A commercial CFD code is utilized for this purpose in which the buoyancy forces are taken into account. The k-epsilon model and the Boussinesq approximation are used to model the turbulent airflow and the buoyancy effect, respectively. A dynamic model is developed to take into account the changes in flow rates and power dissipation in the data center environment. The current dynamic model does not take into account the thermal mass of the CRAC units or the servers. The effect of the CRAC fan speed, instantaneous change in power dissipation, tiles perforation ratio, and servers fan speeds on the total flow rate in the room and the inlet temperatures of the racks are investigated. In the transient model, we investigate the effect of different CRAC failure scenarios on the time history of the temperatures and the flow pattern in the data center. Time constants and safe time are estimated from this study. A fundamental understanding of the effect of different data center entities on the flow and the temperatures is developed. Interesting flow patterns are observed in the case of different CRAC failures that could be used to recommend general design guidelines.

This paper presents the results of an experimentally validated Computational Fluid Dynamics (CFD) model for a data center with fully implemented fan curves on both the servers and the Computer Room Air Conditioner (CRAC). Open and contained cold aisle systems are considered experimentally and numerically. This work is divided into open (uncontained) cold aisle system calibration and validation, and fully contained cold aisle system calibration and leakage characterization. In the open system, the CRAC unit is calibrated using the manufacturer fan curve. Tiles flow measurements are used to calibrate the floor leakage. The fan curves of the load banks are generated experimentally. A full physics based model of the system is validated with two different CRAC fan speeds. The results showed a very good agreement with the tile flow measurements, with an approximate average error of 5%, indicating that the average model prediction of the tile flow is five percent lower that the measured values. In the fully contained cold aisle system, a detailed containment CFD model based on experimental measurements is developed. The model is validated by comparing the flow rate through the perforated floor tiles with the experimental measurements. The CFD results are in a good agreement with the experimental data. The average error is about 6.7%. Temperature measurements are used to calibrate other sources of containment and racks leaks including mounting rails and clearance between racks. The temperature measurements and the CFD results agree well with average error less than 2%. Detailed and equivalent modeling methods for the floor and containment system are investigated. It is found that the simple equivalent models are able to predict the flow rates however they did not succeed in providing detailed fluid flow information. While the detailed models succeeded in explaining the physical phenomena and predicting the flow rates with noticeable tradeoffs regarding the computational time. Important conclusions can be drawn from this study. In order to accurately model the containment system, both the CRAC and the load banks fan curves should be simulated in the numerical model. Unavoidable racks and containment leaks could cause inlet temperature increase even if the cold aisle is overprovisioned with cold air. It is also noted that heat conduction through the floor tiles causes a slight increase the inlet temperature of the cold aisles. Finally, it is noteworthy that using detailed modeling is necessary to understand the details of the thermal systems, however simpler and faster to compute equivalent models can be used in extended optimization studies that show relative rankings of different designs.

The work presented in this paper is an extension of the companion work by the authors on a simplified thermodynamic model for data center optimization, in which a recirculation non-uniformity metric, θ, was introduced and used in a parametric analysis to highlight the deleterious effect of recirculation non-uniformity at the inlet of racks on the data center cooling infrastructure power consumption. In this work, several studies are done using a commercial computational fluid dynamics (CFD) package to verify many of the assumptions necessary in the development of the simplified model and to understand the degree of recirculation non-uniformity present in typical data center configurations. A number of CFD simulations are used to quantify the ability of the simple model at predicting θ . The results show that the simple model provides a fairly accurate estimate of θ , with a standard deviation in the prediction error of ∼10–15%. The CFD analysis are also to understand the effect of row length and server temperature rise ( ΔT s ) temperature non-uniformity. The simulations show that reasonable values of θ range from 2–6 for open aisle data centers depending on operating strategy and data center layout. As a means to understand the effect of buoyancy, a data center Archimedes number ( Ar ), the ratio of buoyancy to inertia forces, is introduced as a function of tile flow rate and server temperature rise. For servers with modest temperature rise (∼ 10.0°C), Ar is ∼0.1; however, for racks with large temperature rise (∼20°C), Ar > 1.0, meaning buoyancy needs to be considered important. Through CFD analysis the significant effect buoyancy has on the inlet rack temperature patterns is highlighted. The Capture Index ( ψ ), the ratio of cold air ingested by the racks to the required rack flow, is used to investigate its relationship to the ratio of server flow to tile flow ( Y ), as the inlet rack temperature patterns are changed by increased Ar . The results show that although the rack inlet temperature patterns are extremely different, ψ does not change significantly as a function of Ar . Lastly, the effect of buoyancy on the assumption of linearity of the temperature field is considered for a range of Ar . The results show the emergence of a stratified temperature pattern at the inlet of the racks as Ar increases and buoyancy becomes more important. It is concluded that under these conditions, a δT change in tile temperature does not produce a δT change in temperature everywhere in the field.

In this paper we experimentally investigate the air flow distribution at the inlets of two opposing racks in a cold aisle. The racks have non uniform heat load and air flow requirements creating a heterogeneous data center environment. The Computer Room Air Conditioning (CRAC) unit fan speed is set to meet the air requirement of the high heat density rack. The effect of perforated tile air velocity on air distribution to both the racks is studied using particle image velocimetry (PIV) technique. PIV images are recorded at various rack heights corresponding to the locations of the servers in the rack. The PIV images recorded at various locations are stitched to construct a complete rack inlet air flow map. Three cases of rack air flow distributions are studied by varying the server work load and perforated tile flow rate. A significant change in the air distribution pattern is observed for the three cases investigated.

The increase in computing performance of electronic equipment is causing higher power consumption and in turn higher heat dissipation. On the larger scale, data centers that house hundreds of these individual electronic equipment are foreseeing an inevitable increase in heat dissipation on a facility level. The cooling cost of these rooms is becoming a major challenge, where a specified inlet temperature must not be exceeded in order to insure the safe and reliable operation of the electronic equipment. Most of the data centers in use today adopt the hot aisle-cold aisle configuration, where air is supplied through a plenum. The major problem with this configuration is the mixing of cold supply air from the tiles with hot exhaust air from the servers. This affects the efficiency of the cooling infrastructure and in turn affects the cost of operation. This study concentrates on the idea of sealing the cold aisle. Completely sealing the cold aisle is not practical due to pressure and noise problems, and therefore a perforated ceiling is proposed. A parametric study is generated to look at the effects of perforated ceiling resistance and different CRAC (Computer Room Air Conditioning) airflow supply percentage on the inlet temperature of the racks. A metric is proposed to quantify the variation of inlet temperature across the height of a rack and for a given cold aisle.

Typical data center architectures utilize a raised floor; cooling airflow is pumped into an under-floor plenum and exits through perforated floor tiles located in front of IT equipment racks. The under-floor space is also a convenient place to locate critical building infrastructure, such as chilled-water piping and power and network cabling. Unfortunately, the presence of such objects can disrupt the distribution of cooling airflow. While the effects of other design parameters, such as room layout, plenum depth, perforated tile type, and leakage paths, have been systematically studied — and corresponding best-practices outlined, there is no specific advice in the literature with regard to the effect of under-floor infrastructure on airflow distribution. This paper studies the effects of such obstructions primarily through CFD analyses of several layouts based on actual facilities. Additionally, corresponding scenarios are analyzed using a Potential Flow Model (PFM), which includes a recently-proposed obstruction-modeling technique. It is found that under-floor obstructions significantly affect airflow distribution only when they are located very near perforated tiles and cooling units and occupy a substantial fraction of the total plenum depth.

We present Avatar , a data center environmental advisory system for raised floor data centers. Using limited information such as inlet and reference temperatures for IT equipment and basic floor plan geometry, Avatar produces recommendation to adjust the operation of computer room air conditioners (CRACs) and the configuration of vent tiles in a data center so as to reducing excess provisioning of cooling and to remove hot spots. Avatar reduces operating expenses by cooling the same load with less energy. Avatar reduces capital expenses by recovering stranded cooling capacity that would otherwise have to be replaced.