The Federal Highway Administration established an Inter-Agency Agreement (IAA) with the Department of Energy’s (DOE) Argonne National Laboratory (ANL) Transportation Analysis Research Computing Center (TRACC) to get access and support for computational fluid dynamics (CFD) modeling for highway hydraulics research conducted at the Turner-Fairbank Highway Research Center (TFHRC) Hydraulics Laboratory. TRACC was established in October 2006. As part of the original project, a supercomputing user facility was established, with full operations beginning in February 2008.  The operation of the Center as a general open high-performance computing (HPC) competing facility ceased in 2011, but it has grown significantly since then, operating in support of specific sponsored research for USDOT. The essential resources for transportation infrastructure research and analysis at TRACC are its high-performance computing (HPC) clusters and expert staff in the areas of computational fluid dynamics (CFD) and computational structural mechanics (CSM). 

TRACC has operated several generations of high-performance computing clusters. Over the past three years, the computing facility has been consolidated into a new single system, ARROW, and has been upgraded with two separate 1 Petabyte highly redundant file systems, the primary file system having been optimized for reliability and performance, and the second being operated as a hot standby file system for backups and system recovery in case of catastrophic failures. Forty-seven new EPYC-based 64-core compute nodes have been added, and work seamlessly together with pre-existing compute nodes, such as the twelve 28-core XEON nodes, and around ninety older AMD 32-core nodes. The total core count is currently at about 6,300, with plans for additional nodes to be purchased.

ARROW, similar to its predecessors ZEPHYR and PHOENIX, has been heavily optimized towards running CFD and CSM codes. High-speed low-latency networks connect the various machines at 40 Gigabits per second and allow individual computers to be grouped into larger collaborative groups running a single case on hundreds of cores. This allows for obtaining results in a much shorter timeframe, using mesh partitioning and remote memory access to tackle large physical problems in an efficient manner. In addition, commercial software licenses are maintained for both CFD and CSM applications, such as STAR-CCM+ from Siemens, and LS-DYNA from ANSYS. In addition, several open-source codes are available to users of the HPC facility, such as the various versions and variants of OpenFOAM.

For the past ten years, a growing community of highway and bridge engineers at FHWA, state DOTs, and local transportation organizations have been employing the expertise of Argonne TRACC staff to analyze and develop methods to address complex problems in monitoring, maintaining, and upgrading transportation infrastructure, highways, and their interaction with vehicles. The research often addresses issues of resiliency, especially during extreme weather and flood events. The primary purpose of the IAA is to provide an expert research and analysis staff proficient in computational fluid dynamics, computational structural mechanics, and in using high-performance computing facilities to perform advanced computational analysis. The researchers have developed methodologies and software applications to run CFD and CSM software more easily on TRACC systems. The team has also held workshops and many training classes to train TRACC users in the use of these software packages for analysis of transportation infrastructure problems. The operation of high-performance computing platforms is particularly effective in Argonne’s context of operating some of the fastest computers in the world, co-located with TRACC HPC clusters. This proximity ensures access to high levels of expertise in case of hardware malfunction, and other service disruptions, as well as integration into a large-scale computing center operation. The staff will install and maintain the newest versions of the STAR-CCM+ and OpenFOAM CFD software used by the staff and external transportation researchers. In cooperation with the software developers, the staff will report and resolve any issues related to software bugs or malfunctioning. As part of this objective, the team will also maintain and improve scripts that are used in conjunction with the CFD software on TRACC’s HPC clusters. 

The Transportation Pooled Fund study shall allow State Departments of Transportation to get access to the high-performance computational fluid dynamics computing capabilities for highway hydraulics research activities at TRACC using the Inter-Agency Agreement. State DOT shall transfer funds to FHWA using the Transportation Pooled Fund study. FHWA will then add the funds to the Inter-Agency Agreement. This will allow TRACC to conduct the State DOT’s research activity.

The objective of this Transportation Pooled Fund study is to provide research and analysis for a variety of highway hydraulics projects managed or coordinated by State DOTs; to provide and maintain a high-performance computational fluid dynamics computing environment for application to highway hydraulics infrastructure and related projects; and to support and seek to broaden the use of CFD among State Department of Transportation employees.

Task 1: Computational Fluid Dynamics Research on a Variety of Projects: The scientific staff in the computational fluid dynamics focus area will perform research, analysis, and parametric computations as required for projects managed or coordinated by State DOTs.

Task 2: Computational Fluid Dynamics Research Support: The support team consisting of highly qualified engineers in the CFD focus areas will provide guidance to users of CFD software on an as-needed or periodic basis determined by the State DOTs.

Task 3: Computing Support: The team will use the TRACC clusters for work done on projects; The system administrator will maintain the clusters and work closely with the Argonne system administrator community, and will install the latest versions of the STAR-CCM+ and OpenFOAM software and other software that may be required for accomplishing projects.

It is estimated that the proposed research will be $50,000. The minimum funding contribution from each partner is $15,000 per year. The Federal Highway Administration will serve as the coordinator for this pooled-fund project. State DOT's will be solicited for their interest and participation to receive Computational Fluid Dynamics (CFD) modeling services for highway hydraulics for State DOT employees.