The Federal Highway Administration established an Inter-Agency Agreement (IAA) with the Department of Energy’s (DOE) Argonne National Laboratory (ANL) Transportation Analysis Research Computing Center (TRACC) to get access and support for High Performance Computational Fluid Dynamics (CFD) modeling for highway hydraulics research conducted at the Turner-Fairbank Highway Research Center (TFHRC) Hydraulics Laboratory. TRACC was established in October 2006. The technical objectives of TRACC included the establishment of a high-performance computing center for use by U.S. Department of Transportation (USDOT) research teams, including those from Argonne and their university partners. The essential resources for transportation infrastructure research and analysis at TRACC are its High Performance Computing (HPC) clusters and expert staff in the areas of computational fluid dynamics (CFD) and computational structural mechanics (CSM). Currently TRACC has two HPC clusters. The five year old Phoenix cluster is a customized system consisting of 1024 cores on 128 compute nodes, each with two quad-core AMD Opteron CPUs and 8 GB of RAM; The new HPC Zephyr cluster became available for use in October 2012. Zephyr is a 92-compute node cluster, with each node having two 16-integer core, eight floating-point core, AMD Interlagos 6273, 2.3-GHz CPUs. Eighty-eight of the nodes have 32 GB of RAM, two nodes have 64 GB of RAM, and two nodes have 128 GB of RAM. Zephyr has a 40 Gb/s InfiniBand interconnect between nodes, which is twice the speed of the Infiniband interconnects on the older Phoenix cluster. Zephyr also has a high-performance Lustre-based, highly redundant, hardware independent ZFS file system with 120 TB of formatted capacity. All nodes run an up-to-date CentOS 7. TRACC researchers have developed methodologies and software applications to more easily run CFD software (STAR-CCM+ and OpenFOAM) on TRACC’s systems. The TRACC team has also held workshops and many training classes to train TRACC users in the use of these software packages for analysis of transportation infrastructure problems. The operation of high-performance computing platforms is particularly effective in Argonne’s context of operating some of the fastest computers in the world, co-located with TRACC’s HPC clusters. This proximity ensures access to high levels of expertise in case of hardware malfunction, and other service disruptions, as well as integration into a large-scale computing center operation. TRACC will install and maintain the newest versions of the STAR-CCM+ and OpenFOAM CFD software used by the TRACC staff and external transportation researchers. In cooperation with the software developers, TRACC will report and resolve any issues related to software bugs or malfunctioning. As part of this objective, the TRACC team will also maintain and improve scripts that are used in conjunction with the CFD software on TRACC’s HPC clusters. The Transportation Pooled Fund study shall allow State Departments of Transportation to get access to High Performance Computational Fluid Dynamics (CFD) computing capabilities for highway hydraulics research activities at TRACC using the Inter-Agency Agreement. State DOT shall transfer funds to FHWA using the TPF study. FHWA will then add the funds to the Inter-Agency Agreement. This will allow TRACC to conduct the State DOT’s research activity.

The objective of these pooled funds is to provide research and analysis for a variety of highway hydraulics projects managed or coordinated by State DOTs; to provide and maintain a high performance Computational Fluid Dynamics (CFD) computing environment for application to highway hydraulics infrastructure and related projects; and to support and seek to broaden the use of CFD among State Department of Transportation employees.

Task 1: Computational Mechanics Research on a Variety of Projects: The TRACC scientific staff in the computational mechanics focus area will perform research, analysis, and parametric computations as required for projects managed or coordinated by State DOTs. Task 2: Computational Mechanics Research Support: The TRACC support team consisting of highly qualified engineers in the CFD focus areas will provide guidance to users of CFD software on an as needed or periodic basis determined by the State DOTs. Task 3: Computing Support: The TRACC team will use the TRACC clusters for work done on projects; The TRACC system administrator will maintain the clusters and work closely with the Argonne system administrator’s community; The TRACC system administrator will also install the latest versions of the STAR-CCM+ and OpenFOAM CFD software and other software that may be required for accomplishing projects.

It is estimated that the proposed research will be $500,000. The minimum funding contribution from each partner is $15,000 per year. The Federal Highway Administration will serve as the coordinator for this pooled-fund project. State DOT's will be solicited for their interest and participation to receive Computational Fluid Dynamics (CFD) modeling services for highway hydraulics for State DOT employees.