High Performance Computational Fluid Dynamics (CFD) Modeling Services for Highway Hydraulics

Print
General Information
Study Number: TPF-5(446)
Lead Organization: Federal Highway Administration
Solicitation Number: 1515
Partners: OH, SC, TX
Status: Cleared by FHWA
Est. Completion Date:
Contract/Other Number:
Last Updated: Oct 15, 2021
Contract End Date:
Financial Summary
Contract Amount:
Total Commitments Received: $460,000.00
100% SP&R Approval: Approved
Contact Information
Lead Study Contact(s): Kornel Kerenyi
kornel.kerenyi@dot.gov
Phone: 202-493-3142
FHWA Technical Liaison(s): Kornel Kerenyi
kornel.kerenyi@dot.gov
Phone: 202-493-3142
Organization Year Commitments Technical Contact Name Funding Contact Name Contact Number Email Address
Ohio Department of Transportation 2020 $160,000.00 Jeffrey Syar General Research 614-644-8135 Research@dot.state.oh.us
Ohio Department of Transportation 2022 $160,000.00 Jeffrey Syar General Research 614-644-8135 Research@dot.state.oh.us
South Carolina Department of Transportation 2021 $80,000.00 Tom Knight Terry Swygert 803-737-6691 SwygertTL@scdot.org
Texas Department of Transportation 2020 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov
Texas Department of Transportation 2021 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov
Texas Department of Transportation 2022 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov
Texas Department of Transportation 2023 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov

Study Description

The Federal Highway Administration established an Inter-Agency Agreement (IAA) with the Department of Energy’s (DOE) Argonne National Laboratory (ANL) Transportation Analysis Research Computing Center (TRACC) to get access and support for High Performance Computational Fluid Dynamics (CFD) modeling for highway hydraulics research conducted at the Turner-Fairbank Highway Research Center (TFHRC) Hydraulics Laboratory. TRACC was established in October 2006. The technical objectives of TRACC included the establishment of a high-performance computing center for use by U.S. Department of Transportation (USDOT) research teams, including those from Argonne and their university partners.

The essential resources for transportation infrastructure research and analysis at TRACC are its High Performance Computing (HPC) clusters and expert staff in the areas of computational fluid dynamics (CFD) and computational structural mechanics (CSM). Currently TRACC has two HPC clusters. The five year old Phoenix cluster is a customized system consisting of 1024 cores on 128 compute nodes, each with two quad-core AMD Opteron CPUs and 8 GB of RAM; The new HPC Zephyr cluster became available for use in October 2012. Zephyr is a 92-compute node cluster, with each node having two 16-integer core, eight floating-point core, AMD Interlagos 6273, 2.3-GHz CPUs. Eighty-eight of the nodes have 32 GB of RAM, two nodes have 64 GB of RAM, and two nodes have 128 GB of RAM. Zephyr has a 40 Gb/s InfiniBand interconnect between nodes, which is twice the speed of the Infiniband interconnects on the older Phoenix cluster. Zephyr also has a high-performance Lustre-based, highly redundant, hardware independent ZFS file system with 120 TB of formatted capacity. All nodes run an up-to-date CentOS 7. TRACC researchers have developed methodologies and software applications to more easily run CFD software (STAR-CCM+ and OpenFOAM) on TRACC’s systems. The TRACC team has also held workshops and many training classes to train TRACC users in the use of these software packages for analysis of transportation infrastructure problems.

The operation of high-performance computing platforms is particularly effective in Argonne’s context of operating some of the fastest computers in the world, co-located with TRACC’s HPC clusters. This proximity ensures access to high levels of expertise in case of hardware malfunction, and other service disruptions, as well as integration into a large-scale computing center operation.

TRACC will install and maintain the newest versions of the STAR-CCM+ and OpenFOAM CFD software used by the TRACC staff and external transportation researchers. In cooperation with the software developers, TRACC will report and resolve any issues related to software bugs or malfunctioning. As part of this objective, the TRACC team will also maintain and improve scripts that are used in conjunction with the CFD software on TRACC’s HPC clusters.

The Transportation Pooled Fund study shall allow State Departments of Transportation to get access to High Performance Computational Fluid Dynamics (CFD) computing capabilities for highway hydraulics research activities at TRACC using the Inter-Agency Agreement. State DOT shall transfer funds to FHWA using the TPF study. FHWA will then add the funds to the Inter-Agency Agreement. This will allow TRACC to conduct the State DOT’s research activity.

Objectives

The objective of these pooled funds is to provide research and analysis for a variety of highway hydraulics projects managed or coordinated by State DOTs; to provide and maintain a high performance Computational Fluid Dynamics (CFD) computing environment for application to highway hydraulics infrastructure and related projects; and to support and seek to broaden the use of CFD among State Department of Transportation employees.

Scope of Work

Task 1: Computational Mechanics Research on a Variety of Projects: The TRACC scientific staff in the computational mechanics focus area will perform research, analysis, and parametric computations as required for projects managed or coordinated by State DOTs.

Task 2: Computational Mechanics Research Support: The TRACC support team consisting of highly qualified engineers in the CFD focus areas will provide guidance to users of CFD software on an as needed or periodic basis determined by the State DOTs.

Task 3: Computing Support: The TRACC team will use the TRACC clusters for work done on projects; The TRACC system administrator will maintain the clusters and work closely with the Argonne system administrator’s community; The TRACC system administrator will also install the latest versions of the STAR-CCM+ and OpenFOAM CFD software and other software that may be required for accomplishing projects.

Comments

It is estimated that the proposed research will be $500,000. The minimum funding contribution from each partner is $15,000 per year.

The Federal Highway Administration will serve as the coordinator for this pooled-fund project. State DOT's will be solicited for their interest and participation to receive Computational Fluid Dynamics (CFD) modeling services for highway hydraulics for State DOT employees.

Subjects: Bridges, Other Structures, and Hydraulics and Hydrology

Documents Attached
Title File/Link Type Privacy Download
Acceptance Memo TPF-5(446) Acceptance Memo-TPF-5(446).pdf Memorandum Public
Quarterly Report Jan-Mar 2020 TPF-5-446-Quarterly-Report-Jan-Mar-2020.pdf Quarterly Progress Report Public
Quarterly Report Apr-Jun 2020 TPF-5-446-Quarterly-Report-April-June-2020.pdf Quarterly Progress Report Public
Quarterly Report Jul-Sep 2020 TPF-5-446-Quarterly-Report-July-Sept-2020.pdf Quarterly Progress Report Public
Quarterly Report Oct-Dec 2020 TPF-5-446-Quarterly-Report-Oct-Dec-2020.pdf Quarterly Progress Report Public
Quarterly Report Jan-Mar 2021 TPF-5-446-Quarterly-Report-January-March-2021.pdf Quarterly Progress Report Public
Quarterly Report Apr-Jun 2021 TPF-5-446-Quarterly-Report-April-June-2021.pdf Quarterly Progress Report Public
Quarterly Report Jul-Sep 2021 TPF-5-446-Quarter-3-Report-July-Sept-2021.pdf Quarterly Progress Report Public

High Performance Computational Fluid Dynamics (CFD) Modeling Services for Highway Hydraulics

General Information
Study Number: TPF-5(446)
Lead Organization: Federal Highway Administration
Solicitation Number: 1515
Partners: OH, SC, TX
Status: Cleared by FHWA
Est. Completion Date:
Contract/Other Number:
Last Updated: Oct 15, 2021
Contract End Date:
Financial Summary
Contract Amount:
Total Commitments Received: $460,000.00
100% SP&R Approval:
Contact Information
Lead Study Contact(s): Kornel Kerenyi
kornel.kerenyi@dot.gov
Phone: 202-493-3142
FHWA Technical Liaison(s): Kornel Kerenyi
kornel.kerenyi@dot.gov
Phone: 202-493-3142
Commitments by Organizations
Organization Year Commitments Technical Contact Name Funding Contact Name Contact Number Email Address
Ohio Department of Transportation 2020 $160,000.00 Jeffrey Syar General Research 614-644-8135 Research@dot.state.oh.us
Ohio Department of Transportation 2022 $160,000.00 Jeffrey Syar General Research 614-644-8135 Research@dot.state.oh.us
South Carolina Department of Transportation 2021 $80,000.00 Tom Knight Terry Swygert 803-737-6691 SwygertTL@scdot.org
Texas Department of Transportation 2020 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov
Texas Department of Transportation 2021 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov
Texas Department of Transportation 2022 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov
Texas Department of Transportation 2023 $15,000.00 Ab Maamar-Tayeb Ned Mattila 512-416-4727 ned.mattila@txdot.gov

Study Description

Study Description

The Federal Highway Administration established an Inter-Agency Agreement (IAA) with the Department of Energy’s (DOE) Argonne National Laboratory (ANL) Transportation Analysis Research Computing Center (TRACC) to get access and support for High Performance Computational Fluid Dynamics (CFD) modeling for highway hydraulics research conducted at the Turner-Fairbank Highway Research Center (TFHRC) Hydraulics Laboratory. TRACC was established in October 2006. The technical objectives of TRACC included the establishment of a high-performance computing center for use by U.S. Department of Transportation (USDOT) research teams, including those from Argonne and their university partners.

The essential resources for transportation infrastructure research and analysis at TRACC are its High Performance Computing (HPC) clusters and expert staff in the areas of computational fluid dynamics (CFD) and computational structural mechanics (CSM). Currently TRACC has two HPC clusters. The five year old Phoenix cluster is a customized system consisting of 1024 cores on 128 compute nodes, each with two quad-core AMD Opteron CPUs and 8 GB of RAM; The new HPC Zephyr cluster became available for use in October 2012. Zephyr is a 92-compute node cluster, with each node having two 16-integer core, eight floating-point core, AMD Interlagos 6273, 2.3-GHz CPUs. Eighty-eight of the nodes have 32 GB of RAM, two nodes have 64 GB of RAM, and two nodes have 128 GB of RAM. Zephyr has a 40 Gb/s InfiniBand interconnect between nodes, which is twice the speed of the Infiniband interconnects on the older Phoenix cluster. Zephyr also has a high-performance Lustre-based, highly redundant, hardware independent ZFS file system with 120 TB of formatted capacity. All nodes run an up-to-date CentOS 7. TRACC researchers have developed methodologies and software applications to more easily run CFD software (STAR-CCM+ and OpenFOAM) on TRACC’s systems. The TRACC team has also held workshops and many training classes to train TRACC users in the use of these software packages for analysis of transportation infrastructure problems.

The operation of high-performance computing platforms is particularly effective in Argonne’s context of operating some of the fastest computers in the world, co-located with TRACC’s HPC clusters. This proximity ensures access to high levels of expertise in case of hardware malfunction, and other service disruptions, as well as integration into a large-scale computing center operation.

TRACC will install and maintain the newest versions of the STAR-CCM+ and OpenFOAM CFD software used by the TRACC staff and external transportation researchers. In cooperation with the software developers, TRACC will report and resolve any issues related to software bugs or malfunctioning. As part of this objective, the TRACC team will also maintain and improve scripts that are used in conjunction with the CFD software on TRACC’s HPC clusters.

The Transportation Pooled Fund study shall allow State Departments of Transportation to get access to High Performance Computational Fluid Dynamics (CFD) computing capabilities for highway hydraulics research activities at TRACC using the Inter-Agency Agreement. State DOT shall transfer funds to FHWA using the TPF study. FHWA will then add the funds to the Inter-Agency Agreement. This will allow TRACC to conduct the State DOT’s research activity.

Objectives

The objective of these pooled funds is to provide research and analysis for a variety of highway hydraulics projects managed or coordinated by State DOTs; to provide and maintain a high performance Computational Fluid Dynamics (CFD) computing environment for application to highway hydraulics infrastructure and related projects; and to support and seek to broaden the use of CFD among State Department of Transportation employees.

Scope of Work

Task 1: Computational Mechanics Research on a Variety of Projects: The TRACC scientific staff in the computational mechanics focus area will perform research, analysis, and parametric computations as required for projects managed or coordinated by State DOTs.

Task 2: Computational Mechanics Research Support: The TRACC support team consisting of highly qualified engineers in the CFD focus areas will provide guidance to users of CFD software on an as needed or periodic basis determined by the State DOTs.

Task 3: Computing Support: The TRACC team will use the TRACC clusters for work done on projects; The TRACC system administrator will maintain the clusters and work closely with the Argonne system administrator’s community; The TRACC system administrator will also install the latest versions of the STAR-CCM+ and OpenFOAM CFD software and other software that may be required for accomplishing projects.

Comments

It is estimated that the proposed research will be $500,000. The minimum funding contribution from each partner is $15,000 per year.

The Federal Highway Administration will serve as the coordinator for this pooled-fund project. State DOT's will be solicited for their interest and participation to receive Computational Fluid Dynamics (CFD) modeling services for highway hydraulics for State DOT employees.

Subjects: Bridges, Other Structures, and Hydraulics and Hydrology

Title File/Link Type Private
Acceptance Memo TPF-5(446) Acceptance Memo-TPF-5(446).pdf Memorandum Public
Quarterly Report Jan-Mar 2020 TPF-5-446-Quarterly-Report-Jan-Mar-2020.pdf Quarterly Progress Report Public
Quarterly Report Apr-Jun 2020 TPF-5-446-Quarterly-Report-April-June-2020.pdf Quarterly Progress Report Public
Quarterly Report Jul-Sep 2020 TPF-5-446-Quarterly-Report-July-Sept-2020.pdf Quarterly Progress Report Public
Quarterly Report Oct-Dec 2020 TPF-5-446-Quarterly-Report-Oct-Dec-2020.pdf Quarterly Progress Report Public
Quarterly Report Jan-Mar 2021 TPF-5-446-Quarterly-Report-January-March-2021.pdf Quarterly Progress Report Public
Quarterly Report Apr-Jun 2021 TPF-5-446-Quarterly-Report-April-June-2021.pdf Quarterly Progress Report Public
Quarterly Report Jul-Sep 2021 TPF-5-446-Quarter-3-Report-July-Sept-2021.pdf Quarterly Progress Report Public

Currently, Transportation Pooled Fund is not supported on mobile devices, please access this Web portal using a desktop or laptop computer.