Performance Portability on HPC Accelerator Architectures with Modern Techniques:

The ParFlow Blueprint

 A Use Case by

Short description

Rapidly changing heterogeneous supercomputer architectures pose a great challenge to many scientific communities trying to leverage the latest technology in high-performance computing. Software development techniques that simultaneously result in good performance and developer productivity while keeping the codebase adaptable and well maintainable in the long-term are of high importance. ParFlow, a widely used hydrologic model based on C, achieves these attributes by using Unified Memory with a pool allocator and abstracting the architecture-dependent code in preprocessor macros (ParFlow eDSL). The implementation results in very good weak scaling with up to 26x speedup from the NVIDIA A100 GPUs over hundreds of nodes each containing 4 GPUs on the new JUWELS Booster system at the Jülich Supercomputing Centre.

Results & Achievements

The GPU support have been successfully implemented for ParFlow through using either CUDA or Kokkos (these are two separate lightweight implementations that can be used separately from each other). Especially the implementation of Kokkos constitutes a big leap in performance portability. Up to 26x speedup is achieved when using GPUs, versus using only CPUs.

Objectives

Objectives are to achieve performance portability developing and applying the ParFlow eDSL including support for coupled simulations where ParFlow is used in combination with other independently developed terrestrial models such as land-surface and atmospheric models. Multi-vendor support (i.e. performance portability) is preferred to guarantee compatibility with different supercomputer architectures.

Technologies

ParFlow, CUDA, Kokkos

Use Case Owner

FZ-Juelich IBG3

Collaborating Institutions

LLNL, FZ-Juelich

List of innovations by the CoEs, spotted by the EU innovation radar

idea-diego1100x350
The EU Innovation Radar aims to identify high-potential innovations and innovators. It is an important source of actionable intelligence on innovations emerging from research and innovation projects funded through European Union programmes. 
 
These are the innovations from the HPC Centres of Excellence as spotted by the EU innovation radar:
 
bioexcel-logo

Title: GROMACS, a versatile package to perform molecular dynamics
Market maturity: Exploring
Project: BioExcel
Innovation Topic: Excellent Science
KUNGLIGA TEKNISKA HOEGSKOLAN - SWEDEN

Cheese_logo

Title: Urgent Computing services for the impact assessment in the immediate aftermath of an earthquake
Market maturity: Tech Ready
Market creation potential: High
Project: ChEESE
Innovation Topic: Excellent Science
EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH - SWITZERLAND
BULL SAS - FRANCE

compbiomed_long_logo
ecam-300x238
eocoe
esiways logo_type_grey_left copy

Table: New coupled earth system model
Market maturity: Tech Ready
Project: ESiWACE
Innovation Topic: Excellent Science
BULL SAS - FRANCE
MET OFFICE - UNITED KINGDOM
EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS - UNITED KINGDOM
 

Excellerat_Logo_ELR_V1_20180209-01-300x106

Title: In-Situ Analysis of CFD Simulations
Market maturity: Tech Ready
Market creation potential: High
Project: Excellerat
Innovation Topic: Excellent Science
KUNGLIGA TEKNISKA HOEGSKOLAN - SWEDEN
FRAUNHOFER GESELLSCHAFT ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG E.V. - GERMAN

Title: Interactive in situ visualization in VR
Market maturity: Tech Ready
Market creation potential: High
Project: Excellerat
Innovation Topic: Excellent Science
UNIVERSITY OF STUTTGART - GERMANY

Title: Machine Learning Methods for Computational Fluid Dynamics (CFD) Data
Market maturity: Tech Ready
Market creation potential: Noteworthy
Project: Excellerat
Innovation Topic: Excellent Science
KUNGLIGA TEKNISKA HOEGSKOLAN - SWEDEN
FRAUNHOFER GESELLSCHAFT ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG E.V. - GERMAN

MaX-logo-without-subline

Title: Quantum Simulation as a Service
Market maturity: Exploring
Market creation potential: Noteworthy
Project: MaX
Innovation Topic: Excellent Science
EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH - SWITZERLAND
CINECA CONSORZIO INTERUNIVERSITARIO - ITALY

NOMAD_Logo_srgb_web_whigh
pop250

ETP4HPC handbook 2020 released

6. November 2020

The 2020 edition of the ETP4HPC Handbook of HPC projects is available. It offers a comprehensive overview over the European HPC landscape that currently consists of around 50 active projects and initiatives. Amongst these are the 14 Centres of Excellence and FocusCoE, that are also represented in this edition of the handbook.

>> Read here

Success Story EoCoE: Renewable Energy - harness the power of vorticity

CoE involved:

EoCoE is an energy-oriented Centre of Excellence for computing applications that builds on its unique expertise at the crossroads of high-performance computing (HPC) and renewable energy. It brings an impulse to accelerate the digitization of the future energy system. The coding developments are assisted by multi-disciplinary teams with expertise in applied mathematics and high performance computing (HPC).

Organisations & Codes Involved:

The end user VORTEX BLADELESS is a Spanish tech startup that is developing an environmentally friendly aerogenerator which needs no blades. It is a new wind energy technology specially designed for on-site generation in residential areas, being able to work on-grid, off- grid, or along with regular solar panels or other generators.

The domain expert, the BARCELONA SUPERCOMPUTING CENTRE is a Spanish public research center that developed the multi-physics simulation code ALYA and provided the MareNostrum supercomputer.

CHALLENGE:​

Vortex Bladeless aims to harness the power of vorticity for a new generation of wind turbines. They develop a single column without bearings or gears. It just oscillates with the wind. Experiments with scaled down prototypes have been encouraging, but the physics behind these devices is highly complex. There is a need to optimize and explore scalability due to the complexity of the flow and the need for time accurate results. The needed Large Eddy Simulation (LES) simulations are computationally demanding.

SOLUTION:​

The company has been working with experts at the Barcelona Supercomputing Centre on the MareNostrum supercomputer. The fluid-structure interaction (FSI) between the Vortex Bladeless device and a turbulent flow is simulated with Alya.

The results from initial simulations of a scaled-down device were very close to the actual wind tunnel tests performed by the Vortex Bladeless team, allowing them to develop the idea of a range of devices at the micro scale and the utility scale. Then, the behaviour of the device at a more realistic scale was studied by means of numerical simulations, helping in the design of real scale experiments and reducing costs.

Business impact:

In order to understand the aerodynamic behaviour of the devise, experimental, or numerical studies can be
performed. For the real scale model, that is around three meters high, experimental studies would have been too expensive.

It was therefore decided to resort to high accuracy numerical simulations of the flow coupled to the oscillating turbine. Since the flow is highly transient, and it is important to capture the dynamically important scales of the flow accurately, Large Eddy Simulation (LES) was selected as the modeling technique.

Modeling of turbulent flow is still nowadays one of the most computational demanding problems. The interaction with the Barcelona Supercomputing Centre was, therefore, crucial to develop a much better understanding of the aerodynamics of the device. BSC provided the computation resources and the inhouse code Alya that can efficiently run on high-end Supercomputers.

Moreover, they contributed with their know-how on LES turbulence modeling and Fluid-Structure Interaction. Technological advantages the HPC provides are important for small to medium-sized enterprises (SME) to remain competitive. For instance, the savings regarding the CPU time are key for SMEs to be able to use highly advanced techniques such as Large Eddy Simulation.

Benefits for further research:

  • Reduced costs by preparing the real scale experiments by means of numerical simulations
  • The highly optimized code enabled to avoid throwing away costly computational resources
  • The cpu time for assembly has been reduced up to 38%.
  • A new solver has provided speed ups of up to five times with respect to Alya’s own solvers.
Image: Simulation of the fluid-structure interaction (FSI) between the Vortex Bladeless device and a turbulent flow from Alya