FocusCoE at EuroHPC Summit Week 2022

With the support of the FocusCoE project, almost all European HPC Centres of Excellence (CoEs) participated once again in the EuroHPC Summit Week (EHPCSW) this year in Paris, France: the first EHPCSW in person since 2019’s event in Poland. Hosted by the French HPC agency Grand équipement national de calcul intensif  (GENCI), the conference was organised by Partnership for Advanced Computing in Europe (PRACE), the European Technology Platform for High-Performance Computing (ETP4HPC), The EuroHPC Joint Undertaking (EuroHPC JU), and the European Commission (EC).As usual, this year’s event gathered the main European HPC stakeholders from technology suppliers and HPC infrastructures to scientific and industrial HPC users in Europe.

At the workshop on the European HPC ecosystem on Tuesday 22 March at 14:45, where the diversity of the ecosystem was presented around the Infrastructure, Applications, and Technology pillars, project coordinator Dr. Guy Lonsdale from Scapos talked about FocusCoE and the CoEs’ common goal.

Later that day from 16:30 until 18:00h, the FocusCoE project hosted a session titled “European HPC CoEs: perspectives for a healthy HPC application eco-system and Exascale” involving most of the EU CoEs. The session discussed the key role of CoEs in the EuroHPC application pillar, focussing on their impact for building a vibrant, healthy HPC application eco-system and on perspectives for Exascale applications. As described by Dr. Andreas Wierse on behalf of EXCELLERAT, “The development is continuous. To prepare companies to make good use of this technology, it’s important to start early. Our task is to ensure continuity from using small systems up to the Exascale, regardless of whether the user comes from a big company or from an SME”.

Keen interest in the agenda was also demonstrated by attendees from HPC related academia and industry filling the hall to standing room only. In light of the call for new EU HPC Centres of Excellence and the increasing return to in-person events like EHPCSW, the high interest in preparing the EU for Exascale has a bright future.

FocusCoE Hosts Intel OneAPI Workshop for the EU HPC CoEs

On March 2, 2022 FocusCoE hosted Intel for a workshop introducing the oneAPI development environment. In all, over 40 researchers representing the EU HPC Centres of Excellence (CoEs)were able to attend the single day workshop to gain an overview of OneAPI. The 8 presenters from Intel gave presentations through the day covering the OneAPI vision, design, toolkits, a use case with GROMACS (which is already used by some of the EU HPC CoEs), and specific tools for migration and debugging.

Launched in 2019, the Intel OneAPI cross-industry, open, standards-based unified programming model is being designed to deliver a common developer experience across accelerator architectures. With the time saved designing for specific accelerators, OneAPI is intended to enable faster application performance, more productivity, and greater innovation. As summarized on Intel’s OneAPI website, “Apply your skills to the next innovation, and not to rewriting software for the next hardware platform.” Given the work that EU HPC CoEs are currently doing to optimise codes for Exascale HPC systems, any tools that make this process faster and more efficient can only boost CoEs capacity for innovation and preparedness for future heterogeneous systems.

The OneAPI industry initiative is also encouraging collaboration on the oneAPI specification and compatible oneAPI implementations. To that end, Intel is investing time and expertise into events like this workshop to give researchers the knowledge they need not only to use but help improve OneAPI. The presenters then also make themselves available after the workshop to answer questions from attendees on an ongoing basis. Throughout our event, participants were continuously able to ask questions and get real-time answers as well as offers for further support from software architects, technical consulting engineers, and the researcher who presented a use case. Lastly, the full video and slides from presentations are available below for any CoEs who were unable to attend or would like a second look at the detailed presentations.

NAFEMS 2021 World Congress

In the last week of October, nearly 950 participants from industry and academia gathered for the International Association for the Engineering Modeling, Analysis and Simulation 2021 World Congress (NWC21). Originally planned as a hybrid event, all programming unfortunately had to be shifted online. However, this year’s Congress was also held in conjunction with the 5th International SPDM (Simulation Process & Data Management) Conference, the NAFEMS Multiphysics Simulation Conference, and a dedicated Automotive CAE Symposium. 

Unsurprisingly, the online and truly globally accessible conference also attracted a record breaking over 550 abstract submissions. Among those accepted, EXCELLERAT and POP Centres of Excellence gave five 20-minute presentations ranging from improving airplane simulations to improving the new user HPC experience. Throughout the week Monday – Thursday, representatives from EXCELLERAT, POP, and FocusCoE also presented more general videos and materials on current projects at their 3D interactive virtual booth. This year, we welcomed over 100 visitors to the booth, and the completely new NAFEMS World of Engineering Simulation 3D experience had up to 600 conference participants strolling the virtual exhibition hall at any given time. 

Beginning on Tuesday the 26th, Ricard Borrell presented Airplane Simulations on Heterogeneous Pre-Exascale Architectures on behalf of EXCELLERAT CoE. He used the example of improved airplane aerodynamics simulations to discuss how their Alya code is preparing researchers to fully utilise the next generation of Exascale HPC systems and their heterogeneous hardware architectures. Without codes optimised like Alya, current HPC codes wouldn’t be capable of running in parallel on different types of processors to the extent that new Exascale systems will require. Thus, they would use only a fraction of the speed and processing power available at Exascale.

Continuing in this theme, Amgad Dessoky presented an overview of how the EXCELLERAT Centre of Excellence is Paving the Way for the Development to Exascale Multiphysics Simulations including the 6 computational codes they are optimizing for Exascale: Nek5000, Alya, AVBP, TPLS, FEniCS, and Coda. Using automotive, aerospace, energy, and manufacturing industrial sector use cases, he described the common challenges for developers and potential users of HPC Exascale applications. Participants were also invited to discuss how EXCELLERAT can best support the needs and required competencies of the Multiphysics simulation community going forward.

One of these required competencies is the ability to understand and improve the performance bottlenecks of parallel codes. In answer, Federico Panichi presented a talk titled Improving the Performance of Engineering Codes on behalf of the Performance Optimisation and Productivity (POP) Centre of Excellence. The set of hierarchical metrics forming the POP performance assessment methodology cuts out the time-consuming trial and error process of code optimisation by identifying issues such as memory bottlenecks, communication inefficiencies, and load imbalances. He demonstrated with the example of optimised engineering codes how POP services could enable any EU or UK academic or commercial organisation to speed up time to solution, solve larger, more challenging problems or reduce compute costs free of charge.

On Wednesday the 27th, presentations continued with Christian Gscheidle representing EXCELLERAT and A Flexible and Efficient In-situ Data Analysis Framework for CFD Simulations. Because Computational Fluid Dynamics (CFD) simulations Increasingly produce far more data than they can save in real time, researchers often see final analysis results and lose the intermediate data. It also means that they must wait until the full simulation runs before being able to make any improvements. In contrast, he presented the EXCELLERAT tool that uses machine learning to perform in-situ analyses on data produced during large-scale simulations in real time so that researchers can see intermediate results and early trends. Using this tool, an example HVAC duct from an automotive set up use case showed a reduced compute time because researchers were able to abort simulations with unwanted behaviour.

Wrapping up the presentations, Janik Schüssler presented Creating Connections: Enabling High Performance Computing for Industry through a Data Exchange & Workflow Platform on behalf of EXCELLERAT. Continuing in the theme of the previous presentations, this tool addresses the needs and competencies of current and potential HPC users, albeit in a slightly different way. As discussed in the talk, where other tools or trainings work to develop HPC competencies in the user, this prototype platform would bring HPC to the user both in terms of competencies and location. The user would be able to remotely access clusters (currently both HLRS Hawk and Vulcan) from anywhere using an authorised device. Additionally, they would be able to run simulations in the web front end without any command line interactions, which can have a steep learning curve for new users. Ultimately, the savings in logistical coordination, travel, and training time would drastically lower the entry barrier to HPC use.

If you’re interested in following how EXCELLERAT, POP, and all the European HPC Centres of Excellence are preparing us for Exascale and improving the HPC user experience, sign up for our newsletters:

EXCELLERAT CoE Newsletter

POP CoE Newsletter

Focus CoE Newsletter (for coverage of all European HPC CoEs)

POP and PerMed Centers of Excellence are Getting Cell-Level Simulations Ready for Exascale

Collaboration of CoEs workflow

Success story # Highlights:

  • Industry sector: Computational biology
  • Key codes used: PhysiCell
  • Keywords:
    • Memory management
    • Cell-cell interaction simulation
    • OpenMP
    • Good practice

Organisations & Codes Involved:

Technical Challenge:

One of PerMedCoE’s use cases is the study of tumour evolution based on single-cell omics and imaging using PhysiCell. In order to simulate such large-scale problems that replicate real-world tumours, High Performance Computing (HPC) is essential. However, memory usage presents a challenge in HPC architectures and is one of the obstacles to optimizing simulations for running at Exascale.

With a high number of threads computing in parallel, performance can be degraded because of concurrent memory allocations and deallocations.  We observe that as the number of threads goes up, runtime is not reduced as expected. This is not a memory-bound problem, but a memory management one.

Solution:

Following the POP methodology and using the BSC tools (Extrae, Paraver, and BasicAnalysis), we determine that an overloading of a specific implementation of C++ operators is causing a high number of concurrent memory allocations and deallocations, causing the memory management library to perform synchronization and “steal” cycles of CPU from the running application.

The proposal from POP is to avoid the overloading of operators that need allocation and deallocation of data structures. However, achieving this implies a major code change. To demonstrate the potential of the suggestion without doing the major code change, we suggest using an external library to improve the memory management: Jemalloc.

Jemalloc is “a general purpose malloc implementation that emphasizes fragmentation avoidance and scalable concurrency support” [1]. It can be integrated easily into any code by preloading the library, with LD_PRELOAD in Linux for example. After applying this solution to PhysiCell, we execute the same experiment and obtain a 1.45x speed up when using 48 OpenMP threads.

[1] cited from Jemalloc website. http://jemalloc.net

Business impact:

As the only transversal CoE, one of POP’s objectives is to advise and help the other HPC CoEs prepare their codes for the Exascale. This collaboration between POP and PerMed is a good example of the potential of these kinds of partnerships between CoEs.

POP provides the performance analysis expertise and tools, the experience of hundreds of codes analysed, and the best practices gathered from those analyses. PerMedCoE bring state of the art use cases and real-world problems to be solved. Together, they improve the performance and efficiency of cell-level agent-based simulation software: in this case PhysiCell.

One of the goals of PerMedCoE is the scaling-up of four different tools that address different types of simulations in personalized medicine and that were coded in different languages (C++, R, python, julia). These tools are being re-factored to scale up to Exascale. In this scaling-up, audits such as the ones POP can offer are essential to evaluate past developments and guide future ones.

Currently, only one of PerMed’s tools is able to use several nodes to run a single simulation, whereas the rest of the tools can only use all of the processors of a single node. Our scaling-up strategy is a heterogeneous one.  We are implementing MPI on some tools, re-factoring others to other languages that can ease the use of HPC clusters such as Julia, or even targeting “many-task computing“ paradigms like in the case of model fitting.

The main motivation to have HPC versions of these simulation tools is to be able to simulate bigger, more complex cell-level agent-based models. Current models can obtain up to 10^6 cells, but it has been proven that most of the problems addressed in computational biology (cancerogenesis, cell lines growth, COVID-19 infection) usually target from 10^9 to 10^12 cells. In addition, most of these current simulations consider an over-simplified environment that is nowhere close to a real-life scenario. We aim to have complex multi-scale simulation frameworks that target these bigger, more complex simulations.

Benefits:

  • Speedup of 1.45x on runtime with 48 threads in PhysiCell execution
  • A good practice for High Performance C++ applications exported
  • This behaviour in memory management systems will be detected more easily in future analysis

CoEs at Teratec Forum 2021 and ISC21

15. June 2021

With the support of FocusCoE, a number of HPC CoEs will give short presentations at the virtual PRACE booth in the following two HPC-related events: Teratec Forum 2021 and ISC2021 that will take place towards the end of this month. See the schedule below for more details. Please reserve the slots in your calendars, registration details will be provided on the PRACE website soon!

“We are happy to see that FocusCoE was able to help the HPC CoEs to have a significant presence at this year’s editions of ISC and Teratec Forum, two major HPC events, enabled through our good synergies with PRACE”, says Guy Lonsdale, FocusCoE coordinator.

 

Teratec Forum 2021 schedule

Date / Event

Time slot CEST

Title

Speaker

Organisation

Tue 22 June

11:00 – 11:15

EoCoE-II: Towards exascale for Energy

Edouard Audit, EoCoE-II coordinator

CEA (France)

 

14:30 – 14:45

POP CoE: Free Performance Assessments for the HPC Community

Bernd Mohr

 Jülich Supercomputing Centre

 Thu 24 June

13:45 – 14:00

EXCELLERAT – paving the way for the evolution towards Exascale

Amgad Dessoky / Sophia Honisch

HLRS

 

ISC 2021 schedule

Date / Event

Time slot CEST

Title

Speaker

Organisation

 Thu 24 June

13:45 – 14:00

EXCELLERAT – paving the way for the evolution towards Exascale

Amgad Dessoky / Sophia Honisch

HLRS

Fri 25 June

11:00 – 11:15

The Center of Excellence for Exascale in Solid Earth (ChEESE)

Alice-Agnes Gabriel

Geophysik, University of Munich

 

15:30 – 15:45

EoCoE-II: Towards exascale for Energy

Edouard Audit, EoCoE-II coordinator

CEA (France)

Tue 29 June

11:00 – 11:15

Towards a maximum utilization of synergies of HPC Competences in Europe

Bastian Koller, HLRS

HLRS

Wed 30 June
ISC2021

10:45 -11:00

CoE
RAISE: Bringing AI to Exascale

Dr.-Ing. Andreas Lintermann

Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH

Thu 1 July
ISC2021

11:00 -11:15

POP CoE: Free Performance Assessments for the HPC Community

Bernd Mohr

 Jülich Supercomputing Centre

 

14:30 -14:45

 TREX: an innovative view of HPC usage applied to Quantum Monte Carlo simulations

 Anthony Scemama (1), William Jalby (2), Cedric Valensi (2), Pablo de Oliveira Castro (2)

(1) Laboratoire de Chimie et Physique Quantiques, CNRS-Université Paul Sabatier, Toulouse, France

(2) Université de Versailles St-Quentin-en-Yvelines, Université Paris Saclay, France

 

Please register to the short presentations through the PRACE event pages here:

PRACE Virtual booth at Teratec Forum  2021PRACE Virtual booth at ISC2021
prace-ri.eu/event/teratec-forum-2021/prace-ri.eu/event/praceisc-2021/

List of innovations by the CoEs, spotted by the EU innovation radar

idea-diego1100x350.jpg
The EU Innovation Radar aims to identify high-potential innovations and innovators. It is an important source of actionable intelligence on innovations emerging from research and innovation projects funded through European Union programmes. 
 
These are the innovations from the HPC Centres of Excellence as spotted by the EU innovation radar:
 
bioexcel-logo.png
GROMACS, a versatile package to perform molecular dynamics
Title: GROMACS, a versatile package to perform molecular dynamics
Market maturity: Exploring
Project: BioExcel
Innovation Topic: Excellent Science
KUNGLIGA TEKNISKA HOEGSKOLAN – SWEDEN
HADDOCK, a versatile information-driven flexible docking approach for the modelling of biomolecular complexes
PMX, a versatile (bio-) molecular structure manipulation package
Cheese_logo.png
Faster Than Real Time (FTRT) environment for high-resolution simulations of earthquake generated tsunamis
Probabilistic Seismic Hazard Assessment (PSHA)
Urgent Computing services for the impact assessment in the immediate aftermath of an earthquake
Title: Urgent Computing services for the impact assessment in the immediate aftermath of an earthquake
Market maturity: Tech Ready
Market creation potential: High
Project: ChEESE
Innovation Topic: Excellent Science
EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH – SWITZERLAND
BULL SAS – FRANCE
compbiomed_long_logo.png
Alya, HemeLB, HemoCell, OpenBF, Palabos-Vertebroplasty simulator, Palabos – Flow Diverter Simulator, BAC, HTMD, Playmolecule, Virtual Assay, CT2S, Insigneo Bone Tissue Suit
ecam-300x238-1.png
E-CAM Training offer on effective use of HPC simulations in quantum chemistry
Improved Simulation Software Packages for Molecular Dynamics
Improved software modules for Meso– and multi–scale modelling
eocoe.png
Code auditing, optimization and performance assessment services for energy-oriented HPC simulations
esiways-logo_type_grey_left-copy-2.png
New coupled earth system model
Excellerat_Logo_ELR_V1_20180209-01-300x106-1.png
AMR capability in Alya code to enable advanced simulation services for engineering
Data Exchange & Workflow Portal: secure, fast and traceable online data transfer between data generators and HPC centers
FPGA Acceleration for Exascale Applications based on ALYA and AVBP codes
Use of GASPI to improve performance of CODA
GPU acceleration in TPLS to exploit the GPU-based architectures for Exascale
In-Situ Analysis of CFD Simulations
Title: In-Situ Analysis of CFD Simulations
Market maturity: Tech Ready
Market creation potential: High
Project: Excellerat
Innovation Topic: Excellent Science
KUNGLIGA TEKNISKA HOEGSKOLAN – SWEDEN
FRAUNHOFER GESELLSCHAFT ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG E.V. – GERMAN
Interactive in situ visualization in VR
Title: Interactive in situ visualization in VR
Market maturity: Tech Ready
Market creation potential: High
Project: Excellerat
Innovation Topic: Excellent Science
UNIVERSITY OF STUTTGART – GERMANY
Machine Learning Methods for Computational Fluid Dynamics (CFD) Data
Title: Machine Learning Methods for Computational Fluid Dynamics (CFD) Data
Market maturity: Tech Ready
Market creation potential: Noteworthy
Project: Excellerat
Innovation Topic: Excellent Science
KUNGLIGA TEKNISKA HOEGSKOLAN – SWEDEN
FRAUNHOFER GESELLSCHAFT ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG E.V. – GERMAN
Parallel I/O in TPLS using PETSc library
MaX-logo-without-subline.png
Highly scalable Material Science Simulation Codes
Quantum Simulation as a Service
Title: Quantum Simulation as a Service
Market maturity: Exploring
Market creation potential: Noteworthy
Project: MaX
Innovation Topic: Excellent Science
EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH – SWITZERLAND
CINECA CONSORZIO INTERUNIVERSITARIO – ITALY
Simulation Code Optimisation and Scaling
NOMAD_Logo_srgb_web_whigh-1.jpg
pop250.png
Customer-Specific Performance Analysis for Parallel Codes
Repository of powerful tools (codes, pattern/best-practice descriptions, experimental results) for the HPC community

Strong scaling performance for human scale blood-flow modelling

 A Use Case by

Short description

To get the best out of future exascale machines, current codes must be able to demonstrate good scaling performance on current machines. The HemeLB code has demonstrated such characteristics to full machine scale on SuperMUC-NG. However, this performance must be aligned with the ability to solve practical problems. We have developed and deployed a self-coupled version of HemeLB that allows us to simultaneously study flows in linked arterial and venous vasculatures in 3D. In this use case, we look towards the application of flow in an arteriovenous fistula. A fistula is created in patients with kidney failure to increase flow in a chosen vein in order to provide an access point for dialysis treatment.

Results & Achievements

In collaboration with POP CoE, we were able to demonstrate HemeLB’s capacity for strong scaling behaviour up to the full production partition of SuperMUC-NG (>300,000 cores) whilst using a non-trivial vascular domain. This highlighted several challenges of running simulations at scale and also identified avenues for us to improve the performance of the HemeLB code. We’ve also run self-coupled simulations on personalised 3D arteries and veins of the left forearm with and without an arteriovenous fistula being created. The initial flow from our modified model showed good agreement with that seen in a clinical study.

Objectives

The first objective of this use case was to demonstrate and assess the strong scaling performance of HemeLB to the largest core counts possible. This was to enable us to evaluate current performance and identify improvements for future exascale machines. The second main objective was to demonstrate the ability of HemeLB to utilise this performance to study flows on human-scale vasculatures. The combination of both aspects will be essential to enabling the creation of a virtual human able to simulate the specific physiology of an individual for diagnostic purposes and evaluation of potential treatments.

Technologies

Use Case Owner

Peter Coveney
University College London

Collaborating Institutions

CompBioMed
POP CoE (Julich Supercomputing Centre)

Video of the Week: POPCast #4: Why Does Code Matter?

10. February 2021
pop png
In this POPCast, Jonathan Boyle, an HPC Application Analyst for POP, chats to regular host Fouzhan Hosseini about why code matters, its value to the customer and how POP works with customers to increase that value.

POP Webinar: PyPOP - An Interactive Tool for Performance Assessment

13. December 2020

This live webinar gave an overview of the NAG PyPOP tool and how it can be used to streamline the process of application performance analysis. 

Phil Tooley demonstrated the PyPOP analysis workflow, showed how the PyPOP wizard can be used to quickly calculate the POP metrics from captured profiling data and generate a performance report.

>> Read the full post

EXCELLERAT, MaX and POP at the International CAE Conference 2020

4. December 2020

   

The three HPC centres of excellence EXCELLERAT, POP and MaX participated in the 36th edition of the International CAE conference 2020, that was held online from 30th November until 3rd December 2020.

Under the topic “At the epicentre of the digital transformation of industry”, high-performance computing is a key enabler for this digital transformation and it was presented at a dedicated collateral event on Wednesday, December 2nd at 14:00h CET. 

In this session, the technical director of EXCELLERAT Amgad Dessoky presented a session titled “EXCELLERAT: paving the way for the evolution towards Exascale”. The EXCELLERAT activity brings together European experts to establish a Centre of Excellence (CoE) in Engineering Applications on HPC with a broad service portfolio, paving the way for the evolution towards Exascale. The aim is to solve highly complex and costly engineering problems, and create enhanced technological solutions even at the development stage.

In the exhibition, MaX and EXCELLERAT had a joint virtual booth together to show their latest results. The virtual format made it possible to interact with both CoEs via video and chat. The booth was visible for three months after the event. 

POP CoE was also present at the event with a virtual booth to exhibit its latest research results. 

>> CAE Conference Website