POP and PerMed Centers of Excellence are Getting Cell-Level Simulations Ready for Exascale
Success story # Highlights:
- Industry sector: Computational biology
- Key codes used: PhysiCell
- Keywords:
- Memory management
- Cell-cell interaction simulation
- OpenMP
- Good practice
Organisations & Codes Involved:
Technical Challenge:
One of PerMedCoE’s use cases is the study of tumour evolution based on single-cell omics and imaging using PhysiCell. In order to simulate such large-scale problems that replicate real-world tumours, High Performance Computing (HPC) is essential. However, memory usage presents a challenge in HPC architectures and is one of the obstacles to optimizing simulations for running at Exascale.
With a high number of threads computing in parallel, performance can be degraded because of concurrent memory allocations and deallocations. We observe that as the number of threads goes up, runtime is not reduced as expected. This is not a memory-bound problem, but a memory management one.
Solution:
Following the POP methodology and using the BSC tools (Extrae, Paraver, and BasicAnalysis), we determine that an overloading of a specific implementation of C++ operators is causing a high number of concurrent memory allocations and deallocations, causing the memory management library to perform synchronization and “steal” cycles of CPU from the running application.
The proposal from POP is to avoid the overloading of operators that need allocation and deallocation of data structures. However, achieving this implies a major code change. To demonstrate the potential of the suggestion without doing the major code change, we suggest using an external library to improve the memory management: Jemalloc.
Jemalloc is “a general purpose malloc implementation that emphasizes fragmentation avoidance and scalable concurrency support” [1]. It can be integrated easily into any code by preloading the library, with LD_PRELOAD in Linux for example. After applying this solution to PhysiCell, we execute the same experiment and obtain a 1.45x speed up when using 48 OpenMP threads.
[1] cited from Jemalloc website. http://jemalloc.net
Business impact:
As the only transversal CoE, one of POP’s objectives is to advise and help the other HPC CoEs prepare their codes for the Exascale. This collaboration between POP and PerMed is a good example of the potential of these kinds of partnerships between CoEs.
POP provides the performance analysis expertise and tools, the experience of hundreds of codes analysed, and the best practices gathered from those analyses. PerMedCoE bring state of the art use cases and real-world problems to be solved. Together, they improve the performance and efficiency of cell-level agent-based simulation software: in this case PhysiCell.
One of the goals of PerMedCoE is the scaling-up of four different tools that address different types of simulations in personalized medicine and that were coded in different languages (C++, R, python, julia). These tools are being re-factored to scale up to Exascale. In this scaling-up, audits such as the ones POP can offer are essential to evaluate past developments and guide future ones.
Currently, only one of PerMed’s tools is able to use several nodes to run a single simulation, whereas the rest of the tools can only use all of the processors of a single node. Our scaling-up strategy is a heterogeneous one. We are implementing MPI on some tools, re-factoring others to other languages that can ease the use of HPC clusters such as Julia, or even targeting “many-task computing“ paradigms like in the case of model fitting.
The main motivation to have HPC versions of these simulation tools is to be able to simulate bigger, more complex cell-level agent-based models. Current models can obtain up to 10^6 cells, but it has been proven that most of the problems addressed in computational biology (cancerogenesis, cell lines growth, COVID-19 infection) usually target from 10^9 to 10^12 cells. In addition, most of these current simulations consider an over-simplified environment that is nowhere close to a real-life scenario. We aim to have complex multi-scale simulation frameworks that target these bigger, more complex simulations.
Benefits:
- Speedup of 1.45x on runtime with 48 threads in PhysiCell execution
- A good practice for High Performance C++ applications exported
- This behaviour in memory management systems will be detected more easily in future analysis
Excellerat Success Story: Bringing industrial end-users to Exascale computing: An industrial level combustion design tool on 128K cores
CoE involved:
Success story # Highlights:
- Exascale
- Industry sector: Aerospace
- Key codes used: AVBP
Organisations & Codes Involved:
CERFACS is a theoretical and applied research centre, specialised in modelling and numerical simulation. Through its facilities and expertise in High Performance Computing (HPC), CERFACS deals with major scientific and technical research problems of public and industrial interest.
GENCI (Grand Equipement National du Calcul Intensif) is the French High-Performance Computing (HPC) National Agency in charge of promoting and acquiring HPC, Artificial Intelligence (AI) capabilities, and massive data storage resources.
Cellule de Veille technologique GENCI (CVT) is an interest group focused on technology watch in High Performance Computing (HPC) pulling together French public research, CEA, INRIA among Others. It offers first time access to novel architectures and access to technical support towards preparing the codes for the near future of HPC.
AVBP is a parallel Computational Fluid Dynamics (CFD) code that solves the three-dimensional compressible Navier-Stokes on unstructured and hybrid grids. Its main area of application is the modelling of unsteady reacting flows in combustor configurations and turbomachines. The prediction of unsteady turbulent flows is based on the Large Eddy Simulation (LES) approach that has emerged as a prospective technique for problems associated with time dependent phenomena and coherent eddy structures.
CHALLENGE:
Some physical processes like soot formation are so CPU intensive and non deterministic that their predictive modelling is out of reach today, limiting our insights to ad hoc correlations, and preliminary assumptions. Moving these runs to Exascale level will allow simulation longer by orders of magnitudes, achieving the compulsory statistical convergence required for a design tool.
The complexity at the code level to unlock node level and system level performance is challenging and requires code porting, optimisation and algorithm refactoring on various architectures in the way to enable Exascale performance.
SOLUTION:
In order to prepare the AVBP code to architectures that were not available at the start of the EXCELLERAT project, CERFACS teamed up with the CVT from GENCI, Arm, AMD and the EPI project to port, benchmark and (when possible) optimise the AVBP code for Arm and AMD architectures. This collaboration ensures an early access to these new architectures and prime information to prepare our codes for the wide spread availability of systems equipped with these processors. The AVBP developers got access to the IRENE AMD system of PRACE at TGCC with support from AMD and Atos, which allowed to characterise the application on this architecture and create a predictive model on how the different features of the processor (frequency, bandwidth) could affect the performance of the code. They were also able to port the code to flavors of Arm processors singling out compiler dependency and identify performance bottlenecks to be investigated before large systems become available in Europe.
Business impact:
The AVBP code was ported and optimised for the TGCC/GENCI/PRACE system Joliot-CURIE ROME with excellent strong and weak scaling performance up to 128,000 cores and 32,000 cores respectively. These optimisations impacted directly four PRACE projects on this same system on the following call for proposals.
Beside AMD processors, the EPI project and GENCI’s CVT as well as EPCC (EXCELLERAT’s partner) provided access to Arm based clusters respectively based on Marvell ThunderX2 (Inti Prototype hosted and operated by CEA) and Hi1616 (early silicon from Huawei) architectures. This access provided important feedback on code portability using the Arm and gcc compilers, single processor and strong scaling performance up to 3072 cores.
Results from this collaboration have been included on the Arm community blog [1]. A white paper on this collaboration is underway with GENCI and CORIA CNRS.
Benefits for further research:
- Code ready for the wide spread access of the Rome Architecture.
- Strong and weak scaling measurements up to 128,000 cores
- Initial optimisations for Arm architectures
Excellerat Success Story: Enabling High Performance Computing for Industry through a Data Exchange & Workflow Portal
CoE involved:
Success story # Highlights:
- Keywords:
- Data Transfer
- Data Management
- Data Reduction
- Automatisation, Simplification
- Dynamic Load Balancing
- Dynamic Mode Decomposition
- Data Analytics
- combustor design
- Industry sector: Aeronautics
- Key codes used: Alya
Organisations & Codes Involved:
As an IT service provider, SSC-Services GmbH develops individual concepts for cooperation between companies and customer-oriented solutions for all aspects of digital transformation. Since 1998, the company, based in Böblingen (Germany), has been offering solutions for the technical connection and cooperation of large companies and their suppliers or development partners. The corporate roots lie in the management and exchange of data of all types and sizes.
RWTH Aachen University is the largest technical university of Germany. The Institute of Aerodynamics of RWTH Aachen University possesses extensive expertise in the simulation of turbulent flows, aeroacoustics and high-performance computing. For more than 18 years large-eddy simulations with an advanced analysis of the large scale simulation data are successfully performed for various applications.
Barcelona Supercomputing Center (BSC) is the national supercomputing centre in Spain. BSC specialises in High Performance Computing (HPC) and manages MareNostrum IV, one of the most powerful supercomputers in Europe. BSC is at the service of the international scientific community and of industry that requires HPC resources. The Computing Applications for Science and Engineering (CASE) Department from BSC is involved in this application providing the application case and the simulation code for this demonstrator.
The code used for this application is the high performance computational mechanics code Alya from BSC designed to solve complex coupled multi-physics / multi-scale / multi-domain problems from the engineering realm. Alya was specially designed for massively parallel supercomputers, and the parallelization embraces four levels of the computer hierarchy. 1) A substructuring technique with MPI as the message passing library is used for distributed memory supercomputers. 2) At the node level, both loop and task parallelisms are considered using OpenMP as an alternative to MPI. Dynamic load balance techniques have been introduced as well to better exploit computational resources at the node level. 3) At the CPU level, some kernels are also designed to enable vectorization. 4) Finally, accelerators like GPU are also exploited through OpenACC pragmas or with CUDA to further enhance the performance of the code on heterogeneous computers. Alya is one of the only two CFD codes of the Unified European Applications Benchmark Suite (UEBAS) as well as the Accelerator benchmark suite of PRACE.
Technical Challenge:
SSC is developing a secure data exchange and transfer platform as part of the EXCELLERAT project to facilitate the use of high-performance computing (HPC) for industry and to make data transfer more efficient. Today, organisations and smaller industry partners face various problems in dealing with HPC calculations, HPC in general, or even access to HPC resources. In many cases, the calculations are complex and the potential users do not have the necessary expertise to fully exploit HPC technologies without support. The developed data platform will be able to simplify or, at best, eliminate these obstacles.
The data roundtrip starts with a user at the Barcelona Supercomputing Center that wants to simulate the Alyafiles. Therefore, the user uploads the corresponding files through the data exchange and workflow platform and selects Hawk at HLRS as a simulation target. After the files have been simulated, RWTH Aachen starts the post-processing process at HLRS. In the end the user downloads the post processed data through the platform.
With the increasing availability of computational resources, high-resolution numerical simulations have become an indispensable tool in fundamental academic research as well as engineering product design. A key aspect of the engineering workflow is the reliable and efficient analysis of the rapidly growing high-fidelity flow field data. RWTH develops a modal decomposition toolkit to extract the energetically and dynamically important features, or modes, from the high-dimensional simulation data generated by the code Alya. These modes enable a physical interpretation of the flow in terms of spatio-temporal coherent structures, which are responsible for the bulk of energy and momentum transfer in the flow. Modal decomposition techniques can be used not only for diagnostic purposes, i.e. to extract dominant coherent structures, but also to create a reduced dynamic model with only a small number of degrees of freedom that approximates and anticipates the flow field. The modal decomposition will be executed on the same architecture as the main simulation. Besides providing better physical insights, this will reduce the amount of data that needs to be transferred back to the user.
Scientific Challenge:
Highly accurate, turbulence scale resolving simulations, i.e. large eddy simulations and direct numerical simulations, have become indispensable for scientific and industrial applications. Due to the multi-scale character of the flow field with locally mixed periodic and stochastic flow features, the identification of coherent flow phenomena leading to an excitation of, e.g., structural modes is not straightforward. A sophisticated approach to detect dynamic phenomena in the simulation data is a reduced-order analysis based on dynamic mode decomposition (DMD) or proper orthogonal decomposition (POD).
In this collaborative framework, DMD is used to study unsteady effects and flow dynamics of a swirl-stabilised combustor from high-fidelity large-eddy simulations. The burner consists of plenum, fuel injection, mixing tube and combustion chamber. Air is distributed from the plenum into a radial swirler and an axial jet through a hollow cone. Fuel is injected into a plenum inside the burner through two ports that contain 16 injection holes of 1.6 mm diameter located on the annular ring between the cone and the swirler inlets. The fuel injected from the small holes at high velocity is mixed with the swirled air and the axial jet along a mixing tube of 60 mm length with a diameter of D = 34 mm. At the end of the mixing tube, the mixture expands over a step change with a diameter ratio of 3.1 into a cylindrical combustion chamber. The burner operates at Reynolds number Re = 75,000 with pre-heated air at T_air = 453 K and hydrogen coming at T_H2 = 320 K. The numerical simulations have been conducted on a hybrid unstructured mesh including prisms, tetrahedrons and pyramids, and locally refined in the regions of interest with a total of 63 million cells.
SOLUTION:
The developed Data Exchange & Workflow Portal will be able to simplify or even eliminate these obstacles. First activities have already started. The new platform enables users to easily access the two HLRS clusters, Hawk and Vulcan, from any authorised device and to run their simulations remotely. The portal provides relevant HPC processes for the end users, such as uploading input decks, scheduling workflows, or running HPC jobs.
In order to be able to perform data analytics, i.e. modal decomposition, of the large amounts of data that arise from Exascale simulations, a modal decomposition toolkit has been developed. An efficient and scalable parallelisation concept based on MPI and LAPACK/ScaLAPACK has been used to perform modal decompositions in parallel on large data volumes. Since DMD and POD are data-driven decomposition techniques, the time resolved data has to be read for the whole time interval to be analysed. To handle the large amount of input and output, the software tool has been optimised to effectively read and write the time resolved snapshot data parallelly in time and space. Since different solution data formats are utilised by the computational fluid dynamics community, a flexible modular interface has been developed to easily add data formats of other simulation codes.
The flow field of the investigated combustor exhibits a self-excited flow oscillation known as a precessing vortex core (PVC) at a dimensionless Strouhal Number of Sr=0.55, which can lead to inefficient fuel consumption, high level of noise and eventually combustion hardware damage. To analyse the dynamics of the PVC, DMD is used to extract the large-scale coherent motion from the turbulent flow field characterised by a manifold of different spatial and temporal scales shown in Figure 2 (left). The instantaneous flow field of the turbulent flame is visualised by an iso-surface of the Q-criterion coloured by the absolute velocity. The DMD analysis is performed on the three-dimensional velocity and pressure field using 2000 snapshots. The resulting spectrum of the DMD, showing the amplitude of each mode as a function of the dimensionless frequency is given in Figure 2 (top). One dominant mode, marked by a red dot, at Sr=0.55 matching the dimensionless frequency of the PVC is clearly visible. The temporal reconstruction of the extracted DMD mode showing the extracted PVC vortex is illustrated in Figure 2 (right). It shows the iso-surface of the Q-criterion coloured by the radial velocity.
Scientific impact of this result:
The Data Exchange & Workflow Portal is a mature solution for providing seamless and secure access to high-performance computing resources by end users. The innovative thing about the solution is that it combines the know-how about secure data exchange with an HPC platform. This is fundamental because the combination of know-how provision and secure data exchange between HPC and SMEs is unique. Especially the web interface is very easy to use and many tasks are automated, which leads to a simplification of the whole HPC complex and there is an easier entry for HPC engineers.
In addition, the data reduction programming technology ensures a more intelligent, faster transfer of files. There will be a highly increased performance speed when transferring the same data sets over and over. If the file is already known by the system and there is no need to transfer it again. Only the changed parts need to be exchanged.
The developed data analytics, i.e. modal decomposition, toolkit provides an efficient and user-friendly way to analyse simulation data and extract energetically and dynamically important features, or modes, from complex, high-dimensional flows. To address a broad user community having different backgrounds and expertise in HPC applications, a server/client structure has been implemented allowing an efficient workflow. Using this structure, the actual modal decomposition is done on the server running in parallel on the HPC cluster which is connected via TCP with the graphical user interface (client) running on the local machine. To efficiently visualise the extracted modes and reconstructed flow fields without writing large amounts of data to disk, the modal decomposition server can be connected to a ParaView server/client configuration via Catalyst enabling in-situ visualisation.
Finally, this demonstrator shows an integrated HPC-based solution that can be used for burner design and optimisation using high-fidelity simulations and data analytics through an interactive workflow portal with an efficient data exchange and data transfer strategy.
Benefits for further research:
- Higher HPC customer retention due to less complex HPC environment
- Reduction of HPC complexity due to web frontend
- Shorter training phases for inexperienced users and reduced support effort for HPC centres
- Calculations can be started from anywhere with a secure connection
- Time and cost savings due to a high degree of automation that streamlines the process chain
- Efficient and user-friendly post-processing/ data analytics
HiDALGO success story: Assisting decision makers to solve Global Challenges with HPC applications – Covid-19 modelling
HIGHLIGHTED CENTRE OF EXCELLENCE
Organisations & Codes Involved:
The National Health Service (NHS) oversees offering public health services in United Kingdom. It is now dealing with the COVID-19 pandemic.
Brunel University London is a dynamic institution that plays a significant role in the higher education sector. It carries out applied research on different topics, such as software engineering, intelligent data analysis, human computer interaction, information systems, and systems biology.
CHALLENGE:
The current pandemic situation has increased the NHS need of supporting tools to detect, predict and even prevent the virus spread behaviour. Knowing in advance this information will support them to take the appropriate decisions while considering health and care capabilities. In addition, the advance warning of new pandemic waves (or when they may subside) can help health authorities to rescale the capacity for non-urgent care, and ensure the timely arrangement of surge intensive-care capacity.
SOLUTION:
To tackle these challenges HiDALGO developed a tool: FACS, the Flu and Coronavirus Simulator, which is an agent-based model that also incorporates SEIRDI (Susceptible-Exposed-Infectious-Recovered-Dead-Immunized) states for all agents.
FACS approximates viral spread on the individual building level, and incorporates geospatial data sources from OpenStreetMap. In this way COVID-19 spread is modelled at local level, providing estimations of the spread of infections and hospital arrivals, given a range of public health interventions. Lastly, FACS supports the modelling of vaccination policies, as well as the introduction of new viral strains and changes in vaccine efficacy.
SOCIETAL & ECONOMIC IMPACT:
Although it is not a market topic itself, global challenges have been gaining importance in the last years, especially those related to climate change but also others such as peace and conflict. HiDALGO offers a set of tools, services, datasets and resources to define models that may predict situations under certain scenarios that can influence any decision to be taken.
More specifically, this tool FACS provides the needed information to decision makers so they can set up the appropriate measure and provide the necessary means at any time.
Indeed the tool helps the NHS to identify peaks of contagion in order to avoid sanitary collapses. Taking the appropriate decisions at the right moment represents a better investment of public resources and, what is more important, saving lives. Moreover, it supports to make better decisions and at appropriate time, to limit the problematic economic consequences of lockdowns and of the other measures taken in pandemic times.
BENEFITS FOR FURTHER RESEARCH:
- Support for the preparatory efforts by the health service for the second and third waves of the pandemic in West London.
- Better understanding of the nature of the current situation and the effect of different measures, such as lockdowns and vaccine efficacy levels.
- Provide models and elements about the foreseen evolution to limit the problematic economic consequences of the lockdowns and the various limitations due to the pandemic.
HiDALGO success story: Assisting decision makers to solve Global Challenges with HPC applications – Migration issues
HIGHLIGHTED CENTRE OF EXCELLENCE
Organisations & Codes Involved:
Save the Children is an NGO promoting policy changes to gain more rights for young people, especially by enforcing the UN Declaration of the Rights of the Child.
Brunel University London is a dynamic institution that plays a significant role in the higher education sector. It carries out applied research on different topics, such as software engineering, intelligent data analysis, human computer interaction, information systems, and systems biology.
CHALLENGE:
At the same time of the current COVID-19 pandemic, other crises have not stopped like forced migration due to conflicts. In fact, the number of forcibly displaced people is still very high, with over 70 million persons being forced to leave their homes. Save the Children provides support in these countries and needs more accurate estimations on people flows and even destinations to send the appropriate amount of help to the right place.
SOLUTION:
To tackle these challenges HiDALGO provides the tool Flee 2.0, a forced migration model which includes refugees and internally displaced people. It places agents that represent displaced persons in areas of conflict and uses movement rules to mimic their behavior as they attempt to find safety.
The code extracts location and route data from OpenStreetMap, and can be validated against UNHCR data for historical conflicts. As output, Flee provides a forecast of the amount (and location) of people that can be displaced given different conflict development scenarios.
SOCIETAL & ECONOMIC IMPACT:
Although it is not a market topic itself, global challenges have been gaining importance in the last years, especially those related to climate change but also others such as peace and conflict. HiDALGO offers a set of tools, services, datasets and resources to define models that may predict situations under certain scenarios that can influence any decision to be taken.
More specifically, Flee 2.0 provides the needed information to decision makers so they can set up the appropriate measure and provide the necessary means at any time.
For instance Flee 2.0 is used to estimate the expected number of refugee arrivals when conflicts occur in North Ethiopia, and to investigate how different conflict developments could affect the number of arrivals. The model development is partially guided by on-site experts and typically provides forecasts of approximately 3 month duration. Although work is ongoing on this project and the models are still basic, the aim is to establish a systematically developed mathematical model to improve the understanding of Save the Children about the migration situation, and support the preparation to mitigate the humanitarian impact of a potential upcoming crisis.
BENEFITS FOR FURTHER RESEARCH:
- Accurate predictions of where people may arrive in Sudan and how quicky, if new violence occurs in North
- Estimate of the effect of different conflict developments on the expected number of arrivals.
- Estimate of the effect of specific policy decisions on the expected number of arrivals.
Success Story: AiiDA Platform Accelerates Materials Discovery
Highlighted Centre of Excellence
MAX (MAterials design at the eXascale) is a European Centre of Excellence which enables materials modelling, simulations, discovery and design at the frontiers of the current and future High Performance Computing (HPC), High Throughput Computing (HTC) and data analytics technologies. >> Learn more about MAX
Quick Summary
Organisations Involved:
NCCR MARVEL is a center on Computational Design and Discovery of Novel Materials created by the Swiss National Science Foundation in May 2014. MARVEL targets the accelerated design and discovery of novel materials, via a materials’ informatics platform of database-driven high-throughput quantum simulations, powered by:
- advanced electronic-structure capabilities, for predictive accuracy
- innovative sampling methods to explore configuration/composition space
- application of big-data concepts to computational materials science.
Bosch Research and Technology Center: Founded in 1999, the North American division of Corporate Research at Bosch has been shaping the technology of Bosch’s future for nearly 20 years. The team has worked in close collaboration with its colleagues and counterparts in Germany and around the world. The center is committed to providing technologies and systems for the four business sectors of Bosch — Mobility Solutions, Energy and Building Technology, Industrial Technology and Consumer Goods – by scouting and collaborating with top universities and industry partners in North America.
CHALLENGE:
Solid-state electrolytes have the potential to enhance both safety and performance of Li-ion batteries, allowing for novel cathode and anode chemistry, preventing the growth of Li–metal dendrites — the needle-like formations of lithium that grow inside batteries, causing devices to lose power more quickly, short out, or sometime even catch fire — and pushing the miniaturization of battery cells.
Despite intense research in this field for decades though, no known solid-state ionic conductor satisfies all the requirements needed for battery applications. This makes the search for new materials a worthwhile endeavor. Computational approaches in the search for new materials are less human-intensive and easily parallelizable and precede synthesis and characterization in the laboratory. Computational screening relies on simulations of the electronic structure, to determine the insulating character of a material, and molecular dynamics simulations to predict the Li-ion diffusion coefficients.
Overall, thousands of calculations need to be performed, making automatization and reproducibility a key requirement. In addition, methods need to be computationally inexpensive enough to be run on thousands of materials, yet accurate enough to be predictive.
SOLUTION:
We first reduced the computational burden of modelling the potential energy surface of lithium diffusing in a solid-state ionic conductor to develop a workable framework. We then demonstrated a procedure for running these extensive molecular dynamics simulations in a largescale computational screening. AiiDA made this possible by allowing the automation and explicit storage of the provenance. The novelty of AiiDA in the field of materials informatics is that every calculation is stored as a node in a graph, with input data forming incoming nodes, and output data stored as outcoming nodes, that can again be input to a different calculation.
In addition, AiiDA allows for a high degree of automation and parallelization via its daemon. Every calculation presented in the paper “High-throughput computational screening for solid-state Li-ion conductors” was run with AiiDA.
Business impact:
We found five materials with fast ionic diffusion, some in the range of the well-known superionic conductor Li10GeP2S12, such as for example the Li-oxide chloride Li5Cl3O, the doped halides Li2CsI3, LiGaI4, and LiGaBr3, or the Li-tantalate Li7TaO6. In addition, we found 40 materials that show significant diffusion at 1000 K, though they will need to be investigated more thoroughly before their suitability can be determined. All of these potential fast-ionic conductors could be studied further, in more detail, by experiment and simulation, and could result in new fast-ionic conductors or even electrolytes for next generation solid-state Li-ion batteries. Our data could also serve to search for descriptors of fast ionic conduction, which would be of significant interest to the community.
This work benefits society by identifying inorganic solid-state lithium-ionic conductor compounds that could be used as electrolytes to mitigate or overcome the severe safety challenges posed by the use of volatile and flammable liquid or polymer electrolytes in today’s Li-ion batteries. Complete replacement of the liquid electrolyte by a solid ceramic would result in an all-solid-state Li-ion battery, highly beneficial due to the higher electrochemical stability of inorganic electrolytes, compared to their organic counterparts.
Benefits for further research:
- We developed efficient ways of simulating the diffusion of lithium in the solid state and gained physical insight into how charge-density rearrangements or lattice vibrations affect it.
- We developed a framework for predicting the diffusion of Li ions in solid-state materials and a process for applying it in largescale computational screening.
- We identified new ceramic compounds for in-depth experimental investigation
Related Images :
ESiWACE success story: Improving weather and climate forecasting with a new NEMO configuration
Highlighted Centre of Excellence
ESiWACE, the “Centre of Excellence in Simulation of Weather and Climate in Europe” has been funded by the European Commission to substantially improve efficiency and productivity of numerical weather and climate simulation. Overall, the Centre of Excellence prepares the European weather and climate community to make use of future exascale systems in a co-design effort involving modelling groups, computer scientists and HPC industry. .
Organisations & Codes Involved:
Atos Center for Excellence in Performance Programming (CEPP) provides expertise and services in High Performance Computing, Artificial Intelligence and Quantum Computing.
LOCEAN laboratory, part of the CNRS-IPSL, conducts studies on the physical and biogeochemical processes of the ocean and their role in climate in interaction with marine ecosystems.
CERFACS research center is specialized in modelling and numerical simulation, through its facilities and expertise in high-performance computing.
CHALLENGE:
A key model for weather and climate comprehension is NEMO (Nucleus for European Modelling of the Ocean), a modelling framework for research activities and forecasting services in ocean and climate sciences. NEMO is used by a wide variety of applications at global or regional focus, with different resolutions, different numerical schemes, parameterizations and therefore with different performance constraints. The technical challenge here was to find a way to ease the profiling and benchmarking of NEMO for its versatile uses in order to increase the performance of this framework.
SOLUTION:
In response to this challenge, ESiWACE has developed a new configuration, adapted to HPC benchmarking, that is polymorphic and can very simply reproduce any use of NEMO.
Thanks to a close collaboration between the Atos CEPP, LOCEAN and CERFACS, a dedicated configuration of NEMO to ease its profiling and benchmarking has been set up.
Many tests of this configuration, including large-scale experiments on the Atos Bull supercomputers at Météo France and CEA’s Very Large Computing Centre (TGCC) have been performed. This resulted in several optimisations improving the performance of the NEMO code in this configuration by up to 38%.
The open source NEMO 4.0, which was released in 2019, benefited from this work and included the following improvements: an automatic MPI sub-domain decomposition, and a rewriting of the communication routines with an optimisation of treatment of the North pole singularity.
Business impact:
Some of the NEMO uses such as weather or climate forecasts are among the key challenges our society must address. Improvements of NEMO numerical performance allow to refine model results, to reduce forecast uncertainties and to better predict high-impact extreme events, thus saving lives. On Earth the Ocean has a huge impact on the atmosphere. Thus, NEMO is widely used coupled with atmospheric models. For example, it is used to simulate ocean eddies, temperature and salinity that play a key role in cyclone intensity and trajectory forecast. Therefore, business impacts of improving NEMO may be indirect but there are significant as they concern everyone and all kinds of companies and entities.
This work benefits society by improving the efficiency and productivity of numerical weather and climate simulation and by preparing them for future exascale systems. It fosters expertise exchanges which enable researchers and industry to be more productive, leading to scientific excellence in Europe, through direct and tight collaborations.
Benefits for further research:
- Simplified profiling and benchmarking of NEMO at different scales to find the most relevant optimisation paths.
- Up to 38% efficiency and scalability increase of NEMO with the optimized configuration on HPC systems.
- Reduction in time/cost of ocean simulations for both research and production purpose improving weather and climate predictions, allowing to protect property, interests and saving lives.
Success Story: Designing control pulses for superconducting Qubit systems with local control theory
CoE involved:
The European HPC Centre of Excellence (E-CAM) is an e-infrastructure for software development, training, and industrial discussion in simulation and modelling that started in October 2015. E-CAM focuses on four scientific areas of interest to computational scientists: Classical Molecular Dynamics, Electronic Structure, Quantum Dynamics, Meso- and MultiScale Modelling
Organisations & Codes Involved:
CECAM Centre Européen de Calcul Atomique et Moléculaire (Host beneficiary), located at the EPFL in Lausanne, is an organization devoted to the promotion of fundamental research on advanced computational methods and to their application to important problems in frontier areas of science and technology.
IBM Research Laboratory – Zurich (Industrial partner) is the European branch of IBM research, which is the research and development division of the American multinational information technology company IBM.
CHALLENGE:
The aim of this pilot project was to develop a new method and dedicated software for designing control pulses to manipulate qubit systems (see Fig.1A) based on the local control theory (LCT) The system is composed of two fixed frequency superconducting transmon qubits (Q1 and Q2) coupled to a tunable qubit (TQ) whose frequency is controlled by an external magnetic field. Changing the frequency, the TQ behaves as a targeted quantum logic gate, effectively enabling an operation on the qubit states. The system schematizes an approach to construct real quantum universal gates currently investigated by IBM.
SOLUTION:
Local control theory (LCT), the main theoretical tool used, originates from physical chemistry where it is used to steer chemical reactions towards predetermined products, but it had never been used to design a quantum gate. To create the software, researchers added new functionalities to the open source QuTip program package. Two main modules were developed during the project: LocConQubit, which implements the LCT and accompanying procedures, and OpenQubit, a patch to the first module which introduces Lindblad master equation propagation scheme into the LCT which also enables direct construction of pulses under the presence of decoherence effects. All modules were written in Python and expand the functionalities of the QuTip program package.
Business impact:
In order to understand the aerodynamic behavior of the devise, experimental, or numerical studies can be performed. For the real scale model, that is approximately three meters high, experimental studies would have been too expensive. It was therefore decided to resort to high accuracy numerical simulations of the flow coupled to the oscillating turbine. Since the flow is highly transient, and it is important to capture the dynamically important scales of the flow accurately, Large Eddy Simulation (LES) was selected as the modeling technique.
Modeling of turbulent flow is still nowadays one of the most computational demanding problems. The interaction with the Barcelona Supercomputing Center was, therefore, crucial to develop a much better understanding of the aerodynamics of the device. BSC provided the computation resources and the inhouse code Alya that can efficiently run on high-end Supercomputers. Moreover, they contributed with their know-how on LES turbulence modeling and Fluid-Structure Interaction.
Technological advantages the HPC provides are important for small to medium-sized enterprises (SME) to remain competitive. For instance, the savings regarding the CPU time are key for SMEs to be able to use highly advanced techniques such as Large Eddy Simulation.
Benefits for further research:
- Reduced costs by preparing the real scale experiments by means of numerical simulations
- The highly optimized code enabled to avoid throwing away costly computational resources
- The cpu time for assembly has been reduced up to 38%.
- A new solver has provided speed ups of up to five times with respect to Alya’s own solvers.
Success Story EoCoE: Renewable Energy - harness the power of vorticity
CoE involved:
EoCoE is an energy-oriented Centre of Excellence for computing applications that builds on its unique expertise at the crossroads of high-performance computing (HPC) and renewable energy. It brings an impulse to accelerate the digitization of the future energy system. The coding developments are assisted by multi-disciplinary teams with expertise in applied mathematics and high performance computing (HPC).
Organisations & Codes Involved:
The end user VORTEX BLADELESS is a Spanish tech startup that is developing an environmentally friendly aerogenerator which needs no blades. It is a new wind energy technology specially designed for on-site generation in residential areas, being able to work on-grid, off- grid, or along with regular solar panels or other generators.
The domain expert, the BARCELONA SUPERCOMPUTING CENTRE is a Spanish public research center that developed the multi-physics simulation code ALYA and provided the MareNostrum supercomputer.
CHALLENGE:
Vortex Bladeless aims to harness the power of vorticity for a new generation of wind turbines. They develop a single column without bearings or gears. It just oscillates with the wind. Experiments with scaled down prototypes have been encouraging, but the physics behind these devices is highly complex. There is a need to optimize and explore scalability due to the complexity of the flow and the need for time accurate results. The needed Large Eddy Simulation (LES) simulations are computationally demanding.
SOLUTION:
The company has been working with experts at the Barcelona Supercomputing Centre on the MareNostrum supercomputer. The fluid-structure interaction (FSI) between the Vortex Bladeless device and a turbulent flow is simulated with Alya.
The results from initial simulations of a scaled-down device were very close to the actual wind tunnel tests performed by the Vortex Bladeless team, allowing them to develop the idea of a range of devices at the micro scale and the utility scale. Then, the behaviour of the device at a more realistic scale was studied by means of numerical simulations, helping in the design of real scale experiments and reducing costs.
Business impact:
In order to understand the aerodynamic behaviour of the devise, experimental, or numerical studies can be
performed. For the real scale model, that is around three meters high, experimental studies would have been too expensive.
It was therefore decided to resort to high accuracy numerical simulations of the flow coupled to the oscillating turbine. Since the flow is highly transient, and it is important to capture the dynamically important scales of the flow accurately, Large Eddy Simulation (LES) was selected as the modeling technique.
Modeling of turbulent flow is still nowadays one of the most computational demanding problems. The interaction with the Barcelona Supercomputing Centre was, therefore, crucial to develop a much better understanding of the aerodynamics of the device. BSC provided the computation resources and the inhouse code Alya that can efficiently run on high-end Supercomputers.
Moreover, they contributed with their know-how on LES turbulence modeling and Fluid-Structure Interaction. Technological advantages the HPC provides are important for small to medium-sized enterprises (SME) to remain competitive. For instance, the savings regarding the CPU time are key for SMEs to be able to use highly advanced techniques such as Large Eddy Simulation.
Benefits for further research:
- Reduced costs by preparing the real scale experiments by means of numerical simulations
- The highly optimized code enabled to avoid throwing away costly computational resources
- The cpu time for assembly has been reduced up to 38%.
- A new solver has provided speed ups of up to five times with respect to Alya’s own solvers.
Success Story CompBioMed: Drug Discovery with Janssen Pharmaceutica NV
CoE involved:
CompBioMed is a user-driven Centre of Excellence in Computational Biomedicine. They have users within academia, industry and clinical environments and are working to train more people in the use of their products and methods.
Organisations & Codes Involved:
SURFsara, the domain expert, is the National Supercomputing and e-Science Support Center in the Netherlands. SURFsara provides expertise and services in the areas of High Performance Computing, e-Science & Cloud Services, Data Services, Network support, and Visualisation.
Janssen Pharmaceutica NV, the industrial partner, is an affiliate of the Pharmaceutical branch of the US Johnson & Johnson company. Janssen’s interests are in developing and using advanced molecular simulation methods to optimize lead compounds in discovery programs, predicting the activity of compounds with specific targets.
CHALLENGE:
Janssen’s primary challenge is in developing and using advanced molecular simulation methods to optimize lead compounds in discovery programs. Such methods, if proven robust and accurate could have a profound impact on the way drug discovery is performed. They would permit reliable computational triaging of very close analogue molecules greatly improving efficiency. Also, this would lead to high-confidence design of synthetically more challenging molecules leading to better drugs in new chemical space.
SOLUTION:
Through the CompBioMed HPC allocations service JAN obtained an allocation of 1.8M core/hours on Cartesius (SURFsara). JAN evaluated the use of non-commercial software for predicting the free energy of a series of compounds and built a protocol which uses open-source software for MD and free energy perturbation simulations. The main simulation code used was GROMACS (version 2016.1) configured to run on GPU, and the results compared to previous simulations performed with Schrodinger’s FEP+ commercial software. Slurm support for job array and job dependencies, has been used to orchestrate and streamline the workflow and to optimise the MD simulations execution on multiple compute nodes.
Scientific impact of this result:
New computational approaches described above can have a major impact on the drug discovery process. Importantly the work studied here will go to the heart of the design-make-test cycle and contribute higher quality methods for compound prioritization.
This is a fundamental issue of drug discovery, because, whilst compounds with acceptable potency can often be found, and found quickly, they do not always come with other desired properties. Hence, during a typical lead optimization (LO) program the challenge often becomes to maintain potency whilst modifying the chemical structure of the lead molecules to overcome these other issues. In this regard computational tools which can accurately predict binding mode combined with accurate binding affinity prediction will be extremely powerful and reduce the number of ‘backwards’ steps required to subsequently move forwards in an LO program.
The project has introduced Janssen to numerous research groups throughout Europe that are able to assist them in their aims and to combine expertise to produce useful results. The use of HPC is critical for these experiments, without which the compute power does not satisfy the needs of such complex systems.
Benefits for further research:
- Higher quality methods for compound prioritization.
- Increased efficiency during lead optimisation program.
- Reduction in time/cost of synthesising redundant drug molecules
- Higher probability of determining new active drug
