• CMake (≥3.0)

• MPI (Message Passing Interface) Windows:

• Microsoft Visual Studio Community (≥2015)

• Microsoft C++compiler, installed with Visual Studio Community

• CMake (≥3.0)

• Microsoft MPI

The GPU branch of the npFEM solver supports NVIDIA GPUs with compute capability≥6.0 (extensively tested and validated). However, the code could support GPUs with compute capa-bility<6 (tested but not fully validated), by replacing the atomic operations (e.g.atomicAdd) with ones based onatomicCAS(Compare and Swap). For more information, one should consult the CUDA toolkit documentation. In this case, the user should modify as well the CMake file to target the correct GPU architecture (currently set to-arch=sm_60).

The periodic boundary conditions introduce a dependency on parallelisation. In more de-tails, the directions with periodicity should be subdivided by at least two sub-domains (each sub-domain belongs to a different MPI task). This is due to how the area per surface vertex is computed, i.e. if there is no domain subdivision, then the crossing bodies (from the outlet to inlet) are considered as stretched (erroneous deformation, leading to code crash). On the con-trary, when the outlet and inlet belong to different sub-domains (MPI-wise), then the crossing bodies are duplicated in memory and thus they are not considered as stretched/deformed.

This means that Palabos-npFEM depends strictly on MPI, which is not a hard constraint given the computational intensity of blood flow simulations. Of course, this dependency could be eliminated by modifying/extending the library (area computation part).

**Software location**

**Code repository**
**Name:** Palabos

**Persistent identifier:** https://gitlab.com/unigespc/palabos.git
**Licence:** Palabos→AGPL v3 & npFEM→MPL v2

**Date published:** 03/07/2020 (v2.2.0)

**6.8 Reuse potential**

The Palabos-npFEM library gives special attention to modularity and low complexity. In more details, the software is designed based on a plug-and-play approach, where it is up to the user’s preference to choose the individual solvers for the resolution of the various phases of blood. The Computational Biomedicine community is a vibrant and dynamic community,

with numerous research contributions in various directions, thus we expect other researchers
to possibly plug their own solvers in our platform and experiment with it. Starting point for
extending and reusing Palabos-npFEM is the principal application located inexamples_{→}
showCases_{→}bloodFlowDefoBodies(utilised for the example applications). The end user
can either deploy the library as is by executing this provided application, or build on top of it
further functionalities/alterations.

Currently, the principal application that we provide treats simple geometries, i.e. box and tubular flows. However, its extension to more complicated geometries is well supported both by Palabos and npFEM. We consider that promising application areas for our software are provided by microfluidic devices and lob-on-a-chip systems, for which a large interest can be observed in the community.

Our library uses the CMake^{§} tool for building and compiling. CMake is an open-source
and cross-platform tool, which allows the libraries using it to be compiled in any supported
platform. Thus, the users can deploy Palabos-npFEM in cross-platform environments (from
personal computers/workstations to supercomputers) and speedup their development &

research.

Cellular blood flow simulations are extremely computationally expensive. For example [75],
to simulate a box of dimensions 50^{3}*µm*^{3}under a shear flow at 35% hematocrit, for physical
time of 1*s, we need about 5 days in a high-end supercomputer (using 5 compute nodes, i.e.*

12 cores and 1 GPU per node). However, an allocation of 5 consecutive days is rarely available in supercomputing centres. For this reason, we have developed an efficient check-pointing system, which allows the user to pause at any time the simulation, and restart seamlessly from where it previously stopped. This feature offers an attractive advantage for other researchers to actively use our library.

The library is specialised on cellular blood flow simulations, but its methodology could easily be applied to the simulation of other complex suspensions, and fluid-structure/solid interac-tion applicainterac-tions in general. A recent example is the simulainterac-tion of Paragliders [95], where the researchers used Palabos and a structural solver similar to npFEM. Thus we strongly believe that our library could be used as a building component for other research topics.

The Palabos library has a large and active community. Integrating npFEM into Palabos serves
the purpose of sharing and exposing all the details with this global and dynamic group of
researchers and engineers. The users can find support in the Palabos forum^{*}, and thus our
library benefits from the same high-quality support mechanism that is already in-place for
Palabos.

§https://cmake.org/

*https://palabos-forum.unige.ch/

**6.8. Reuse potential**

**Funding**

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 823712 (CompBioMed2 project), and by the Swiss PASC project “Virtual Physiological Blood: an HPC framework for blood flow simulations in vasculature and in medical devices”.

**Acknowledgements**

We acknowledge support from the Swiss National Supercomputing Centre (CSCS, Piz-Daint supercomputer), the National Supercomputing Centre in the Netherlands (Surfsara, Cartesius supercomputer), and the HPC Facilities of the University of Geneva (Baobab cluster).

**7** **Conclusion**

In this thesis, we proposed and built models for the simulation of human blood in both the micrometre scale (Palabos-npFEM) and the macroscopic level (stochastic models based on fat tailed distributions). Our models omit for simplicity and computational efficiency biochemical processes (e.g. adhesion-aggregation of activated PLTs on blood vessels) and consider only mechanically driven processes. We carefully validated and verified the consistency of our models and compared their output with various in vitro experimental studies. Our work contributed towards the following directions:

• We proved that high-fidelity FEM solvers (such as the npFEM solver) can be a viable solution for the simulation of large systems of deformable bodies in comparison to the more simplified mass-spring-systems.

• We built a highly-scalable HPC-centric computational framework for the direct numeri-cal simulation of blood. As the current supercomputers become “fatter” in both the CPU

& GPU sides, we built a framework that can efficiently utilise the available infrastructure in an efficient, flexible and modular way.

• We used our numerical models along with in vitro experiments and proved their excel-lent complementarity for in-depth investigations of highly complex dynamical systems.

We would like to highlight that the in vitro experiments, used for this thesis, were specifically designed along their in silico counterparts. This has led to unique in-depth investigations of blood rheology and a proof-of-concept on the future of digital labs.

• We proposed a disruptive view, by showing that the random part of PLT velocities is governed by a fat-tailed probability distribution, usually referred to as a Lévy flight. This observation limits the applicability of the widely used advection-diffusion equation for the description of PLTs transport physics. This has great implications on understanding the complex platelet transport and on designing high-fidelity and clinically relevant tools, such as PLT function tests (e.g. impact-R device).

• We open-sourced all our software tools (Palabos-npFEM, statistical analysis of PLT

velocities & random walks) contributing to an open discussion in our field. Our tools can be found in the following links:

**–** Palabos-npFEM: https://gitlab.com/unigespc/palabos

**–** Statistical Analysis & Fat-tailed distributions of PLT velocities: https://github.com/

kotsaloscv/PLTs-FatTails

**Ongoing & Future work**

The main directions that we are actively exploring are the following:

• The fully resolved blood flow simulations that we performed in this thesis considered
just one shape of RBCs per simulation. However, reality departs from this simplified
view and RBCs have substantially different shapes compared to the standard one. In
chapter 4, we used equation (4.1) to generate healthy RBC shapes, with the standard
shape characterised by (R, c_{0}, c_{1}, c_{2}) = (3.91*µ*m, 0.1035805, 1.001279, -0.561381). Our
collaborators from the Laboratory of Experimental Medicine in CHU de Charleroi (Dr.

Jérôme Dohet-Eraly & Professor Karim Zouaoui Boudjeltia) have conducted a unique and extended study (in vitro) on healthy donors to define the different shapes of RBCs (probability distribution function of RBC shapes). In Figure 7.1, we present the resulted sampling of RBC cross-sections. Our computational framework (Palabos-npFEM) sup-ports naturally this extension, i.e. every RBC in the simulation to have a different shape.

Therefore, we are interested on (publication in preparation) validating the PLT velocity distribution under this new setup, and overall to observe possible differences on cell transport properties in comparison to the simplified approach that we followed so far (due to the absence of data).

• Regarding the macroscopic/stochastic models, their computational efficiency for the simulation of clinically relevant applications constitutes the main advantage over the ex-tremely computationally intensive fully resolved simulations. Nevertheless, the stochas-tic models can readily give misleading results if the wrong underlying distributions are used. In more details, Figure 7.2 shows the stochastic model of the impact-R platelet function analyser [29]. This model disregards the fully resolved motion of blood cells and instead deploys suitable probability distributions for the PLT velocities. By using Brownian-like random walks (matching the Zydney-Colton [162] diffusivity), we showed in chapter 5 that they heavily underestimate the in vitro experimental outcome. On the other hand, if we deploy fat tailed distributions (PLT movement), then the in silico experiment gives matching results with its in vitro counterpart. As a future work, we are planning on extensively deploying the fat-tailed models and study further the deposition processes as described in Dutta et al. [41] and Chopard et al. [29] (replacing the 1D diffusion equations with Lévy flights). Keep in mind that these studies [41, 29] were using an enhanced Brownian motion for the description of PLT transport. Of course,

the stochastic models can easily be applied to other experimental devices other than the impact-R, such as lab-on-a-chip systems.

• In chapter 5, we showed the anomalous PLT transport by carefully analysing 64
numeri-cal experiments. Since we focused on reproducing the in vitro experiments performed
by Chopard et al. [29], we fixed the hematocrit at 35%, the shear rate at 100 s^{-1}, and
experimented numerically with the domain size. Nevertheless, we strongly believe that
there should be a threshold for hematocrit and shear rate, where the Brownian-like
motion of platelets is restored (this is why we are mentioning the limited applicability
of the advection-diffusion equation). Therefore, we are interested on charting these
thresholds and further clarify where the Brownian hypothesis breaks down and the Lévy
flight takes over.

• The numerical tools presented in this thesis consider only mechanical processes, and omit any biochemical process. For example, during the activation of platelets, there is a series of chemical reactions that take place, with huge importance on clot formation and other physiological processes, e.g. adhesion/aggregation. For now, any biochemical action is implicitly taken into account through mechanical processes, e.g. repulsive forces through increased collision potential energies. Ultimate goal would be to couple our computational framework with biochemical processes and study the evolution of this kind of dynamical systems.

• In this thesis, we performed numerical simulations in very simple geometric setups (Couette and Poiseuille flow), as dictated by the impact-R device. Nevertheless, the cardiovascular system heavily departs from this simplified view. For this reason, we are planning on extending Palabos-npFEM to simulate more complicated geometries.

• In simulations of multiple blood cells, the interaction forces between them constitute a very critical element for physically correct results. These forces help us avoid inter-penetrations but also take into account the possible surface charge of the blood cells.

Throughout this thesis, we modelled these interactions as springs (Hooke’s law) between bodies that are closer than a specified threshold. In the literature, there are various techniques to simulate this interaction, but none of them (including ours) have a rigor-ous explanation. Our plan is to study further how the varirigor-ous blood cells interact with each other, through specially designed in vitro experiments. These experiments are expected to investigate pair collisions between isolated red blood cells and platelets (activated/non-activated). Following, we plan to use the in vitro measurements for the calibration of our spring models and potentially try alternative modelling methods (e.g.

Morse potential [25]).

• Based on the limited prognostic capacity in clinical practice of the existing platelet
func-tion tests (as discussed in chapter 5), our team^{*}is planning on designing a new platelet

*Laboratory of Experimental Medicine-CHU Charleroi & Scientific and Parallel Computing Group-University of Geneva

function analyser. This new device aims at integrating all the knowledge acquired on the transport physics of platelets and help us shed more light on the idiosyncratic behaviour of PLTs.

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50

**Half-Thickness [μm]**

**Radius [μm]**

**RBCs cross-section - Part I **

Average Shape Median Shape Standard Shape

**(a)**

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50

**Half-Thickness [μm]**

**Radius [μm]**

**RBCs cross-section - Part II **

Average Shape Median Shape Standard Shape

**(b)**

**Figure 7.1 –**RBC cross-sections as extracted from in vitro measurements from healthy donors.

Data published with the permission of Dr. Jérôme Dohet-Eraly & Professor Karim Zouaoui
Boudjeltia. The use of two graphs (Part I & II) is for visualisation purposes, since fitting all data
in one graph results in a cluttered graph. The standard shape (red line) refers to the commonly
used RBC shape with parameters (R, c0, c1, c2) = (3.91*µm, 0.1035805, 1.001279, -0.561381).*

**V**

_{top}Red Blood Cell Non-Activated Platelet

**y**
Albumin

**V**

_{bottom}**= 0**

Activated Platelet

**x**
**z**

**Figure 7.2 –**Window of Impact-R device: The bottom wall is a fixed boundary of dimensions
1×1*mm*^{2}, the wall-bounded direction is 0.82*mm. The bulk contains whole blood at 35%*

hematocrit. The discretization of the substrate is such that in every cell can fit just one platelet.

The initial densities of the blood particles are determined by the*in vitro*experiment [29] and
usually are about: 172 200 (*µl)*^{−}^{1}for NAP, 4808 (*µl)*^{−}^{1}for AP. The grey-scale inset next to the
discretised substrate represents the deposited PLTs as observed from the impact-R device.

**A** **Bridging the computational gap between mesoscopic** **and continuum modelling of red blood cells for fully** **resolved blood flow**

**A.1 Alternating solver**

The minimisation of the objective function (2.7) with strictly quadratic potential energies is
performed using an alternating (local/ global) solver. In the local step,* x*is fixed while the
projections

*per element are computed. In the global step,*

**p***is fixed and the minimum of*

**p***g*is found by equating its gradient to zero:

∇g(x* _{n+1}*)= 1

*h*^{2}**M(x**e * _{n+1}*−

**y***)+Lx*

_{n}*−*

_{n+1}**Jp**=0, (A.1) so

**x**^{∗}_{n}_{+1}=¡

**M/h**e ^{2}+**L**¢−1¡

**Jp**+**My**e * n*/h

^{2}¢

. (A.2)

The matrix**M/h**e ^{2}+Lis symmetric positive definite and thus**x**_{n}^{∗}_{+}_{1}is a global minimum for fixed
projections. The above scheme is guaranteed to converge monotonically to a minimum. The
local step decreases each constraint keeping fixed the vector* x. The global step, keepingp*fixed,
finds a compromise of the new state towards the projected positions and thus minimises the
objective function. Consequently, this sequence is non-increasing and bounded from below
(minimisation of Frobenius norms). The local/global steps are repeated until convergence is
satisfied (typically below 10 iterations).

This alternating solver is a*special case of the quasi-Newton method*presented in section 2.2.1.

Using the same Hessian approximation**H**e =**M/h**e ^{2}+L, it is revealed that the descent direction
* d*:

* d*= −¡

**M/h**e ^{2}+**L**¢−1

∇*g(x)*=**x**^{∗}−* x.* (A.3)

Thus, projective dynamics can be viewed as a quasi-Newton method that computes the next
iterate as* x*+

*. For normal quasi-Newton methods, we have to use line search techniques to find*

**d***α*such that

*+*

**x***α*reduces the objective function as much as possible [107]. The particularity of PD is that

**d***α*=1 [94] and thus there is no need for line search techniques. At this point, we must highlight that the choice of

**H**e is inspired by the local/global solver, as seen

from equation (A.2), and thus it is deeply rooted in projective dynamics.

**A.2 FEM reminder**

When an object deforms, every material point* X* (undeformed state) is displaced to a new
deformed location

*:R*

**x. The deformation function****φ**^{3}→R

^{3}maps every material point to its respective deformed location, such that

*=*

**x***). The Jacobian matrix of the deformation function*

**φ(X***∂φ*/

*∂*is an important physical quantity for describing elastic bodies and is called deformation gradient

**X****F. Let us focus on a triangular element isometrically embedded in**R

^{2}. The deformation function can be defined to be a

*piecewise linear*function ˆ

*(approximation of*

**φ***) over the element. In more details, for the triangular elements we have [127]*

**φ***φ*ˆ(X)=**AX**+* b,* (A.4)

where**A**&* b*are element specifics. Differentiating equation (A.4) with respect to

*, we con-clude that the deformation gradient*

**X****F**=

*∂φ*ˆ/

*∂*=

**X****A**is constant for the element. Let us denote with~

*X*

_{1},~

*X*

_{2},~

*X*

_{3}the undeformed vertex positions of an arbitrary triangular element, and let

~*x*1,~*x*2,~*x*3be the deformed counterparts. Each vertex must satisfy the deformation function

a more compact form reveals that

h~*x*1−~*x*3 ~*x*2−~*x*3

Consequently, the deformation gradient**F**∈R^{2×2}of an arbitrary triangular element embedded
inR^{2}is

**F**=**D***s***D**^{−1}* _{m}*. (A.8)

Of special interest is the derivation of various strain tensors from the deformation gradient.

The*Green strain tensor*is given by

**E**=1
2

¡**F**^{T}**F**−I¢

, (A.9)

while for small deformations, the*infinitesimal strain tensor*is given by

* ²*=1
2

¡**F**+**F*** ^{T}*¢

−**I.** (A.10)

**B** **Digital Blood in Massively Parallel CPU/GPU Systems** **for the Study of Platelet Transport**

**B.1 Simulation Parameters**

**Parameter** **Value**

∆*x* 0.5*µm*

relaxation time*τ* 2

∆*t* *ν*=*C*^{2}* _{s}*(

*τ*−1/2)∆

*t*

Skalak B,C,D 5, 5000, 35*pN/µm*

*k**bend i ng* 1*pNµm*

*k**vol ume* 250

Rayleigh*α**D*,*β**D* 0, 0.01

*κ*damping {0.5, 0.7, 0.8, 0.9}

IBM cycles 1

RBC mesh resolution (surface vertices) 258 Platelet mesh resolution (surface vertices) 66

Fluid density, kinematic viscosity 1025*f g*/*µm*^{3}, 1*µm*^{2}/*µs*

**B.2 Methods**

The computational framework used in this study (chapter 3) simulates the particulate nature of blood, i.e. blood cells that are submerged in blood plasma. In more details, we suggest a high-performance computational framework for fully resolved 3D blood flow simulations.

The tool is based on three computational modules, namely the fluid solver, the solid solver and the fluid-solid interaction (FSI).

The fluid solver is based on the lattice Boltzmann method (LBM) [79] and the module uses Palabos [84, 1] as for the implementation of LBM. Palabos stands for Parallel Lattice Boltzmann Solver and it is an open source software maintained by the Scientific and Parallel Computing Group (SPC) at the University of Geneva. The FSI is done via an immersed boundary method known as multidirect forcing scheme and its implementation is realised in Palabos. Lastly,

the solid solver is based on the nodal projective finite elements method (npFEM) [73] and the module uses the npFEM library developed and maintained by Palabos team, based on the open-source ShapeOp library [2]. The npFEM library is integrated in Palabos core library (for more information on the released software see chapter 6).

In this section, a basic and brief outline of the used methods is given. For a thorough
presen-tation of the numerical methods, the interested readers should consult Kotsalos*et al.*[73]

(chapter 2).

**B.2.1 Nodal Projective FEM (npFEM)**

The deformable blood cells are simulated by the nodal projective finite elements method. The npFEM is a mass-lumped linear FE solver that resolves both the trajectories and deformations of blood cells with high accuracy. The solver has the capability of capturing the rich and non-linear viscoelastic behaviour of red blood cells as shown and validated in [73]. The rest of blood cells, platelets for the current study, are simulated as nearly-rigid bodies by increasing the stiffness of the material in the solid solver.

Let us assume a surface mesh consisting of*n*vertices with positions* x*∈R

*and velocities*

^{n×3}*∈ R*

**v**

^{n}^{×}

^{3}. The evolution of the body (trajectory & deformed shape) in time follows Newton’s laws of motion. At time

*t*

*n*, the system is described by {x

*n*,

**v***n*}. The external forces are defined as

**F***ext*∈R

^{n}^{×}

^{3}and are due to the fluid-solid interaction, while the internal forces are

**F***∈R*

_{i nt}

^{n}^{×}

^{3}and are due to the material properties. The internal forces are given by

**F***i nt*(x)= −P

*i*∇E*i*(x),
where*E**i*(x) is a scalar discrete elemental potential energy (the summation of all elemental
potential energies results in the total elastic potential energy of the body). The*implicit Euler*
*time integration*leads to the following advancement rules:

**v***n*+1=**x***n*+1−**x***n*

*h* , (B.1)

**F***i nt*(x*n*+1)+**F***ext*(x*n*)−**Cv***n*+1=**M****v***n*+1−**v***n*

*h* , (B.2)

where**M**∈R^{n}^{×}* ^{n}*is the mass matrix,

*h*is the time step, subscripts

*n*and

*n*+1 refer to time

*t*and

*t*+

*h, respectively and*

**C**∈R

*is the damping matrix acting like a proxy for viscoelasticity.*

^{n×n}Except for Rayleigh damping, we augment the viscoelastic behaviour through another proxy

Except for Rayleigh damping, we augment the viscoelastic behaviour through another proxy