Higher-order compact scheme for high-performance computing of stratified rotating flows

(1)

HAL Id: hal-02111489

https://hal-amu.archives-ouvertes.fr/hal-02111489

Submitted on 26 Apr 2019

HAL

is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire

HAL, est

destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Higher-order compact scheme for high-performance computing of stratified rotating flows

Stéphane Abide, Stéphane Viazzo, Isabelle Raspo, Anthony Randriamampianina

To cite this version:

Stéphane Abide, Stéphane Viazzo, Isabelle Raspo, Anthony Randriamampianina. Higher-order com-

pact scheme for high-performance computing of stratified rotating flows. Computers and Fluids,

Elsevier, 2018, 174, pp.300-310. �10.1016/j.compfluid.2018.07.016�. �hal-02111489�

(2)

stratiﬁed rotating ﬂows

Stéphane Abide^a^,^∗, Stéphane Viazzo^b, IsabelleRaspo^b, Anthony Randriamampianina^b

aUniversité de Perpignan Via Domitia, Perpignan LAMPS EA 4217, France

bAix Marseille Univ, CNRS, Centrale Marseille, M2P2, Marseille, France

a rt i c l e i nf o

Article history:

Received 13 April 2018 Revised 5 July 2018 Accepted 23 July 2018 Available online 25 July 2018 Keywords:

Higher-order compact scheme High-Performance computing Rotating stratiﬁed ﬂows 2D-pencil decomposition Reduced parallel diagonal dominant

a b s t ra c t

To takeadvantageofmoderngeneration computinghardware,ascalable numericalmethod, basedon higher-ordercompact scheme, isdescribed tosolve rotatingstratifiedflows incylindrical annulardo- mains.Anoriginalapproachcombining2d-pencildecompositionandreducedParallelDiagonalDominant is proposed toimprove the parallelization performance duringthe computation of Poisson/Helmholtz solversandtimeexplicitterms.Thedevelopedtechniqueisvalidatedwithrespecttoanalyticalsolutions, usingthemethodofmanufacturedsolutions,andavailabledatafortwospecificconfigurations.Thepur- poseistodemonstrateitsabilitytocorrectlycapture theflowcharacteristicsinstrato-rotationalinsta- bilityand inbaroclinicinstabilitywithassociatedsmall-scalefeatures.Moreover,thiscodeisfoundto drasticallyreducethehugeexecutiontimesoftenpreventingdetailednumericalinvestigationsofthese complexphenomena.Strongscalingtestiscarriedouttoassesstheperformance forupto1024cores usinggridupto128×568×568inradial,axialandazimuthaldirections.

1. Introduction

Currenttrendsincomputationalsciencestemfromtheincreas- ingprominenceofmodernmulti-andmany-corehardwaresuchas graphics processingunits (GPUs) in high-performance computing (HPC)infrastructure.Improvedmulti-core centralprocessingunits (CPUs) associated with GPU-accelerated computing can greatly accelerate scientiﬁc studies especially in large-scale simulations [1–3].However,thereductionofexecutiontimesstronglydepends ontheoptimizationoftheperformanceofdiscretizationmethods usedoneach nodewhilegoodcomputationalscalingresultsfrom smallcommunicationfootprint[4].

Even though finite element methods are becoming common- placeforsimulationsofmostengineeringandbiomedicalapplica- tionsoncomplexgeometries,low-orderdiscretizationssufferfrom strongmeshrefinement requiredtocapturecertain specific com- plexsolution features. Incontrast,high-order techniques,such as spectralapproaches,provideimprovednumericalcharacteristicsat reducedcomputationalcostforagivennumberofdegreesoffree- dombutthey usually remain restricted toregular configurations.

Recently,thespectral/hpelementmethods[5–7]combinehighac- curacy, known as the spectral convergence, and geometric ﬂex-

∗ Corresponding author.

E-mail address: [email protected] (S. Abide).

ibility to tackle challenging aeronautical flow simulations using between O(¹⁰⁴⁾ ând O(¹⁰⁵⁾ ^nodes ^[8]^, ând êven ûp ^to O(¹⁰⁶⁾ nodes[9].

In the present study, a higher-order compact scheme solver [10]istailoredforsolvingcomplexrotatingstratifiedflows inan- nulardomainstotakeadvantageofmoderngenerationcomputing hardware. Our goal is to reduce the huge total time due to the very longtransientstage resultingfromthe thermalstratification associatedwiththerotation.Wehavechosentotreattwospecific idealizedconfigurations: a Taylor–Couette setup submittedto ax- ial stratification forthe strato-rotational instability (SRI) [11] and a uniformly rotating cavity submitted to radial thermal gradient for the baroclinic instability [12]. In the first case, the geometry isdefinedbya largeaspectratiowhilethesecond oneischarac- terizedbythecoexistenceofverydifferentspatio-temporalscales [13].Beforeconsideringmoredeeplyactualphysicsarisinginthese twoconfigurations, thiswork reportsonthe validationofthede- velopednumericaltoolwithrespecttodataavailableinthelitera- ture. Wehaverestrictedoursimulations toremaininthedomain of theBoussinesq approximation: weak densitystratification and weakrotationratetoneglectthecentrifugalacceleration.

Most of common discretization such as finite volumes, finite differences, finite elements or lattice Boltzmann method, is ef- ficiently implemented in parallel computing environment, even though massively parallel linear solvers still remain a challenging work [14–17]. Concerning high accuracy discretization, the https://doi.org/10.1016/j.compfluid.2018.07.016

(3)

picture is more mixed. The computational stencils of such discretizations arelargerthanthoseofthecommonlower-orderfor- mulations, therefore inducing large parallel communications. The 2d-pencil decomposition became the common strategy to deal with large computational stencils [18–22]. The successful exam- ple of the massively parallel FFT computations with1d then 2d pencildecomposition[21]hasbeentailoringtotheNavier–Stokes solvers based on higher-order discretizations [18,23]. Nowadays, thisstrategyofparallelizationoffersa significantreductionofthe wall-time and allows the study of highly turbulent flows. Be- causeoftheexpensivecostofthecommunicationsinherentto2d- pencildecompositions,somechoicesinthedesignofthenumeri- cal methodmust bemadeto limitcommunications.Forinstance, one canchoose time-explicitadvancementschemes forthediffu- sive terms,Fourierseriesexpansions, oracollocatedvariablelay- out[18].Compactschemesarefinitedifferencesbasedonimplicit relations[24] whichrelyonsolutionsoftridiagonalorpentadiag- onallinearsystems.Therefore,higher-ordercompactschemescan beconsideredasaninterestingalternativetospectralmethods,as- sociating the robustness offinite difference with improvedaccu- racy. Inarecentwork, Abideetal.[10]proposed an approximate parallel tridiagonal solver [25] to avoidthe use of 2d-pencil de- compositionforthecomputationofcompactderivativesandinter- polations.Inthiscase,thecommunicationsbetweenneighborpro- cessrely onhaloexchange algorithm.Nevertheless,the 2d-pencil decompositionisstillconsideredtotailoraparallelversion ofthe full diagonalizationmethod [26], despitethe cost ofthecommu- nications involvedby thedatatranspositions. Thecombinationof thetwoparallelizationstrategiesallowstheauthorstosuccessfully simulateincompressibleflow intrulythree-dimensionalcartesian domain.Moreover,thebasicnumericalmethodfeaturesforturbu- lentflow simulationssuch asthestaggered gridandimplicitvis- coustermsareconserved,whileexhibitingagoodstrongscaling.

The proposed numerical method is based on Fourier/Fourth- order compact scheme discretization deﬁned on staggered grid.

The second order Adams–Bashforth/Backward Euler time scheme combined with an efficient projection scheme has been consid- ered:thisapproachrelaxesthestringenttime stepconstraintsin- herenttothediffusiveterms[27].Itcorrespondstoanextensionof apreviousnumericalcode,developedincartesiancoordinates[10], forsolvingrotatingstratifiedflowsinannulardomains.Moreover, the presentparallelizationstrategy isconcerned witha 2d-pencil decompositionandreducedParallelDiagonal Dominant(rPDD)to address the numerical solutions of Helmholtz/Poisson problems and the computations of the compact derivatives and interpola- tionsrespectively.Theimplementationofthecombinationofthese two techniquesbrings novelty andoriginality to the present ap- proachincomparisonwithpreviousworksbasedonhigher-order codes[18].

Therotatingstratifiedflowsconcernawidespectrumofappli- cationsaswellasingeophysicssuch aslargescalecirculationsin atmosphereandoceans, astrophysics such asaccretiondisk,than in diskstorage, turbomachinery… The Strato-RotationalInstabil- ity (SRI) is a purely hydrodynamicinstability with distinctive lo- calfeaturesandmaybestudiedfromspecificallydesignedlabora- toryexperimentsandnumericalsimulationsinanaxially-stratified Taylor–Couettesetup[11,28,29].Hereaxialstratificationisobtained bycoolingthebottomandheatingtheupperlid[11].Thetempera- turestratificationrepresentstheaxialstratificationofanaccretion diskcentered around ahot starandthe rotationof thecylinders mimics theastrophysicalrotationlawswithrotationratedecreas- ing outwards. Understanding the mechanisms that can result in an outward angular momentum transport is thecentral problem of planetformation, particularly inthe theory of accretiondisks.

Whenaplanetformsinadisk,angularmomentumhastobecar- ried awayfrom theplanetotherwise its rotationspeed wouldbe

fartoolarge.Onlyturbulencecanachievesuchalargeangularmo- mentum transport. Recent studies mentioned that, atﬁxed value ofthe Froudenumber, Fr=i/N,with N=

gα∂^T/∂^z ^the ^buoy-

ancyfrequencyandithe angularvelocity oftheinner cylinder, thesystem ismore stablewhen increasing theReynolds number Re=iR_i(^Ro−R_i)/ν^[30,31]^.^The^finalôbjectiveîs^toêventuallyêx-

plorethenotyetunderstoodturbulentﬂows,togetclosertoactual astrophysicalapplications,bytakingadvantageofoptimizedHPC.

Baroclinic instability is recognized to be one of the dominant energetic processes in the large-scale atmospheres of terrestrial planets,suchastheEarthandMars[32],andintheoceans[33].Its time-dependentbehaviour exertsadominantinfluenceonthein- trinsicpredictabilityoftheatmosphericcirculationandthedegree ofchaoticvariabilityinitslarge-scalemeteorology[32,34,35].Iner- tiagravitywaves(IGWs)areubiquitousintheatmosphereandthe oceansandareknownto playafundamentalroleina widevari- etyofprocesses.Thecontributionsmainlyconcernthetransportof asignificantamountofenergyandmomentum,theinitiationand organizationofconvection,theinductionandmodulationofturbu- lence,aswellasthemodificationofthemeancirculationandther- malstructure ofatmospheric andoceanicmotions[36].Observa- tionsandsimulationshaverevealedtheirspontaneousoccurrence duringthedevelopmentofbaroclinicinstability.Inspiteofinten- siveresearchactivitiescarriedoutoverthelastdecades,thegener- ationmechanismandthepropagationofIGWs,aswellastheirin- teractionwithlarge-scalestructures, remainpoorly understood.A betterunderstanding ofthesephenomenaisthereforemandatory fortheimprovementoftheIGWparameterizationschemesactually required to upgrade numerical global weather predictions. Since thepioneeringworksofHide[12],thedifferentiallyheated,rotat- ingcylindricalannulushasbeenan archetypalmeansofstudying thepropertiesoffully-developedbaroclinicinstabilityinthelabo- ratory.Recentworksmainlybasedondirectnumericalsimulations reported the occurrenceof IGWs in water-filled baroclinic cavity alongtheinnercoldcylinder[37–39].Moreover,VonLarcheretal.

[39]observedthepresenceofadditionalsmall-scaleripplesresult- ingfromhydrodynamicalinstabilityalongthehotwall.

Thepaperisorganizedasfollows.First,thephysicalandmath- ematical models of the SRI and Baroclinic conﬁgurations are de- tailed.ThenumericalmethodstosolvethecoupledNavier–Stokes andenergy equations in annuli are introduced before describing the parallelizations strategies. The last section is devoted to the validationofthenewnumericalsolver. Accuracywill bechecked, validations with existing ﬁndings for strato-rotational and baro- clinicinstabilitiesareperformedincludingastrongscalingtest.

2. Physicalandmathematicalmodels 2.1. Physicalmodel

This studyfocuses on flows in rotatingannular cavities filled withanincompressiblefluidandsubmittedtoeitheraradialora verticalthermalgradient.Intheseconfigurations,theflow results fromthecompetitionbetweentherotationofthecylindersandthe buoyancyforcearisingfromtheprescribedtemperaturegradient.

Thetwoconﬁgurations consideredtogether withthenotations aredepictedinFig.1.

Thefirstconfigurationcorresponds toa Taylor–Couettesystem (Fig.1a)wherethe fluidissubmitted toa stableverticalthermal gradient, whichmeans that the top boundary is heatedwhereas the bottom one is cooled. The inner cylinder rotates faster than the outer one. To reduce the edge effects, stress-free conditions are prescribed for velocity on the two horizontal disks. Despite the stabilizing effect of the vertical stratification induced by the stable thermal gradient, an hydrodynamic instability, named the strato-rotationalinstability,maydevelopforspecific valuesofthe

(4)

Fig. 1. Rotating annular cavities conﬁgurations.

control parameters which are the angular speed ratio μ^, ^the

ReynoldsnumberReandtheFroudenumberFrdeﬁnedby:

μ= o

i, Re=idRi

ν ^, ^F^r⁼^Nⁱ ⁽¹⁾

withd=Ro−R_i,ν ^the^kinematic ^viscosity,i andô ^the ângu- larspeedsoftheinnerandouter cylindersrespectively,andN the Brunt–Väisälä frequencywhich isa measureofthe verticalstrat- ificationscale.Intheframework oftheBoussinesqapproximation (seesection2.2),Nisdefinedby:

N²=α^g^T

H (2)

withα^the^thermal^expansion^coeﬃcient.

Inthesecond configuration(Fig.1b),aradial temperaturegra- dientisprescribed by heatingtheouter cylinder andcoolingthe innerone.Unlikethepreviouscase,thetwo verticalcylindersro- tate with the same angular velocity ^. ^The ^top ^boundary îs ân open free surface. This configuration was found to be a suitable testbedforstudyingmid-latitudeatmosphericflowswhereabaro- clinicinstability togetherwithinertia gravitywavesmaydevelop.

TheclassicalcontrolparametersaretheTaylornumberTaandthe RossbynumberRo[12],deﬁnedby:

Ta= 4²^d⁵

ν²^H ^, ^Ro⁼α^T^gH

²^d² ⁽³⁾

2.2.Mathematicalmodel

Inbothconﬁgurations,theﬂowandheattransferaregoverned by the incompressible Navier–Stokesequations coupled with en- ergyequation through the Boussinesq approximation.With these assumptions,thesetofequationsreads:

∇.u=0 inD

∂^tû⁺¹2[(û.∇)û+∇^.(ûu)^]=−∇^p⁺νû+F inD

∂^t^T⁺¹2[(^u^.∇)^T⁺∇^.(^u^T)^]⁼κ∇²^T ⁱⁿ^D ⁽⁴⁾

D beingthe computational domain,κ ^the ^ﬂuid ^thermal ^diffusiv-

ity and u=(^u^r,u_θ,uz), p and T standing for the velocity, pres- sureand temperature ﬁelds, respectively. The vectorial Laplacian

Table 1

Source terms F and boundary conditions in the SRI and baroclinic ﬂows ( T hand T cstand for the hot and cold temperatures).

Vertical ∇^T ^Radial∇^T

F α(T −T 0)g α(T −T 0)g+2 ×u+×(×r ) Top wall Free stress Free stress

Bottom wall Free stress No-slip Lateral walls Differential rotation No-slip

Top wall T h Adiabatic

Bottom wall T c Adiabatic

Inner wall Adiabatic T c

Outer wall Adiabatic T h

^u^expressed ⁱⁿcylindricalcoordinatesis givenby thefollowing formula:

^u=

∇²(^u^r)⁻^u^r^/r²⁻(²^/r²)∂θu_θ

∇²(^uθ)⁻^uθ/r²+(²^/r²)∂θur

∇²(^uz)

(5)

where∇²^(.)^stands^for^the^scalar^Laplacian^deﬁned^by:

∂r²+(¹/r)∂^r⁺(¹/r²)∂_θ²⁺∂z² (6) InEq. (4), thesource termFaccountsforexternal forces,mainly the buoyancyforce. For theﬁrst casecorresponding toa Taylor–

Couette configuration (Fig. 1a), the equations are expressed in a fixed frame. For the second configuration with a radial thermal gradient(Fig.1b),theequationsaresolved inarotatingreference frame and, asa consequence, inaddition to the buoyancy force, Coriolisandcentrifugalforcesare alsoincludedin F.The expres- sionofFandtheboundaryconditionsaregiveninTable1forthe twoconfigurations.

3. Numericalmethod

3.1. Timediscretizationandprojectionscheme

The second order semi-implicit Adams–Bashforth/Backward- Euler scheme is used for the time advancement of Eq. (4). The