Оптимизированная реализация теста HPCG для суперкомпьютера с сетью "Ангара" (Optimized implementation of HPCG benchmark on supercomputer with "Angara" interconnect.)

Îïòèìèçàöèÿ òåñòà HPCG äëÿ ñóïåðêîìïüþòåðîâ ñ

ñåòüþ ¾Àíãàðà¿

À.À. Àãàðêîâ, À.Ñ. Ñåìåíîâ, À.Ñ.Ñèìîíîâ

ÎÀξÍÈÖÝÂÒ¿

ÄàííàÿðàáîòàïîñâÿùåíàîïòèìèçèðîâàííîéðåàëèçàöèèòåñòàHPCGäëÿñóïåð-

êîìïüþòåðîâññåòüþ¾Àíãàðà¿,ðàçðàáîòàííîéâ ÎÀξÍÈÖÝÂÒ¿.ÒåñòHPCG

çàêëþ÷àåòñÿ â ðåøåíèè ñèñòåìû ëèíåéíûõ óðàâíåíèéñ ðàçðåæåííîé ìàòðèöåé

ìåòîäîì ñîïðÿæåííûõ ãðàäèåíòîâ ñ ïðåäîáóñëàâëèâàòåëåì. Ïðåäëàãàåìàÿ ðåà-

ëèçàöèÿ âêëþ÷àåò îïòèìèçàöèè óðîâíÿâû÷èñëèòåëüíîãî óçëà, â òîì ÷èñëå èç-

ìåíåíèåîðìàòà õðàíåíèÿðàçðåæåííîéìàòðèöû,OpenMP-ðàñïàðàëëåëèâàíèå

è âåêòîðèçàöèþ, à òàêæå îïòèìèçàöèþìåæïðîöåññíûõ îáìåíîâäëÿ ñåòè ¾Àí-

ãàðà¿ïðè ïîìîùèèñïîëüçîâàíèÿ áèáëèîòåêèSHMEM âìåñòîáèáëèîòåêèMPI.

 ðàáîòå ïðåäñòàâëåíû ïåðâûå ðåçóëüòàòû íà ìàêåòíîì 36-óçëîâîì êëàñòåðå ñ

ñåòüþ ¾Àíãàðà¿, êîòîðûå ïðåâîñõîäÿò ðåçóëüòàòû áàçîâîé âåðñèè òåñòà HPCG

2.4.Êëþ÷åâûåñëîâà:êîììóíèêàöèîííàÿñåòü¾Àíãàðà¿,áèáëèîòåêàSHMEM,HPCG

Ââåäåíèå

HPCG [1℄ íîâûé òåñò, ïðåäíàçíà÷åííûé äëÿ äîïîëíåíèÿ òåñòà LINPACK (HPL) [2℄,

êîòîðûé â íàñòîÿùåå âðåìÿ èñïîëüçóåòñÿ äëÿ ðàíæèðîâàíèÿ ñóïåðêîìïüþòåðîâ â ñïèñêå

TOP500. Â îñíîâå HPCG ëåæèò ðåøåíèå ëèíåéíûõ óðàâíåíèé ðàçðåæåííîé ìàòðèöåé

áîëüøîé ðàçìåðíîñòè ïðè ïîìîùè èòåðàöèîííîãî ìåòîäà ñîïðÿæåííûõ ãðàäèåíòîâ ñ ìíî-

ãîñåòî÷íûìïðåäîáóñëàâëèâàòåëåì.Âîòëè÷èåîòHPLòåñòHPCGîáåñïå÷èâàåòñòðåññîâóþ

íàãðóçêó ïîäñèñòåìûïàìÿòè âû÷èñëèòåëüíûõ óçëîâ è êîììóíèêàöèîííîé ñåòè, ïðåäñòàâ-

ëÿÿçíà÷èòåëüíûé êëàñññîâðåìåííûõ ñóïåðêîìïüþòåðíûõïðèëîæåíèé.

Òåñò HPCG ñòàë ïðåäìåòîì ïðèñòàëüíîãî âíèìàíèÿ èññëåäîâàòåëåé âñåãî ìèðà, ïî-

ÿâèëèñü ðàáîòû ïî åãîîïòèìèçàöèèäëÿ ñóïåðêîìïüþòåðîâTianhe-2 [3,4℄,K-omputer [5℄,

èññëåäóþòñÿâîçìîæíîñòèóñêîðèòåëåéNVIDIAGPU[6℄èIntelXeonPhi[3℄,âûøëàðàáîòà

ïî ïðåäñêàçàíèþïðîèçâîäèòåëüíîñòèHPCG[7℄.îññèéñêèåèññëåäîâàòåëèòàêæåóäåëÿþò

âíèìàíèåýòîìó òåñòó[8, 9℄.

 ÎÀξÍÈÖÝÂÒ¿ ðàçðàáàòûâàåòñÿâûñîêîñêîðîñòíàÿ êîììóíèêàöèîííàÿ ñåòü ¾Àí-

ãàðà¿ ñ òîïîëîãèåé ¾ìíîãîìåðíûé òîð¿ [1014℄.  2013 ãîäó âûïóùåí êðèñòàëë ìàðøðó-

òèçàòîðà ñåòè [15℄, â 2014 ãîäóïðåäñòàâëåíû ïðåäâàðèòåëüíûå ðåçóëüòàòû îöåíî÷íîãî òå-

ñòèðîâàíèÿ12-óçëîâîãîêëàñòåðàñýòîéñåòüþ[16℄.Âíàñòîÿùèéìîìåíòâåäeòñÿîöåíî÷íîå

òåñòèðîâàíèå36-óçëîâîãî êëàñòåðàñ ñåòüþ¾Àíãàðà¿.

Ïåðå÷èñëåííûå ðàáîòû ïî îïòèìèçàöèè HPCG íå ïðåäîñòàâëÿþò èñõîäíûõ êîäîâ, â

ñâÿçè ýòèì è äëÿ áîëåå ãëóáîêîãî ïîíèìàíèÿ ýòîãî òåñòà áûëî ïðèíÿòî ðåøåíèå ðàçðà-

áîòàòü ñîáñòâåííóþ ðåàëèçàöèþ, ïðèçâàííóþ, â ÷àñòíîñòè,äåìîíñòðèðîâàòüâîçìîæíîñòè

ñåòè¾Àíãàðà¿.Äàííàÿ ðàáîòàïðåäñòàâëÿåòïåðâûå ðåçóëüòàòûîïòèìèçàöèèHPCG.

1. Îïèñàíèå òåñòà HPCG

ÂòåñòåHPCGðåøàåòñÿñèñòåìàëèíåéíûõóðàâíåíèé

Ax = b

ñ ðàçðåæåííîéìàòðèöåé

A

è äàííûì íà÷àëüíûì ïðèáëèæåíèåì

x ₀

. Ìàòðèöà ñèñòåìû ïðåäñòàâëÿåòñîáîé äèñêðå-

òèçàöèþ óðàâíåíèÿ â ÷àñòíûõ ïðîèçâîäíûõ ýëëèïòè÷åñêîãî òèïà ñ 27-òî÷å÷íûì øàáëî-

íîì íàðåãóëÿðíîé òðåõìåðíîéñåòêå.Ñòðîêèâ ðàçðåæåííîéìàòðèöå ïðåäñòàâëÿþò òî÷êè

â ñåòêå. Ìàòðèöà èìååò äîñòàòî÷íî ïðîñòóþ ñòðóêòóðó, ÷òî óäîáíî äëÿ èíèöèàëèçàöèè

çàäà÷è è ïðîâåðêè ïðàâèëüíîñòè îïòèìèçèðîâàííûõ ðåøåíèé, îäíàêî îïòèìèçèðîâàííûå

ïîëüçîâàòåëüñêèå ðåàëèçàöèè íåäîëæíû îïèðàòüñÿíà ñòðóêòóðó ìàòðèöû äëÿóñêîðåíèÿ

âû÷èñëåíèé, ïîýòîìó ñëåäóåò ïîëàãàòü,÷òî èìååòñÿðàçðåæåííàÿ ìàòðèöà îáùåãî âèäà.Â

÷àñòíîñòè,íåëüçÿîòêàçûâàòüñÿîòêîñâåííîéàäðåñàöèèïðèäîñòóïåêñòðóêòóðàìäàííûõ,

â êîòîðûõ õðàíèòñÿ ìàòðèöà.

 HPCG êàæäûé MPI-ïðîöåññ îòâå÷àåò çà ïîäìíîæåñòâî ñòðîê ìàòðèöû

A

^{è ýëåìåí-}

òîâ âåêòîðà

b

^,ñîîòâåòñòâóþùåå ðàçìåðó ïîäîáëàñòè

N _x ∗ N _y ∗ N _z

^{,êîòîðûé çàäàåòñÿ ïîëü-}

çîâàòåëåì â êîíèãóðàöèîííîì àéëå. Ïðîöåññû âûñòðàèâàþòñÿ â òðåõìåðíóþ ðåøåòêó

P = P _x ∗ P _y ∗ P _z

^,ãäå

P

^{îáùåå÷èñëîïðîöåññîâ.Èòîãîâûéðàçìåðçàäà÷èðàâåí}

G _x ∗ G _y ∗ G _z

^,

ãäå

G _x = P _x ∗ N _x , Gy = P _y ∗ N _y , G _z = P _z ∗ N _z

^.

 òåñòå HPCG ðåøàåòñÿ ñèñòåìà óðàâíåíèé ïðè ïîìîùè âûïîëíåíèÿ â áàçîâîé ðåà-

ëèçàöèè (âåðñèÿ 2.4) 50 èòåðàöèé ìåòîäà ñîïðÿæåííûõ ãðàäèåíòîâ CG.  íà÷àëå êàæäîé

èòåðàöèè ìåòîäà CG âûïîëíÿåòñÿ ïðåäîáóñëàâëèâàíèå ñèñòåìû ïîìîùüþ èòåðàòèâíîãî

ìíîãîñåòî÷íîãî ìåòîäà (âûïîëíÿåòñÿ 3 èòåðàöèè) ñ èñïîëüçîâàíèåì ñèììåòðè÷íîãî ñãëà-

æèâàòåëÿ àóññà-Çåéäåëÿ.

 ñîîòâåòñòâèè ñî ñïåöèèêàöèåé ïîëüçîâàòåëü ìîæåò îïòèìèçèðîâàòü ñëåäóþùèå

óíêöèè:

1. åàëèçàöèþ ìåòîäà ñèììåòðè÷íîãî àóññà-Çåéäåëÿ (ComputeSYMGS) è êîä ìíîãîñå-

òî÷íîãîïðåäîáóñëàëèâàòåëÿ.

2. Âû÷èñëåíèå ïðîèçâåäåíèÿðàçðåæåííîéìàòðèöûíà âåêòîð(ComputeSPMV).

3. Âû÷èñëåíèå ñêàëÿðíîãîïðîèçâåäåíèÿ âåêòîðîâ(ComputeDOT).

4. Âû÷èñëåíèå ñóììûâåêòîðîâ âèäà:

a ∗ X + b ∗ Y

^{(ComputeWAXPBY).}

Ïîëüçîâàòåëüñêîåðåøåíèåìîæåò òàêæåèçìåíÿòüîðìàòõðàíåíèÿìàòðèöû èïðîèç-

âîäèòüïåðåñòàíîâêèñòðîê.Îäíàêîïîòðåáîâàíèþñïåöèèêàöèèîïòèìèçèðîâàííàÿïîëü-

çîâàòåëüêàÿ ðåàëèçàöèÿ äîëæíà äîñòè÷ü òîé òî÷íîñòè ðåøåíèÿ ñèñòåìû, ÷òî è áàçîâàÿ

âåðñèÿHPCG. Îïòèìèçàöèè ìîãóòïðèâîäèòü êóìåíüøåíèþ ñêîðîñòèñõîäèìîñòè çàäà÷è,

â ýòîìñëó÷àåïðè òåñòèðîâàíèèðåàëèçàöèèâûïîëíÿåòñÿíåîáõîäèìîå÷èñëîèòåðàöèé,êî-

òîðîå áîëüøå áàçîâûõ 50 èòåðàöèé. Âðåìÿ, òðåáóþùååñÿ íà äîïîëíèòåëüíûå èòåðàöèè, à

òàêæå âðåìÿ ïðåîáðàçîâàíèÿ ñòðóêòóð äàííûõ, íåîáõîäèìûõ äëÿ îïòèìèçàöèè, âõîäèò âî

âðåìÿ ðåøåíèÿçàäà÷èè ó÷èòûâàåòñÿïðè ïîäñ÷åòå ïðîèçâîäèòåëüíîñòè.

2. Îïòèìèçàöèè

Âûïîëíåííûå â ðàáîòå îïòèìèçàöèè òåñòà HPCG ðàçäåëÿþòñÿ íà äâå êàòåãîðèè: îä-

íîóçëîâûå îïòèìèçàöèèè îïòèìèçàöèèìåæóçëîâûõêîììóíèêàöèé.Îäíîóçëîâûåîïòèìè-

çàöèè âêëþ÷àþò îðãàíèçàöèþ áîëåå ýåêòèâíîé ðàáîòû ñ îïåðàòèâíîé ïàìÿòüþ âíóòðè

âû÷èñëèòåëüíîãî óçëà, âåêòîðèçàöèþ è ðàñïàðàëëåëèâàíèå óíêöèé ñ ïîìîùüþ òåõíîëî-

ãèè OpenMP.Òàêèå îïòèìèçàöèèîáùåèçâåñòíû,îíè ðàññìàòðèâàëèñüâ ðàáîòàõ[35℄.Íà-

ïðÿìóþ ýòè îïòèìèçàöèè íå êàñàþòñÿ ñåòè ¾Àíãàðà¿, íî îíè íåîáõîäèìû äëÿ ïîëó÷åíèÿ

âûñîêîéïðîèçâîäèòåëüíîñòè,èõîïèñàíèåíàõîäèòñÿâðàçäåëàõ 2.12.3.Âðàçäåëå2.4ðàñ-

ñìîòðåíû ñïåöèè÷åñêèå äëÿñåòè¾Àíãàðà¿ ìåæóçëîâûåêîììóíèêàöèè.

åçóëüòàòûïðîèçâîäèòåëüíîñòèäëÿâàðèàíòîâîäíîóçëîâîéîïòèìèçàöèèïîëó÷åíûíà

óçëå ñ äâóìÿ øåñòèÿäåðíûìèïðîöåññîðàìèñ âêëþ÷åííûì ðåæèìîìHyper-Threading.Ïà-

ðàìåòðû óçëà(òèïÀ)ïðåäñòàâëåíûâî âòîðîì ñòîëáöåòàáëèöû1.

Òàáëèöà1. Ïàðàìåòðûâû÷èñëèòåëüíûõ óçëîâ

Ïàðàìåòð

Óçåë òèïàA Óçåë òèïà B

Ïðîöåññîð

2xIntel XeonE5-2630 IntelXeonE5-2660

Òàêòîâàÿ÷àñòîòà ïðîöåññîðà,ö

2.3 2.2

Êîëè÷åñòâîÿäåð â óçëå

12 8

àçìåð êýøàL3,ÌÁ

15 20

Ïèêîâàÿïðîèçâîäèòåëüíîñòüóçëà,ëîïñ

220.8 140.8

Òàáëèöà 2.Ïàðàìåòðûâû÷èñëèòåëüíîãî êëàñòåðà ññåòüþ ¾Àíãàðà¿

Ïàðàìåòð

Çíà÷åíèå

Êîëè÷åñòâîóçëîâ òèïà À

24

Êîëè÷åñòâîóçëîâ òèïà B

12

Îáùàÿïèêîâàÿ ïðîèçâîäèòåëüíîñòü,ëîïñ

6988.8

Èíòåðåéñ ïðîöåññîðàñ ñåòüþ¾Àíãàðà¿

PCIegen2x16

Òàêòîâàÿ÷àñòîòàñåòåâîãîàäàïòåðà, Ìö

500

Çàäåðæêà ïåðåäà÷è 16-áàéòîâîãî ñîîáùåíèÿ

ìåæäó äâóìÿ óçëàìè,SHMEM/MPI,ìêñ

0.7/1

2.1. Âûäåëåíèå íåïðåðûâíîãî áëîêà ïàìÿòè

 èñõîäíîì âàðèàíòå HPCG ïàìÿòü äëÿ ìàññèâîâ ýëåìåíòîâ è èíäåêñîâðàçðåæåííîé

ìàòðèöûâûäåëÿåòñÿáëîêàìè,êàæäûéáëîêèìååòðàçìåð,ðàâíûéìàêñèìàëüíîìóêîëè÷å-

ñòâóíåíóëåâûõýëåìåíòîââ ñòðîêå.Áëî÷íîå âûäåëåíèåïàìÿòè ïðåïÿòñòâóåòýåêòèâíî-

ìóèñïîëüçîâàíèþâñåéïîäñèñòåìûïàìÿòè,â÷àñòíîñòèêýøåéäàííûõ,êýøàäåñêðèïòîðîâ

ñòðàíèö (TLB).Ïåðâîå, ÷òî áûëîïðåäïðèíÿòî ýòîíåïðåðûâíîå âûäåëåíèå ïàìÿòè äëÿ

ýòèõìàññèâîâ. åçóëüòàòûïðîèçâîäèòåëüíîñòèäëÿëîêàëüíîéîáëàñòè16x16x128ïðèâåäå-

íû íà ðèñ. 2.Âûäåëåíèå íåïðåðûâíîãî áëîêà ïàìÿòè ïîçâîëèëî ïîëó÷èòü ïðèðîñò ïðîèç-

âîäèòåëüíîñòè âñðåäíåìâ ïîëòîðàðàçàïî ñðàâíåíèþñ áàçîâîéðåàëèçàöèåé.

2.2. àñêðàñêà ìàòðèöû

Ôóíêöèÿ ComputeSYMGS, ðåàëèçóþùàÿ ñèììåòðè÷íûé ìåòîä àóññà-Çåéäåëÿ

(SYMGS), çàíèìàåò íàèáîëüøåå âðåìÿ ñðåäè âñåõ îñòàëüíûõ óíêöèé, ïîýòîìó ÿâëÿåòñÿ

îäíîé èçîñíîâíûõ,íî èíàèáîëåå ñëîæíûõçàäà÷ äëÿîïòèìèçàöèè.

 òåñòå HPCG äëÿ ïîñòðîåíèÿ èñõîäíîé ìàòðèöû èñïîëüçóåòñÿ 27-òî÷å÷íûé øàáëîí

â òðåõìåðíîì ïðîñòðàíñòâå (ñì. ðèñ. 1), êàæäàÿ òî÷êà çàäàííîé îáëàñòè ñîîòâåòñòâóåò

êàêîé-òîñòðîêå ìàòðèöûè ýëåìåíòó âåêòîðà. Ïðèìåíåíèå SYMGSíà êàæäîìóðîâíå ñåò-

êè âêëþ÷àåò â ñåáÿ îáìåí ãðàíè÷íûìè çíà÷åíèÿìè ìåæäó ñîñåäíèìè âû÷èñëèòåëüíûìè

óçëàìè, çà êîòîðûì ñëåäóåò ïðÿìîéè îáðàòíûé õîä SYMGS. Íàâûõîäå óíêöèÿ SYMGS

âîçâðàùàåò îáíîâëåííûéâåêòîð çíà÷åíèé

x

. Îñíîâíàÿ ñëîæíîñòü çàêëþ÷àåòñÿ â òîì,÷òî äëÿ ïîëó÷åíèÿ çíà÷åíèÿ

x _i

èñïîëüçóþòñÿ íîâûå çíà÷åíèÿ

x ₁ ...x _i

− 1

^{â ïðÿìîì è}

x _i+1 ...x _n

â îáðàòíîìõîäå,÷òî äåëàåòíåâîçìîæíûìïàðàëëåëüíîå âûïîëíåíèåâñåõ òàêèõ îïåðàöèé.

ÄëÿðàñïàðàëëåëèâàíèÿSYMGS,â÷àñòíîñòèïðèîïòèìèçàöèèòåñòàHPCG[35℄,ïðè-

ìåíÿåòñÿðàñêðàøèâàíèåòî÷åêîáëàñòè. Ïåðâûéâàðèàíòîïòèìèçàöèèñîñòîÿëâðàñêðàñêå

îáëàñòè â 8 öâåòîâ (ñì. ðèñ. 1), ïðè òàêîé ðàñêðàñêå ýëåìåíòû âåêòîðà, ñîîòâåòñòâóþùèå

îäíîìóöâåòó,ìîãóò âû÷èñëÿòüñÿïàðàëëåëüíî.Ýòî ñäåëàëîâîçìîæíûì ïàðàëëåëüíîåâû-

÷èñëåíèå ýëåìåíòîâñíà÷àëà âî âðåìÿïðÿìîãî,çàòåìâî âðåìÿîáðàòíîãî õîäà SYMGS,íî

óõóäøèëîñõîäèìîñòüìåòîäà.Íà12-òèïîòîêàõOpenMPóñêîðåíèåîòíîñèòåëüíîïðåäûäó-

ùåãîâàðèàíòàñ íåïðåðûâíîéïàìÿòüþ ñîñòàâèëîáîëåå3-õ ðàç,ñì.ðèñ.2.

Âî âòîðîì âàðèàíòå ðàñêðàñêà ïðîèçâîäèëàñü âñåãî â 2 öâåòà (ñì. ðèñ. 3) âäîëü îñè

Z

.Òî÷êè, ñîîòâåòñòâóþùèåïëîñêîñòÿì

X Y

îäíîãî öâåòà,ìîãóò ðàññ÷èòûâàâàòüñÿ ïàðàë- ëåëüíîäðóãäðóãó,âñðàâíåíèèñïðåäûäóùèìâàðèàíòîìðàñêðàñêèýòîíåìíîãîóëó÷øàåò

ñõîäèìîñòü,÷òîâëèÿåòíàïðîèçâîäèòåëüíîñòü,ñì.ðèñ.2.Åñëèðàñêðàøèâàòüíåïîîäíîé,

àïîäâå ïëîñêîñòè

X Y

(ñì.ðèñ.4),òîñõîäèìîñòüóëó÷øàåòñÿåùåáîëüøå,èíàðèñ.2âèä- íî, ÷òî ýòîò âàðèàíòïîêàçûâàåòíàèëó÷øèå ðåçóëüòàòû èç âñåõ ïðåäëîæåííûõ âàðèàíòîâ

ðàñêðàñêè. Ïðîèçâîäèòåëüíîñòü ýòîãî âàðèàíòà â 4.8 ðàçàïðåâûøàåò ïðîèçâîäèòåëüíîñòü

âàðèàíòñíåïðåðûâíîéïàìÿòüþíà12-òèïîòîêàõOpenMP.Òàêîéâàðèàíòíåóïîìèíàåòñÿ

íè â îäíîé èç äðóãèõ ðàáîò [35℄. Îáúÿñíåíèåì âûñîêîé ïðîèçâîäèòåëüíîñòè ìîæåò áûòü

òî,÷òîïðèðàñêðàøèâàíèèïîïëîñêîñòÿìïðîèñõîäèòáîëüøåïåðåèñïîëüçîâàíèÿäàííûõâ

êýøåïðèîáðàùåíèè êïàìÿòè,ïîòîìó÷òîïîòîêèïîñëåäîâàòåëüíîïåðåáèðàþò ýëåìåíòû,

ñîîòâåòñòâóþùèå ñîñåäíèìòî÷êàìîäíîéïëîñêîñòè.

èñ. 1.àñêðàñêà îáëàñòèâ 8öâåòîâ

0 2 4 6 8 10 12

1 6 12 24

,

OpenMP

!

"#$% #% " # & '($)*+ (

% ,'% ,'"

8

^{- #+}

% ,'% ,'"

2

^{- #+}

(

^$

1

$&,',+ (

)

% ,'% ,'"

2

^{- #+}

(

^$

2

$&,',+ (

)

#'+ % (-( *

èñ. 2.Ïðîèçâîäèòåëüíîñòü îäíîóçëîâûõâàðèàíòîâîïòèìèçàöèèHPCG

èñ.3. àñêðàñêàîáëàñòè â2 öâåòà èñ. 4. àñêðàñêà îáëàñòè â 2 öâåòà ïî äâå

ïëîñêîñòè

2.3. Âåêòîðèçàöèÿ

Ñëåäóþùèì ýòàïîì îïòèìèçàöèé áûëà âåêòîðèçàöèÿ âû÷èñëåíèé. Ïîñêîëüêó âî âñåõ

ñòðîêàõ ìàòðèöû, êðîìå ñîîòâåòñòâóþùèõ ãðàíè÷íûì ýëåìåíòàì, ïî 27 íåíóëåâûõ çíà-

÷åíèé, òî èñïîëüçóåìûé ïî óìîë÷àíèþ îðìàò õðàíåíèÿ ðàçðåæåííîé ìàòðèöû ìîæíî

çàìåíèòüíà îðìàòELLPACK, âêîòîðîì êîëè÷åñòâî íåíóëåâûõýëåìåíòîââ ñòðîêåèê-

ñèðîâàíî, èýòîòàêòèñïîëüçóåòñÿäëÿ óïðîùåíèÿñòðóêòóðûäàííûõ.Âäàííîéðåàëèçà-

öèèäëÿîðìàòàELLPACKêîëè÷åñòâîíåíóëåâûõýëåìåíòîââñòðîêåâûáðàíîðàâíûì28,

÷òî òðåáóåòñÿ äëÿêðàòíîñòè äëèíû ìàññèâàýëåìåíòîâ ñòðîêèäëèíå âåêòîðíîãî ðåãèñòðà

ðàñøèðåíèÿñèñòåìû êîìàíäIntelAVX ïî 4çíà÷åíèÿ äâîéíîéòî÷íîñòèââåêòîðå.Ñìå-

íà îðìàòà ìàòðèöûïîçâîëèëàóëó÷øèòüâåêòîðèçàöèþ êîäà óíêöèéComputeSYMGS è

ComputeSPMV, ÷òîîáåñïå÷èëî ïðèðîñòïðîèçâîäèòåëüíîñòè 5%.

ÝåêòèâíàÿâåêòîðèçàöèÿêîäàóíêöèéComputeSYMGSèComputeSPMVçàòðóäíè-

òåëüíàèç-çàòîãî,÷òîîáðàùåíèÿêâåêòîðó

x

ÿâëÿþòñÿêîñâåííûìè,òîåñòüïðîèñõîäÿòïî

èíäåêñàì,ñîîòâåòñòâóþùèìíåíóëåâûìýëåìåíòàìñòðîêìàòðèöû.Îñíîâíîéàêòîð,ñäåð-

æèâàþùèéïðîèçâîäèòåëüíîñòüýòèõóíêöèéïðîïóñêíàÿñïîñîáíîñòü÷òåíèÿèçïàìÿòè

âðåæèìåìàëîé ïîëåçíîéäîëèäàííûõ,âûáèðàåìûõèçïàìÿòèïðèîäíîìîáðàùåíèè.Îä-

íàêî ïîëó÷åííûéïðèðîñòïðîèçâîäèòåëüíîñòè íå ÿâëÿåòñÿîêîí÷àòåëüíûìðåçóëüòàòîì.

2.4. Îïòèìèçàöèÿ ìåæóçëîâûõ êîììóíèêàöèé

Ïîñêîëüêóìàðøðóòèçàòîðûñåòè¾Àíãàðà¿ñîçäàâàëèñüñàïïàðàòíîéïîääåðæêîéáèá-

ëèîòåêèSHMEM [17℄, äëÿóâåëè÷åíèÿ ïðîèçâîäèòåëüíîñòèìåæóçëîâûõ êîììóíèêàöèé èñ-

ïîëüçîâàëàñü ýòà áèáëèîòåêà âìåñòî áèáëèîòåêè MPI [18℄, èñïîëüçóåìîé â áàçîâîé âåðñèè

HPCG.

 óíêöèè ExhangeHalo, êîòîðàÿ èñïîëüçóåòñÿ ïðè âûïîëíåíèè óíêöèé SYMGS è

SPMV äëÿ îáìåíà ãðàíè÷íûìè çíà÷åíèÿìè, ïîñûëêè çíà÷åíèé â îïòèìèçèðîâàííîé âåð-

ñèè îñóùåñòâëÿþòñÿ ïðè ïîìîùè óíêöèè çàïèñè äàííûõ íà óäàëåííûé óçåë áèáëèîòå-

êè SHMEM (shmem_double_put). Îäíàêî ïîñëå óäàëåííîé çàïèñè äàííûõ â áèáëèîòåêå

SHMEM íåîáõîäèìà ñèíõðîíèçàöèÿ. Ïåðâàÿ ðåàëèçàöèÿ èñïîëüçîâàëà îáùóþ áàðüåðíóþ

ñèíõðîíèçàöèþ: êàæäûé ïðîöåññ îæèäàë, êîãäà çàâåðøàòñÿ óäàëåííûå çàïèñè âñåõ ïðî-

öåññîâ ïðè ïîìîùè óíêöèè shmem_barrier, è òîëüêî ïîñëå ýòîãî íà÷èíàë âûïîëíåíèå

îñíîâíûõöèêëîâSYMGSèSPMV.Âîâòîðîìâàðèàíòåèñïîëüçîâàëàñüñèíõðîíèçàöèÿ,ðå-

àëèçîâàííàÿ ïðè ïîìîùè äîïîëíèòåëüíûõ óäàëåííûõçàïèñåé,íî òîëüêî ñ òåìè ïðîöåññà-

ìè,ñêîòîðûìèïðîèñõîäèòîáìåí.Êàæäûéïðîöåññíà÷èíàëâûïîëíÿòüîñíîâíûåèòåðàöèè

ïîñëåòîãî, êàêïîëó÷èò íåîáõîäèìûåäàííûå äëÿâû÷èñëåíèéîò ñîñåäíèõïðîöåññîâ.

0 50 100 150 200 250 300

0 5 10 15 20 25 30 35 40

. / 0 1 2 3

SHMEM

456789:

HPCG 2.4

èñ. 5. Ïðîèçâîäèòåëüíîñòü íà ðàñïðåäåëåííîé ïàìÿòè áàçîâîãî âàðèàíòà â ñðàâíåíèè ñ

îïòèìèçèðîâàííûìâàðèàíòîìñ èñïîëüçîâàíèåì áèáëèîòåêèSHMEM

Ñðàâíåíèåðåçóëüòàòîâðàáîòûêàæäîéðåàëèçàöèèïðåäñòàâëåíîíàðèñ.5.åçóëüòàòû

èçìåðÿëèñü íà36-óçëîâîìêëàñòåðåññåòüþ¾Àíãàðà¿,åãîïàðàìåòðûïðèâåäåíûâòàáëèöå

2. Íà êàæäîì èç óçëîâ ïðè çàïóñêå òåñòà HPCG èñïîëüçîâàëîñü 8 ïðîöåññîâ. åçóëüòà-

òû ïðîèçâîäèòåëüíîñòè ïðèâåäåíû äëÿðàçìåðà ëîêàëüíîéîáëàñòè ïðîöåññà 16x16x16.Íà

ðèñóíêå ïðèâåäåíûðåçóëüòàòû äëÿáàçîâîé âåðñèè2.4òåñòà è äëÿîïòèìèçèðîâàííîé âåð-

ñèè, âêëþ÷àþùåé îäíîóçëîâûå îïòèìèçàöèè (çà èñêëþ÷åíèåì OpenMP) è èñïîëüçîâàíèå

áèáëèîòåêè SHMEM.

Äèàãðàììà íà ðèñ. 5 ïîêàçûâàåò, ÷òî íà 36 óçëàõâûèãðûø â ïðîèçâîäèòåëüíîñòè îï-

òèìèçèðîâàííîé âåðñèèñ èñïîëüçîâàíèåìSHMEM ñîñòàâëÿåò39%.

Çàêëþ÷åíèå

ÄàííàÿðàáîòàïðåäñòàâëÿåòïåðâóþâåðñèþîïòèìèçèðîâàííîéðåàëèçàöèèòåñòàHPCG

äëÿñóïåðêîìïüþòåðîâññåòüþ¾Àíãàðà¿,ðàçðàáîòàííîéâÎÀξÍÈÖÝÂÒ¿.Ïðåäëàãàåìàÿ

ðåàëèçàöèÿ âêëþ÷àåò îïòèìèçàöèè óðîâíÿ âû÷èñëèòåëüíîãî óçëà, â òîì ÷èñëå èçìåíåíèå

îðìàòàõðàíåíèÿðàçðåæåííîéìàòðèöû,OpenMP-ðàñïàðàëëåëèâàíèå ìåòîäàñèììåòðè÷-

íîãîàóññà-Çåéäåëÿèâåêòîðèçàöèþêîäà.Íàîäíîìóçëåïåðå÷èñëåííûåïðèåìûïîçâîëèëè

ïîëó÷èòü óñêîðåíèå4.8ðàçàïî ñðàâíåíèþñ áàçîâîéâåðñèåé HPCG2.4.

Ìåæïðîöåññíûåîïòèìèçàöèèîáìåíîââñåòè¾Àíãàðà¿âûïîëíåíûïðèïîìîùèèñïîëü-

çîâàíèÿáèáëèîòåêè SHMEMâìåñòîáèáëèîòåêè MPI.Ïîëó÷åííûå âðàáîòåðåçóëüòàòûíà

ìàêåòíîì 36-óçëîâîì êëàñòåðå ñ ñåòüþ ¾Àíãàðà¿ ïîçâîëèëè ïîëó÷èòü âûèãðûø 39% ïî

ñðàâíåíèþ ñáàçîâîé âåðñèåé2.4òåñòàHPCG.

Äàëüíåéøàÿ ðàáîòà áóäåò íàïðàâëåíà íà îáúåäèíåíèå OpenMP-ðàñïàðàëëåëèâàíèÿ è

ìíîãîóçëîâûõ îïòèìèçàöèé, à òàêæå íà óëó÷øåíèå ðåçóëüòàòîâ è áîëåå ãëóáîêîå èññëåäî-

âàíèå.

Ëèòåðàòóðà

1. M. Heroux, J. Dongarra, P. Luszzek. HPCG Tehnial

Speiation. Sandia Report SAND2013-8752. Printed Otober 2013.

URL: https://software.sandia.gov/hpg/do/HPCG-Speifiation.pdf (äàòà îá-

ðàùåíèÿ: 10.06.2015).

2. J. Dongarra, P. Luszzek, A. Petitet. The LINPACK Benhmark: Past,

Present, and Future // Conurreny and Computation: Pratie and

Experiene. Vol. 15(9). 2003. P. 803820. ISSN 1532-0634.

URL:http://www.netlib.org/utk/people/JakDongarra/PAPERS/146_2003_the-linpak -b en hma rk-p ast -p res ent -and -f utu re. pd f

(äàòà îáðàùåíèÿ: 14.06.2015).

3. J. Park, M. Smelyanskiy, K. Vaidyanathan, et al. Eient shared-memory implementation

of high-performane onjugate gradient benhmark and its appliation to unstrutured

matriesi//InProeedingsoftheInternationalConfereneforHighPerformaneComputing,

Networking, Storage and Analysis (SC'14). IEEE Press, Pisataway, NJ, USA, 2014.

P. 945955. URL: http://pl.intel-researh.net/publiations/s14_hpg.pdf (äàòà

îáðàùåíèÿ: 12.06.2015).

4. C.Chen,Y.Du,H.Jiang,K.Zuo,C.Yang.HPCG:PreliminaryEvaluationandOptimization

on Tianhe-2 CPU-only Nodes // In Proeedings of the 2014 IEEE 26th International

Symposiumon Computer Arhiteture andHigh PerformaneComputing(SBAC-PAD'14).

IEEEComputer Soiety,Washington, DC,USA. 2014. P.4148.

5. K. Kumahata, K. Minami. HPCG Performane Improvement on the K

omputer // Presentation at Superomputers Conferene (SC'14), 2014.

URL: http://www.hpg-benhmark.org/downloads/s14/HPCG_on_the_K_omputer.pdf

(äàòà îáðàùåíèÿ: 14.06.2015).

6. E.Phillips,M.Fatia.ACUDAImplementationoftheHighPerformaneConjugateGradient

Benhmark//HighPerformaneComputingSystems.PerformaneModeling,Benhmarking,

and Simulation. Leture Notes in Computer Siene. Vol. 8966. 2015. P. 6884.

URL:http://www.ds.warwik.a.uk/~sdh/pmbs14/PMBS14/Workshop_Shedule_files/8-CU DAH PCG .pd f

(äàòà îáðàùåíèÿ: 12.06.2015).

7. V. Marjanovi, J. Graia, C. Glass. Performane Modeling of the HPCG Benhmark

// High Performane Computing Systems. Performane Modeling, Benhmarking, and

Simulation Leture Notes in Computer Siene. Vol. 8966. 2015. P. 172192.

URL:https://hpgmg.org/stati/MarjanoviGraiaGlass-PerformaneModelHPCG-2014. pd f

(äàòà îáðàùåíèÿ: 14.06.2015).

8. Ëàöèñ, À.Î. Êàêàÿ íàì ïîëüçà îò òåñòà HPCG. / Ëàöèñ, À.Î., Àíäðååâ Ñ.Ñ.,

Ïëîòêèíà Å.À., Äáàð Ñ.À. // Ñáîðíèê òåçèñîâ äîêëàäîâ ÍÑÊÔ'2014. 2014.

URL:http://2014.nsf.ru/TesisAll/4_Systemnoe_i_promezhytohnoe_PO/06_072_Latsi sAO .pd f

(äàòà îáðàùåíèÿ: 12.06.2015).

9. Êèñåëåâ À.Â., Êèñåëåâ Å.À., Êîðíååâ Â.Â. Àíàëèç ñòðóêòóðû èíîðìàöèîííîãî ãðà-

à òåñòà HPCG // Íàó÷íûé ñåðâèñ â ñåòè Èíòåðíåò: ìíîãîîáðàçèå ñóïåðêîì-

ïüþòåðíûõ ìèðîâ: Òðóäû Ìåæäóíàðîäíîé ñóïåðêîìïüþòåðíîé êîíåðåíöèè (22-

27 ñåíòÿáðÿ 2014 ã., ã. Íîâîðîññèéñê). Ì.: Èçä-âî ÌÓ, 2014. C. 4951.

URL:http://agora.guru.ru/abrau2014/pdf/049.pdf (äàòà îáðàùåíèÿ: 12.06.2015).

10. Ìàêàãîí, Ä.Â. Ñåòè äëÿ ñóïåðêîìïüþòåðîâ. / Ä.Â. Ìàêàãîí, Å.Ë. Ñûðîìÿòíèêîâ //

Îòêðûòûåñèñòåìû. ÑÓÁÄ.2011.  7. Ñ.3337.

11. Êîðæ, À.À. Îòå÷åñòâåííàÿ êîììóíèêàöèîííàÿ ñåòü 3D-òîð ñ ïîääåðæ-

êîé ãëîáàëüíî àäðåñóåìîé ïàìÿòè äëÿ ñóïåðêîìïüþòåðîâ òðàíñïåòàëîïñíî-

ãî óðîâíÿ ïðîèçâîäèòåëüíîñòè. / À.À. Êîðæ, Ä.Â. Ìàêàãîí, È.À. Æàáèí,

Å.Ë. Ñûðîìÿòíèêîâ // Ïàðàëåëüíûå âû÷èñëèòåëüíûå òåõíîëîãèè (ÏàÂÒ'2010):

Òðóäû ìåæäóíàðîäíîé íàó÷íîé êîíåðåíöèè (Óà, 29 ìàðòà 2 àïðåëÿ

2010 ã.). ×åëÿáèíñê: Èçäàòåëüñêèé öåíòð ÞÓðÓ, 2010. C. 227237.

URL: http://omega.sp.susu.a.ru/books/onferene/PaVT2010/full/134.pdf (äà-

òàîáðàùåíèÿ: 29.04.2015).

12. Ñèìîíîâ, À.Ñ. àçðàáîòêà ìåæóçëîâîé êîììóíèêàöèîííîé ñåòè ñ òîïîëîãèåé ¾ìíîãî-

ìåðíûé òîð¿ è ïîääåðæêîé ãëîáàëüíî àäðåñóåìîé ïàìÿòè äëÿ ïåðñïåêòèâíûõ îòå÷å-

ñòâåííûõ ñóïåðêîìïüþòåðîâ. / À.Ñ. Ñèìîíîâ, È.À. Æàáèí, Ä.Â. Ìàêàãîí // Íàó÷íî-

òåõíè÷åñêàÿ êîíåðåíöèÿ ¾Ïåðñïåêòèâíûåíàïðàâëåíèÿ ðàçâèòèÿâû÷èñëèòåëüíîéòåõ-

íèêè¿ (Ìîñêâà, 28èþíÿ). Ìîñêâà: ÎÀÎ ¾Êîíöåðí¾Âåãà¿,2011. C.1719.

13. Ñèìîíîâ,À.Ñ.Ïåðâîåïîêîëåíèåâûñîêîñêîðîñòíîéêîììóíèêàöèîííîéñåòè¾Àíãàðà¿/

À.Ñ.Ñèìîíîâ,Ä.Â.Ìàêàãîí,È.À.Æàáèí,À.Í.Ùåðáàê, Å.Ë.Ñûðîìÿòíèêîâ,Ä.À.Ïî-

ëÿêîâ// Íàóêîåìêèåòåõíîëîãèè. 2014. Ò.15, 1.Ñ. 2128.

14. Ñëóöêèí, À.È.àçðàáîòêàìåæóçëîâîé êîììóíèêàöèîííîé ñåòèÅÑ8430 ¾Àíãàðà¿ äëÿ

ïåðñïåêòèâíûõñóïåðêîìïüþòåðîâ/À.È.Ñëóöêèí,À.Ñ.Ñèìîíîâ,È.À.Æàáèí,Ä.Â.Ìà-

êàãîí, Å.Ë. Ñûðîìÿòíèêîâ // Óñïåõè ñîâðåìåííîé ðàäèîýëåêòðîíèêè. 2012.  1.

C.610.

15. Ìàêàãîí, Ä.Â. Êðèñòàëë äëÿ Àíãàðû / È.À. Æàáèí, Ä.Â. Ìàêàãîí, À.Ñ. Ñèìîíîâ //

Ñóïåðêîìïüþòåðû. Çèìà-2013. C.4649.

16. ÈçãàëèíÑ.Ï., ÑèìîíîâÀ.Ñ.,ÌèõååâÂ.À.Îòå÷åñòâåííûåðàçðàáîòêè âîáëàñòèñóïåð-

êîìïüþòåðíûõ òåõíîëîãèé. Ïðåäâàðèòåëüíûå ðåçóëüòàòû îöåíêè ïðîèçâîäèòåëüíîñòè

ÂÊѾÀíãàðà¿ (ÅÑ8430) // Äîêëàä íà ÌîñêîâñêîìÑóïåðêîìïüþòåðíîì Ôîðóìå, 2014.

URL:http://www.osp.ru/netat_files/18/10/07_Otehestvennye_razrabotki_v_oblast i_s upe rko mp_ yu ter nyh _te hno lo giy .pd f

(äàòà îáðàùåíèÿ: 14.06.2015).

17. Feind, K. Shared Memory Aess (SHMEM) Routines. Cray Researh, 1995.

URL:https://ug.org/5-publiations/proeedings_attendee_lists/1997CD/S95PROC/ 303 _30 8.P DF

(äàòà îáðàùåíèÿ: 29.04.2015).

18. Message Passing Interfae Forum, MPI: A Message-Passing Interfae Standard, 1995.

URL: http://www.mpi-forum.org/dos/mpi-1.1/mpi-11-html/node64.html (äàòà îáðà-

ùåíèÿ: 29.04.2015).

Optimized implementation of HPCG benchmark on supercomputer with “Angara” interconnect

Alexander Agarkov, Alexander Semenov and Alexey Simonov

Keywords: HPCG, "Angara" interconnect, SHMEM, vectorization, OpenMP

The paper presents the optimized implementation of the HPCG benchmark on "Angara"

interconnect developed by "NICEVT". HPCG implements conjugate gradient method with preconditioner to solve a sparse linear system. The proposed implementation includes compute node optimizations, such as new sparse matrix storage format, OpenMP- parallelization and vectorization, as well as interprocess communication optimizations targeted for "Angara" interconnect by using the SHMEM library instead of MPI. Compute node optimization includes changing the sparse matrix storage format, OpenMP- parallelization and vectorization. Obtained performance results on the cluster with "Angara"

interconnect significantly exceed the results of the HPCG 2.4 reference implementation.