Deep Reinforcement learning-based Network slicing

HAL Id: hal-03135189

https://hal.inria.fr/hal-03135189

Submitted on 10 Feb 2021

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Deep Reinforcement learning-based Network slicing

Yassine Hadjadj-Aoul, Abdelkader Outtagarts

To cite this version:

Yassine Hadjadj-Aoul, Abdelkader Outtagarts. Deep Reinforcement learning-based Network slicing.

Bell Labs Spring Webinar, Jul 2020, Virtual, France. �hal-03135189�

DEEP REINFORCEMENT LEARNING-BASED NETWORK SLICING

Yassine Hadjadj-Aoul, and Abdelkader Outtagarts

Inria – Nokia Bell Labs common lab research action

« Analytics and machine learning for dynamic management of mobile virtualised network and service optimisation »

TEAM WORKING ON SERVICE PLACEMENT

INRIA

- Yassine Hadjadj-Aoul (Associate prof.) - Gerardo Rubino (Research director) - Eric Rutten (Researcher)

- Anouar Rkhami (Phd student)

- Pham Tran Anh Quang (former Post-doc)

Nokia

- Abdelkader Outtagarts (Senior

researcher)

PLAN

­Introduction

­Challenges related to network slicing

­On using Deep Reinfocement Learning for network slicing

­Conclusions

KEY CONCERNS FOR NETWORK OPERATORS

The ability to support current, emerging and future services

­ Examples: Automotive, mMTC, IoT/Factory automation, e-health

Ability to lease infrastructure to third parties without compromising the network and its efficiency

­ Consequence: Multiple stakeholders in a network

Automation is big deal for Network Operators

­ Objective: Zero touch networks

WHY ARE OPERATORS INTERESTED IN NETWORK SLICING?

Network slicing is seen as one of

the most important technologies to

meet Network Operators’

expectations ^Phys

ical inf ras tru ctu re Inte Mo rne He bil tof alt e br hca Thi oadband re ngs slic slic e e

slic e

Edge cloud Core cloud

Access node

Transport/aggregation node Core node

access

2G/3G/4G/

5G access

Satellite access

xDSL/Cable access

Intra-domain deployment of a slice

Isolated

slices

IN PRACTICE, WHAT DOES SLICING A NETWORK CONSIST OF?

Placement of services in a VNF-FG form

­ Involves not only the placement of VNFs but also addressing a routing problem

­ Addressing the VNFs placement then the routing … or simultanuously

­ Need to consider several metrics (QoS requirements)

­ Intra-domain or multi-domain placement

Extremely large number of possibilities of placement (very large action space)

­ Difficulty in finding an optimal placement,

excepting for very small instances (i.e. NP-hard

problem) Substrate network

Service (e.g. VNF-FG) State

Action

(VNF-FG placement)

LIMITS OF EXISTING WORK

­ Optimization problems (i.e. MILP) have the limitation of not always being

applicable in a real context, given the latency of resolution or unsuitability in a real context

­ Meta-heuristics are slow, require a realistic simulation environment … and there is no learning

­ Heuristics are very fast but present some difficulties in finding good solutions (i.e.

stuck on local minimums)

­ AI-based approaches: as learning progresses, performance becomes more and more important.

­ Safety problem

Objective

Exploring the potential of deep learning for safe placement of VNF-FG

APPLICATION OF DRL FOR VNF-FG PLACEMENT

­ The vanilla DDPG is not suitable for very large-scale discrete action

space

­ In VNF-FG embedding problem, there are constraints of resources such that some discrete actions are not feasible

Proposing to add to DDPG a Heuristic Fitting Algorithm (HFA)

0 50 100 150 200

Episodes

0 20 40 60 80 100

Acceptance Ratio %

DDPG

DDPG-HFA

DDPG-HFA

!"#$%&

'(#)$%*

+&

,%-#-"&

'(#)$%*

. /$00&123"#-$3

0 0 45 40

637-%$38(3#

(90 45 4% 40 :

! ;+ 90 <=:>'

?-3-@5#"A 0

B(2%-0#-"&

1-##-3C&

!DC$%#-A8

9E:

9G: 9F:

9H:

9I:

9I:

9H: 9H: 9J:

9K:

9L:

%

6MN(%-(3"(&O$$D

ADVANTAGE

FFPG-HFA allows improving the reward

­ Convergence to classical heuristic performance is immediate

­ No random behavior at the begining of the learning

Learning is slow, but once learned, execution is immediate The proposed solution allows

­ Improves the heuristic performance

­ Makes the placement safer …

WHY GO DEEP?

0 20 40 60 80 100

Episodes

0 20 40 60 80

Accpetance ratio % K=2 K=4 K=6 K=8 K=10

Impact of the

number of fully

connected Layers

IMPROVING EXPLORATION WITH EEDDPG

­ Noise is added to the proto-action to create H noisy actions.

­ Each proto action is fed into HFA in order to determine the feasible allocation of VNFs.

Evaluation:

­ Multi-critic

­ Different nets due to the arbitrary initialization

­ The best action identified (best Q value) by the evaluator will be executed

Updates of actor:

­ Best critic network

!

"#$%!&

'($)%!*

+

,!-$-#&

'($)%!*&.

/%00&123#$-%3

0 0 45 40

637-!%38(3$

(90 45 4! 40 :

" ;+ 90 <=:

6>?(!-(3#(&@%%A B-3-C5$#D 63D53#(E&

6>?A%!5$-%3

0 F(2!-0$-#&

1-$$-3G&

"AG%!$-D8

9H:

9I:

9J:

9K:

9L: 9L:

9K: 9K: 9HM:

9HH:

9N:

9O: 9P:

,!-$-#&

'($)%!*&.

,!-$-#&

'($)%!*&.

!" # $ %# & '(

"#$-%3&@%%A

9Q:

RESULTS ON THE APPLICATION OF EEDDPG

0 50 100 150 200

Episodes

60 70 80 90 100

Acceptance rate %

FF-Dijkstra DDPG-HFA _E ² _D ² _PG

IMPACT OF THE NUMBER OF CRITIC NETS

ADVANTAGE

Improving exploration allows to discover better action

­ which is very efficient in congested systems

With learning techniques, there is sometimes a degradation of performance

­ the use of several CRITIC networks brings more stability

The complexity is not necessarily much more important

­ the back-propagation is only applied on one CRITIC network, the best

ranked one

EVOLUTIONARY EEDDPG

WF

A: Weighted First Fit Algorithm – WF2A

ADVANTAGE

Discover new and much more interesting states/actions

­ Effective in highly congested systems

CONCLUSIONS

DRL are very efficient in addressing the targeted issue Lack of explainability

­ But combining a heuristic with DRL allows having some guarantees in the placement quality

Efforts are still needed …

­ Ongoing work on using Graph neural nets for services placement

­ Combining control theory and DRL for safer placement strategies.

REFERENCES

1. Pham Tran Anh Quang, Yassine Hadjadj-Aoul, and Abdelkader Outtagarts : “A deep reinforcement learning approach for VNF Forwarding Graph Embedding”. In IEEE, Transactions on Network and Service Management (TNSM) (October 2019)

2. Anouar Rkhami, Pham Tran Anh Quang, Yassine Hadjadj-Aoul, Abdelkader Outtagarts, and Gerardo Rubino : “On the Use of Graph Neural Networks for Virtual Network Embedding”. In proc. of IEEE ISNCC, the 7th International Symposium on Networks, Computers and Communications (ISNCC), Montreal, Canada (October 2020)

3. Pham Tran Anh Quang, Yassine Hadjadj-Aoul, and Abdelkader Outtagarts : “Evolutionary Actor- Multi-Critic Model for VNF-FG Embedding”. In proc. of IEEE CCNC, the 17th Annual IEEE Consumer Communications & Networking, Las Vegas, USA (January 2020)

4. Pham Tran Anh Quang, Yassine Hadjadj-Aoul, and Abdelkader Outtagarts: “ Deep

Reinforcement Learning based QoS-aware Routing in Knowledge-defined networking ”. In proc. of EAI QShine 2018, the 14th EAI International Conference on Heterogeneous

Networking for Quality, Reliability, Security and Robustness 2018, Ho Chi Minh City,

Vietnam (December 2018)