TME, Data Center Switching

(1)

Design and Architecture of Data Center Networking Platforms

Lincoln Dale <ltd@cisco.com>

TME, Data Center Switching

(2)

Agenda

Part I—Design Considerations

Part II—Architectures

Part III—Real World Implementation

Part IV—The Future

(3)

Part I – Design

Considerations

(4)

Part I—Design Considerations

Many Factors to Weigh

Buffering

Multiprotocol

One Plus One Is Not Necessarily Equal to Two

Future (Investment Protection)

(5)

Many Factors to Weigh

Standards requirements

Market requirements

Designability

Silicon technology

Processor technology

Applicable to Any Switch/Router Design

Product is chassis and network

Manufacturability

Time to market

Flexibility

Budget

(6)

Many Factors to Weigh

Buffering

No packet drop

Throughput

Port count

Modular

No single point of failure

In-order delivery

Future protocol

Data Plane

Modular

Restartable (including

active-active state handling)

Non-disruptive code load &

activation

No single point of failure

Scaleable

Unit Testable

Future protocol compatibility

Control plane

Baseline Data Center Switch requirements

(7)

Part I—Design Considerations

Many Factors to Weigh

Buffering

Multiprotocol

One Plus One Is Not Necessarily Equal to Two

Future (Investment Protection)

(8)

In A

In B

Out C

Out D

Switch Switch

Buffering

A-C

Output Output

Input Input

(9)

Out C

Out D

Switch Switch

Output Output

Input Input

In A

In B

Switch Switch

Buffering

A-C

(10)

In A

In B

Out C

Out D

Buffering

A-C

Switch Switch

Output Output

Input Input

(11)

In A

In B

Out C

Out D

Buffering

A-C

B-C

Switch Switch

Output Output

Input Input

(12)

In A

In B

Out C

Out D

Switch Switch

Output Output

Input Input

Buffering

X X

A-C

B-C

(13)

Buffering

In A A-C

In B

Out C

Out D

Switch Switch

Output Output

Input Input

(14)

Out C

Out D Out C

Out D In A

In B

Switch Switch

Output Output

Input Input

Switch Switch

Buffering

A-C

B-C

(15)

Out C

Out D

Switch Switch

Output Output

Input Input

In A

In B

Buffering

A-C

B-C

(16)

Out C

Out D

Switch Switch

Output Output

Input Input

In A

In B

Buffering

A-C

B-C

(17)

Buffering

A-C B-C In A

In B

Out C

Out D

Switch Switch

Output Output

Input Input

(18)

Part I—Design Considerations

Many Factors to Weigh

Buffering

Multiprotocol

One Plus One Is Not Necessarily Equal to Two

Future (Investment Protection)

(19)

Multi Protocol

Protocol Agnostic

Protocol Translation

Drop Behavior

Shared Uplinks

Multiple Topologies

IP/Ethernet

802.11 “WiFi”

802.16 “WiMax”

FDDI

Vines

XNS

AppleTalk

IPX/SPX

Netbeui

DECnet

OSI/ISO

ArcNet

10Base{2,5,T} / 100Base{S,T,TX} / 1000BaseT

10GbE

Fibre Channel

FICON

ESCON

Token Ring

IP & TCP

IPv4/IPv6

802.1q VLANs

MPLS /

MPLS {P,PE} / MPLE {CE,..}, ..

VPLS

q-in-q VLAN encapsulation

802.2 LLC/SNAP

ATM

Frame Relay

DOCSIS

(20)

Part I—Design Considerations

Many Factors to Weigh

Buffering

Multiprotocol

One Plus One Is Not Necessarily Equal to Two

Future (Investment Protection)

(21)

One Plus One Does Not Equal Two

Switch A Switch A Switch BSwitch B

10 Links @ 1Gbps Each Bandwidth = 10Gbps Flow Bandwidth = 1Gbps Serialization Delay = 20uS

1 Link @ 10Gbps Each Bandwidth = 10Gbps Flow Bandwidth = 10Gbps

Serialization Delay = 2uS

Switch A Switch A Switch B Switch B

(22)

Part II –

Architectures

(23)

Part II—Architectures

Centralized Memory

Switch on a Chip / “System on Chip” (SoC)

Bus

Mesh

Centralized memory in a mesh

Two-Tier / Banyan / Clos

Centralized memory in two tiers

Cross-Bar

(24)

Part II—Architectures

Centralized Memory

Switch on a chip / “System on Chip” (SoC)

Bus

Mesh

Centralized memory in a mesh

Two-Tier / Banyan / Clos

Centralized memory in two tiers

Cross-Bar

(25)

Centralized Memory

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Scalability limited by memory bandwidth/size

Typically optimized for fixed configuration—

inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

(26)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(27)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(28)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(29)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(30)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

3

2

1 Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(31)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

3

2

1 Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(32)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

3

2

1 Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(33)

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Centralized Memory

3 Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(34)

Centralized Memory

Input 3 Input 2 Input 1 Output 3 Output 2 Output 1

Scalability limited by memory bandwidth/size

Typically optimized for

fixed configuration— inflexible

Port count 8–32

Cost effective with small port counts

Often used as building block

Almost required for Blade

Server Applications

(35)

Part II—Architectures

Centralized Memory

“Switch on a chip” / System on Chip (SoC)

Bus

Mesh

Centralized memory in a mesh

Two-Tier / Banyan / Clos

Centralized memory in two tiers

Cross-Bar

(36)

Bus

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

3

2

1

(37)

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

Bus

3 2

1

(38)

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

Bus

X X

3 2

1

(39)

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

X X

Bus

X X

3

2

1

(40)

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

X X

Bus

X X

3

2

1

(41)

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

Bus

3

2

1

(42)

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

Bus

3

2

1

(43)

Bus

Input 1 Input 2 Input 3 Out put 1 Out put 2 Out put 3

(44)

Bus with centralized memory

In 1 In 2 In 3 In 4 In 5 In 6 In 7 In 8

In 1 In 2 In 3 In 4 In 5 In 6 In 7

Out 1 Out 2 Out 3 Out 4 Out 5 Out 6 Out 7 Out 8

Out 1

Out 2

Out 3

Out 4

Out 5

Out 6

Out 7

(45)

Part II—Architectures

Centralized Memory

Switch on Chip / “System on Chip” (SoC)

Bus

Mesh

Centralized memory in a mesh

Two-Tier / Banyan / Clos

Centralized memory in two tiers

Cross-Bar

(46)

Mesh

16- 16 -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory 16- 16 -Port Port

Central Central Memory Memory

16 16- -Port Port Central Central Memory Memory

13 External Ports

Total 52 External Ports

13:1 Blocking Factor

(47)

16- 16 -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory 16- 16 -Port Port

Central Central Memory Memory

16 16- -Port Port Central Central Memory Memory

Total 52 External Ports

Mesh

Input A Output B

(48)

16- 16 -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory 16- 16 -Port Port

Central Central Memory Memory

16 16- -Port Port Central Central Memory Memory

Total 52 External Ports

Input A1 Output B1’

Mesh

13:1 Over-Subscription

Input A2 Input A13

Output B2’

Output B13’

(49)

16- 16 -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory 16- 16 -Port Port

Central Central Memory Memory

16 16- -Port Port Central Central Memory Memory

10 External Ports

Total 40 External Ports

Mesh

(50)

16- 16 -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory 16

16- -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory

Total 40 External Ports

Mesh

Input A1 Output B1’

10:2 Over-Subscription

Input A2 Input A10

Output B2’

Output B10’

(51)

16- 16 -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory 16- 16 -Port Port

Central Central Memory Memory

16 16- -Port Port Central Central Memory Memory

Total 16 External Ports Non-Blocking

Mesh

4 External Ports

4 Links

(52)

Mesh

SAME

Total 16 External Ports Non-Blocking

16- 16 -Port Port Central Central Memory Memory

16- 16 -Port Port Central Central Memory Memory 16- 16 -Port Port

Central Central Memory Memory

16- 16 -Port Port Central Central Memory

Memory 4 Links

16- 16 -Port Port

Central

Memory

(53)

Mesh-Based Switch

16 Port Switch

Switch Switch Switch Switch

16 Port Switch 16 Port Switch 16 Port Switch 16 Port Switch

16 Port Switch

16 Port Switch 16 Port Switch

16 Port Switch

(54)

16 Port Switch

Mesh-Based Switch

(55)

16 Port Switch

Mesh-Based Switch

(56)

16 Port Switch

Mesh-Based Switch

A

B

(57)

16 Port Switch

Mesh-Based Switch

A

B C

D

(58)

16 Port Switch

A

B C

D E

F

Mesh Based Switch

2:1 Over-Subscription 2:1 Over-Subscription

G

H

(59)

16 Port Switch

Mesh-Based Switch

A

B

(60)

16 Port Switch

Mesh-Based Switch

A C

B D

2:1 Over-Subscription

(61)

16 Port Switch

A

B C

D

Mesh-Based Switch

E

F G

H

4:1 Over-Subscription

(62)

16 Port Switch

A

B C

D

Mesh-Based Switch

E

F G

H

4:1 Over-Subscription 4:1 Over-Subscription

A

B C

D E

F G

2:1 Over-Subscription 2:1 Over-Subscription

H A C B D

1:1 Non-Blocking

(63)

Part II—Architectures

Centralized Memory

Switch on Chip / “System on Chip” (SoC)

Bus

Mesh

Centralized memory in a mesh

Two-Tier / Banyan / Clos

Centralized memory in two tiers

Cross-Bar

(64)

Two-Tier a.k.a. Banyan a.k.a. Clos

4 Port SW 4 Port

SW

4 Port SW 4 Port

SW

4 Port SW 4 Port

SW

4 Port SW 4 Port

SW 4 Port

SW

4 Port SW 4 Port

SW

(65)

Two-Tier

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

8 Port SW 8 Port

SW

(66)

Two-Tier

4 Port SW 4 Port

SW

4 Port SW 4 Port

SW

4 Port SW 4 Port

SW

4 Port SW 4 Port

SW 4 Port

SW

4 Port SW 4 Port

SW

(67)

Two-Tier

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

(68)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

(69)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

(70)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

Blocking

(71)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

(72)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

(73)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

2

(74)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

2

(75)

In 1 In 2

In 3 In 4

Out 1 Out 2

Out 3 Out 4

4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW 4 Port SW

Two-Tier

2

(76)

Two-Tier-Based Switch—64 Ports

(77)

Two-Tier-Based Switch—128 Ports

(78)

Part II—Architectures

Centralized Memory

Switch on Chip / “System on Chip” (SoC)

Bus

Mesh

Centralized memory in a mesh

Two-Tier / Banyan / Clos

Centralized memory in two tiers

Cross-Bar

(79)

Cross-Bar

3 2 1

Input 1 Out put 1 Out put 2 Out put 3

Input 2 Input 3

Cross-Bar

(80)

Input 1 Out put 1 Out put 2 Out put 3

Input 2 Input 3

Cross-Bar

3

2

1

(81)

3 2 1

Cross-Bar

Input 1 Out put 1 Out put 2 Out put 3

Input 2 Input 3

Cross-Bar

(82)

Cross-Bar

Input 1 Out put 1 Out put 2 Out put 3

Input 2 Input 3

Cross-Bar

(83)

Cross-Bar

In 1 In 2 In 3 In 4 In 5 In 6 In 7 In 8

In 1 In 2 In 3 In 4 In 5 In 6 In 7

Out 1 Out 2 Out 3 Out 4 Out 5 Out 6 Out 7 Out 8

Out 1 Out 2 Out 3 Out 4 Out 5 Out 6 Out 7

Cross-Bar

(84)

Cross-Bar

In 1 In 2 In 3 In 4 In 5 In 6 In 7 In 8

In 1 In 2 In 3 In 4 In 5 In 6 In 7

Out 1 Out 2 Out 3 Out 4 Out 5 Out 6 Out 7 Out 8

Out 1 Out 2 Out 3 Out 4 Out 5 Out 6 Out 7

Cross-Bar

(85)

Cross-Bar

In j

Out k

Switch

(86)

Cross-Bar

In 1

In 2

In 3

In n In 4

Out 1 Out 2 Out 3 Out 4 Out n

(87)

In 1

In 2

In 3

In n In 4

Out 1 Out 2 Out 3 Out 4 Out n

Cross-Bar

3 2 1

4 n

(88)

Advantages of Cross-Bar

Scalable

Multi-speed capable

Non-blocking

Efficient

Cross-Bar Cross-Bar

Line Card

Line Card Line Card Line Card Line Card Line Card Line Card Line Card Line Card Line Card

(89)

Part III – Real World

Implementation

(90)

Part III—Real World Implementation

Cross-bar Performance

Classic crossbar vs. Buffered Crossbar

Virtual Output Queuing

Head-of-Line Blocking

Completing the Switch

Other Features

(91)

Part III—Real World Implementation

Cross-bar Performance

Classic crossbar vs. Buffered Crossbar

Virtual Output Queuing

Head-of-Line Blocking

Completing the Switch

Other Features

(92)

Cross-Bar Model

(93)

Performance Issue with Cross-Bars

Source: M. J. Karol, M.G. Hluchyj, S. P. Morgan, “Input Versus Output Queueing [sic] on

58.6%

(94)

Cat6K & MDS Family Cross-Bar

Classic Cross-Bar

2.2Tbps Classic Cross-Bar

2.2Tbps

FIFO 1 FIFO 1 FIFO 2 FIFO 2 FIFO 3 FIFO 3

FIFO 18 FIFO 18

20/23.07 Gbps Each

60/69.21 Gbps Each

FIFO 1 FIFO 1 FIFO 2 FIFO 2 FIFO 3 FIFO 3

FIFO 18 FIFO 18

60/69.21 Gbps Each

20/23.07 Gbps Each

Single

(95)

Part III—Real World Implementation

Cross-bar Performance

Classic crossbar vs. Buffered Crossbar

Virtual Output Queuing

Head-of-Line Blocking

Completing the Switch

Other Features

(96)

Buffering—Input Buffering

(97)

Head of Line Blocking

Stadium

Beer/Soda/Chips

(98)

Head of Line Blocking

Stadium

Beer/Soda/Chips

(99)

Stadium

Output Queuing

Beer/Soda/Chips

(100)

Head of Line Blocking

A

B

C

B

(101)

Head of Line Blocking

A

B

C

(102)

Head of Line Blocking

A

B

C

A

(103)

Head of Line Blocking

A

B

C A

B

? C

(104)

Virtual Output Queues

A

B

C

ARB

B F

(105)

Virtual Output Queues

A

B

C

ARB

C F

(106)

A A C B A

Virtual Output Queues

A

B

C

ARB

F

(107)

Virtual Output Queues

F

F F Input Port 1

Input Port 2

Input Port 3

Output Port 1

Avoids Head of line Blocking

Output Port 2

Output Port 3

(108)

Virtual Output Queues

F

F F

Larger Effective Buffers

Input Port 1

Input Port 2

Input Port 3

Output Port 1

Output Port 2

Output Port 3

(109)

Virtual Output Queues

F

F F

Better Isolation of Congestion Sources

Input Port 1

Input Port 2

Input Port 3

Output Port 1

Output Port 2

Output Port 3

(110)

Virtual Output Queues

F

F F

Better Isolation of Congested Outputs

Input Port 1

Input Port 2

Input Port 3

Output Port 1

Output Port 2

Output Port 3

(111)

Virtual Output Queues

F

F F

Better Isolation of Congestion In General

Input Port 1

Input Port 2

Input Port 3

Output Port 1

Output Port 2

Output Port 3

(112)

Virtual Output Queues

F

F F Input Port 1

Input Port 2

Input Port 3

Output Port 1

Output Port 2

Output Port 3

Buffer Bandwidth Matched to Port Bandwidth

(113)

Virtual Output Queuing - Summary

PRO

Avoids Head-of-line blocking

More efficient buffer usage

Better traffic isolation

Buffer bandwidth matched to port bandwidth

Necessary for lossless networks.

(e.g. Fibre Channel,

CON

Complex to implement

Locks in scalability

(114)

Part III—Real World Implementation

Cross-bar Performance

Classic crossbar vs. Buffered Crossbar

Virtual Output Queuing

Head-of-Line Blocking

Completing the Switch

Other Features

(115)

Completing the Swtich

FWD

ARB

(116)

Completing the Swtich

ARB

FWD MAC

MAC

FWD MAC

MAC

FWD MAC

MAC

MAC MAC

MAC

(117)

Completing the Swtich

Port

ARB

FWD MAC

MAC

FWD MAC

MAC

FWD MAC

MAC

MAC MAC

MAC

(118)

Re-Draw

MAC

MAC MAC MAC MAC MAC

ARB

FWD FWD FWD

(119)

Larger Switch

Flash Eth

Card Console

Cross-Bar Supervisor/Fabric

Back Plane

Arb

I/F uP

M M M

F Q

I/F uP

M M M

F Q

I/F uP

M M M

F Q

I/F uP

M M M

F Q

F

Q

uP

(120)

Larger Switch

MAC Port Speeds:

• 1 / 2 / 4 / 10Gbps Port Type / Modes:

• F_Port,

• Loop port (FL_Port),

• Private Loop (TL_Port),

• ISL (E_Port),

• EISL (TE_Port),

• SPAN Port (SD_Port)

PHY / MAC Forwarding

Queuing

PHY

Optical to Electrical interface:

• SFP (1 / 2 / 4 Gbps)

• X2 (10Gbps) optics

• Error-Check ingress frame (CRC etc)

• Ingress rate-limiting

• Prepend switch-internal header (Ingress port, Ingress VSAN, QoS bits)

• Timestamp ingress frame arrival time

• Transmit, loopback or Expire frame based on ingress timestamp

• Remove switch-internal header from frame

• Prepend EISL header onto frame where egress port is a Trunked E_Port (TE_Port)

• Per-VSAN Ingress ACLs (Hard Zoning)

• Per-VSAN Forwarding (Fibre Channel, MPLS)

• Load-balancing decision (determine final egress interface where destination is a PortChannel or equal-cost FSPF route)

• Inter VSAN Routing (Fibre Channel NAT)

• Update src/dst / frame/byte statistics

• Virtual Output Queuing

• Frame Buffering

• Quality of Service (QoS)

• Arbitration and Fairness

• Congestion Control (FCC)

• Per-VSAN Egress ACLs (Hard Zoning) including LUN Zoning & read-only Zones

• Re-Validate frame CRC

• Inter VSAN Routing (Fibre Channel NAT)

• Quality of Service (egress QoS)

Central Arbiter Crossbar Switch Fabric Crossbar

Switch Fabric Central Arbiter

Dual Redundant Centralized Arbiters

(capable of scheduling more than 5 billion frames per second)

• Dual Redundant crossbar switch fabrics provide high-speed non-blocking non-oversubscribed capacity from any slot to any slot (25 Gbps raw

capacity per channel, 4 channels per linecard)

• Crossbar operates 3X over-speed (75 Gbps) and

utilizes superframing to improves crossbar efficiency

(121)

Part III—Real World Implementation

Cross-bar Performance

Classic crossbar vs. Buffered Crossbar

Virtual Output Queuing

Head-of-Line Blocking

Completing the Switch

Other Features

(122)

Other Features

Bridging and Routing

SVIs

Multicast

Broadcast

ACLs (+VACLs)

SPAN + RSPAN

Security (e.g. 802.1x)

Redundancy

L4-7 Deep Packet Inspection

NAT

MPLS

*PLS

GRE

Virtualization

Virtual Router Instances

Services /

Service Modules

Roles / RBAC

Diagnostics

(123)

Part IV – The Future

(124)

Future Data Center considerations

Power & Heat becoming the primary consideration

Density of servers rapidly shrinking;

Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure

Moore’s Law

Today’s Commodity CPUs were yesterday’s Supercomputers

Virtualization

Server, Network, Storage, …

Future Data Center Network

At most, dual server connections Higher Speeds, Lower Latency

100 Mbps GbE 10GbE 100+GbE?

(125)

Future Data Center considerations

Power & Heat becoming the primary consideration

Density of servers rapidly shrinking;

Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure

Moore’s Law

Today’s Commodity CPUs were yesterday’s Supercomputers

Virtualization

Server, Network, Storage, …

Future Data Center Network

At most, dual server connections Higher Speeds, Lower Latency

100 Mbps GbE 10GbE 100+GbE?

(126)

Data Center Physical Infrastructure

Desire is to Last 10 to 15 Years

Useful life of a Data Center facility is expected to exceed the life of the gear it contains due to the tremendous cost of relocating or renovating data center facilities

Yr 5 Yr 10 Yr 15

2 to 3 Network Infrastructure Upgrades 5 to 7 Server Upgrade Cycles

Mainframes Æ 1/2RU Servers Æ Multi-Core CPU’s Æ Blade Server

10/100Æ10/100/1000 with L4-7 ServicesÆ 10GbE Attached hosts

(127)

Planning and Coordination Between

IT Staff and Facilities Managers is a Must

Obtaining 10-15 Years of Useful Life Requires Planning for Growth of Power and Cooling

Yr 5 Yr 10 Yr 15

2 to 3 Network Infrastructure Upgrades 5 to 7 Server Upgrade Cycles

Mainframes Æ 1/2RU Servers Æ Multi-Core CPUs Æ Blade Server

10/100 Æ 10/100/1000 with L4-7 Services Æ 10GbE Attached hosts Converged LAN/SAN/IPC

2kW/Rack Æ 6kW/Rack Æ 15+kW/Rack!

(128)

Future Data Center considerations

Power & Heat becoming the primary consideration

Density of servers rapidly shrinking;

Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure

Moore’s Law

Today’s Commodity CPUs were yesterday’s Supercomputers

Virtualization

Server, Network, Storage, …

Future Data Center Network

At most, dual server connections Higher Speeds, Lower Latency

100 Mbps GbE 10GbE 100+GbE?

(129)

1978: 8088: 20K transistors

1985: Intel® 386™: 275K transistors

1989: Intel® 486™: 1.2M transistors

1993: Intel® Pentium™: 3.1M transistors

1999: Intel® Pentium 3™: 9.5M transistors

2000: Intel® Pentium 4™: 42M transistors

2006: Intel® Core™2 Duo: 291M transistors

Moore’s Law

An empirical observation

“The transistor density of Integrated Circuits, with respect to minimum component cost, will double every 24 months.”

― Gordon E Moore, co-founder, Intel, 1965

(130)

Compute Capacity is Increasing

Network Challenges & Implications

Top 500 Super Computer (reported)

0 1 2 3 4 5 6

lo g ( g ig a flo p s )

#1

#500

Growth is not slowing down

More effective way to scale layer 2 networks is needed

Ready access to compute AND storage resources

Powering 10GbE over Copper PHYs on high density linecards (40+ ports)

Impact of an outage/downtime will be magnified- maximize operational uptime and

cluster utilization

Least disruptive upgrade path – hardware, software

& services

(131)

Future Data Center considerations

Power & Heat becoming the primary consideration

Size of servers rapidly shrinking;

Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure

Moore’s Law

Today’s Commodity CPUs were yesterday’s Supercomputers

Virtualization

Server, Network, Storage, …

Future Data Center Network

At most, dual server connections Higher Speeds, Lower Latency

100 Mbps GbE 10GbE 100+GbE?

(132)

VIRTUALIZATION

The capability to decouple logical functions from physical devices.

Increase resource usability – Lower CAPEX

Loosely couple re-usable functions – Flexibility

Dynamic allocation of virtual instances – Automation

Centralized policy Management – Lower TCO

Distributed Capabilities –

Broad use of functions

(133)

Data Center Virtualization Examples

Type Example Benefit

Pooling

802.3ad, Port Channel TCP Connection

Pooling

Symmetric Multi- processing

Oracle Cache Fusion

Improved availability Cost effective scaling Simplify topologies and configuration

TME, Data Center Switching

Design and Architecture of Data Center Networking Platforms

Lincoln Dale <ltd@cisco.com>