MRP in the Context of the ISO OSI Reference Model

MRPfits into a layered protocol scheme, such as the ISO OSI Reference Model [DZ83] at the data-link layer (See Figure 4.1). That is,MRPitself is independent of the underlying physical layer which takes care of raw bit transmissions.MRPis, thus independent of the electrical and mechanical aspects of the interconnection. The protocol is applicable both in situations where the transit time between routers is less than the clock period and in situations where multiple data bits are pipelined over long wires (See Section 3.2). MRPprovides mechanisms for controlling the transmission of data packets and the direction of transmission over interconnection lines. It also provides sufficient information back to the source endpoint so the source can determine when a transmission succeeds and when retransmission is necessary. By leaving the retransmission of corrupted packets to the source, MRP allows the source endpoint to dictate the retransmission policy. As such, both the

MRP-ROUTER andMRP-ENDPOINT are required to completely fulfill the role of the data-link layer.

SinceMRPprovides dynamic self-routing, the protocol layer identified as the network layer by the ISO OSI model is also provided byMRP.

MRP itself is connection oriented, though there is no need for higher-level protocols to be connection oriented. Together, MRP-ROUTER andMRP-ENDPOINT provide a reliable, byte-stream connection from end-to-end through the routing network.

4.4 Terminology

Recall from Chapter 3 that a crossbar has a set of inputs and a set of outputs and can connect any of the inputs to any of the outputs with the restriction that only one input can be connected to each output at any point in time. A dilated crossbar has groups of outputs which are considered equivalent. We refer to the number of outputs which are equivalent in a particular logical direction as the crossbar’s dilation,

d

. We refer to the number of logically distinct outputs which the crossbar can switch among as its radix,

r

^.

A circuit-switched routing component establishes connections between its input and output ports and forwards the data between inputs and outputs in a deterministic amount of time. Notably, there is no storage of the transmitted data inside the routing component. In a network of circuit-switched routing components, a path from the source to the destination is locked down during

M R P

MRPfits into the ISO OSI Reference Model at the data-link layer. The routers in a multipath network useMRP-ROUTERto transfer data through the network. Each endpoint usesMRP

-ENDPOINTto facilitate end-to-end data transfers.

Figure 4.1: METRORouting Protocol in the context of the ISO OSI Reference Model the connection; the resources along the established path are not available for other connections during the time the connection is established. In a pipelined, circuit-switched routing component, all the routing components in a network run synchronously from a central clock and data takes a deterministic number of clock cycles to pass through each routing component.

A crossbar is said to be self-routing if it can establish connections through itself based on signalling on its input channels. That is, rather than some external entity setting the crosspoint configuration, the router configures itself in response to requests which arrive via the input channels.

A router is said to handle dynamic message traffic when it can open and close connections as messages arrive independently from one another at the input ports.

When connections are requested through a router, there is no guarantee that the connections can be made. As long as the dilation of the router is smaller than the number of input channels into a router (i.e.

d < i

), it is possible that more connections will want to connect in a given logical direction than there are logically equivalent outputs. When this happens, some of the connections must be denied. When a connection request is rejected for this reason, it is said to be blocked. The data from a blocked connection is discarded and the source is informed that the connection was not

Crosspoint Array

Backward Port

Backward Port Backward Port

F o r w a r d P o r t F o r w a r d P o r t

F o r w a r d P o r t

The basic router has

i

forward ports and

o

⁼

r

d

backward ports. Any forward port can be connected through the crosspoint array to any backward port. The arrows indicat the initial direction of data flow.

Figure 4.2: Basic Router Configuration established.

Once a connection is established through a crossbar, it can be turned. That is, the direction of data transmission can be reversed so that data flows from the original destination to the original source. This capability is useful for providing rapid replies between two nodes and is important in effecting reliable communications. MRP provides half-duplex, bidirectional data transmission since it can send data in both directions, but only in one direction at a time. When data is flowing between two routers, we call the router sending data the upstream router and the router receiving data the downstream router.

Since connections can be turned around and data may flow in either direction through the crossbar router, it is confusing to distinguish input and output ports since any port can serve as either an input or an output. Instead, we will consider a set of forward ports and a set of backward ports. A forward port initiates a route and is initially an input port while a backward port is initially an output port. The basic topology for a crossbar router assumed throughout this chapter is shown in Figure 4.2.

Signal/Datum Control Bit Control Field

IDLE/DROP 0 all zeros

ROUTE 1 direction specification

TURN 0 all ones

STATUS 1 port specification

CHECKSUM 1 checksum bits

DATA-IDLE 0 distinguished hold pattern

DATA 1 arbitrary

The words sent over the network links can be classified as data words and signalling words.

The use of a single control bit which is separate from the transmitted data bits allows out of band signalling to control the connection state. This table shows how control signals and data are encoded. The control field is a designated log₂⁽max⁽

r;d

⁾⁾bit portion of the data word.

Table 4.1: Control Word Encodings 4.5 Basic Router Protocol

The behavior ofMRP-ROUTERis based on the dialog between each backward port of each router and its companion forward port in the following stage of routers. In this section, we describe the core behavior of the router signalling protocol from the point of view of a single pair of routing components.

4.5.1 Signalling

Routing control signalling is performed over data transmission channels. Using simple state machines and one control bit, this signalling can occur out of band from the data. That is, the control signals are encoded outside of the space of data encodings. Out of band signalling allows the protocol to pass arbitrary data. Table 4.1 shows the encoding of various signals. The control field is a designated portion of the data word. Due to encoding requirements, it is at least log₂⁽max⁽

r;d

⁾⁾

bits long. The remainder of this section explains how these control signals are used to effect routing control.

4.5.2 Connection States

The states of a forward-backward port pair can be described by a simple finite state machine.

Figure 4.3 shows a minimal version of this state machine for the purpose of discussion. Each transition is labeled as:

<

^event

>

^/

<

^result

><

^dir

>

^{. Where}

<

^event

>

is a logical expression, usually including the reception of a particular kind of control word,

<

^result

>

is an output resulting from the reception, and

<

^dir

>

is an arrow indicating the direction which the

<

^result

>

is sent. For instance, the arrow from swallow to forward means that when aDATA word is received and the resulting

output direction specified is not blocked, forward theDATAout the allocated backward port in the forward direction and change to the forward state.

Backward Blocked

Status

Status Blocked

Idle Drop

Forward

Swallow

NoTurn

d a t a * b l o c k e d

− − / d r o p < −

t u r n / s t a t u s < −

− − / c h e c k s u m < −

r o u t e * b l o c k e d

t u r n / s t a t u s < −− − / c h e c k s u m < − r o u t e * − s w a l l o w / r o u t e − >

d a t a * − b l o c k e d / d a t a − >

r o u t e * s w a l l o w

d a t a / d a t a < − i d l e / i d l e − >

d a t a

d a t a / d a t a − >

d r o p + r e s e t / d r o p − >

d r o p / d r o p − >

−−/ d a t a−i d l e−>

t u r n / d a t a−i d l e−>

As a c o n n e c t i o n i s o p e n e d , r e l e a s e d , r e ve r s e d , a n d u s e d i n t h e n e t w o r k , t h e i n d i vi d u a l l i n k s w i t h i n t h e n e t w o r k go t h r o u gh a s e r i e s o f c o n n e c t i o n s t a t e s a s s h o w n a b o ve . Tr a n s i t i o n s a r e i n i t i a t e d b y t h e re c e i p t o f a c o n t r o l w o r d (Se e Ta b l e4.1) a n d m o d i fie d b y t h e l o c a l s t a t e

o f t h e r o u t e r . Ea c h t r a n s i t i o n i s l a b e l e d a s :

<

^event

>

^/

<

^result

><

^dir

>

^.

<

^event

>

d e s c r i b e s

t h e c o n t r o l w o r d r e c e i ve d a l o n g w i t h s t a t e m o d i fie r s ;

<

^result

>

i s a n o u t p u t w o r d r e s u l t i n g fr o m t h e e ve n t ;

<

^dir

>

i s a n a r r o w i n d i c a t i n g t h e d i r e c t i o n w h i c h t h e

<

^result

>

i s s e n t .

Fi gu r e 4.3:MRP-ROUTERCo n n e c t i o n St a t e s

4.5.3 Router Behavior Idle port

Wh e n a c o n n e c t i o n b e t w e e n r o u t e r s i s n o t i n u s e , t h e c o n n e c t i o n i s i n a nidle s t a t e . Wh i l e i n t h e i d l e s t a t e , t h e b a c k w a r d p o r t o n a r o u t e r t r a n s m i t s t h eIDLEw o r d t o i t s c o r r e s p o n d i n g fo r w a r d p o r t

i n t h e n e xt s t a ge o f t h e n e t w o r k . A fo r w a r d p o r t i n t e r p r e t s t h e r e c e p t i o n o f a n i d l e w o r d t o m e a n t h a t i t s h o u l d r e m a i n i n a n i d l e s t a t e a n d h e n c e s h o u l d n o t a t t e m p t t o o p e n a n e w c o n n e c t i o n .

Route

To r o u t e a c o n n e c t i o n t h r o u gh a r o u t e r , aROUTE w o r d i s fe d i n t o a r o u t e r fo r w a r d p o r t . Th e

fo r w a r d p o r t r e c o gn i ze s t h e t r a n s i t i o n o f t h e c o n t r o l b i t fr o me c e i vi n g w h e nt h e ze r o , w h i c h i t w a s r

t h e c o n n e c t i o n w a s i d l e , t o a o n e . Th e r o u t e r t h e n u s e s t h e c o n t r o l fie l d t o d e t e r m i n e t h e d e s i r e d r o u t i n g d i r e c t i o n . Th e r o u t e r fo r w a r d s t h eROUTE w o r d t h r o u gh a b a c k w a r d p o r t i n t h e d e s i r e d

d i r e c t i o n , i f o n e i s a va i l a b l e , a n d l o c k s d o w n t h e c o n n e c t i o n s o s u b s e qu e n tDATA w o r d s c a n b e

p a s s e d i n t h e s a m e d i r e c t i o n . If n o o u t p u t i s a va i l a b l e , t h e c o n n e c t i o n i s b l o c k e d . Data

Al l w o r d s w i t h c o n t r o l s e t b i t s w h i c h a r e r e c e i ve d fo l l o w i n g aROUTE w o r d a r e fo r w a r d e d

t h r o u gh t h e a l l o c a t e d b a c k w a r d p o r t o f t h e r o u t i n g c o m p o n e n t t o t h e fo r w a r d p o r t o f t h e n e xt r o u t e r i n t h e n e t w o r k .

Data Completion

Wh e n a l l o f t h e d a t a w o r d s i n a m e s s a ge h a ve b e e n p a s s e d i n t o a r o u t e r , t h e s e n d e r h a s t w o o p t i o n s . Th e s e n d e r c a n e i t h e r d r o p t h e c o n n e c t i o n w i t h o u t get t i n g a n y r e sp o n s e , o r t u r n t h e

d i r e c t i o n o f t h e c o n n e c t i o n a r o u n d fo r a r e p l y.

To d r o p t h e c o n n e c t i o n , t h e i n p u t i s gi ve n aDROPw o r d ; t h e r e c e p t i o n o f aDROP w o r d c a u s e s

t h e r o u t e r t o c l o s e d o w n a n o p e n c o n n e c t i o n a n d fr e e u p t h e o u t p u t p o r t fo r r e u s e . Wh e n t h e o u t p u t p o r t i s fr e e d vi a aDROPw o r d , i t fo r w a r d s aDROPa l o n g t o t h e n e xt r o u t e r t h e n r e t u r n s t o a n i d l e .

To t u r n a c o n n e c t i o n a r o u n d , t h e i n p u t p o r t i s gi ve n aTURNw o r d ; w h e n t h e r o u t e r r e c e i ve s a

TURNw o r d , i t fo r w a r d s t h e t u r n o u t t h e a l l o c a t e d o u t p u t p o r t , i f t h e r e i s o n e , a n d r e t u r n s s e ve r a l s t a t u s w o r d s . Fi gu r e 4.3 s h o w s a ve r s i o n o f t h e p r o t o c o l w h i c h r e t u r n s t w o s u c h w o r d s . Th e s e w o r d s fil l t h e p i p e l i n e d e l a y a s s o c i a t e d w i t h i n fo r m i n g t h e s u b s e qu e n t r o u t e r t o t u r n t h e c o n n e c t i o n a r o u n d a n d ge t t i n g b a c k w a r d d a t a fr o m t h e s u b s e qu e n t r o u t e r . Th e fil l e r w o r d s d i ffe r b a s e d o n w h e t h e r t h e c o n n e c t i o n i s s e n d i n g d a t a i n t h e fo r w a r d o r r e ve r s e d i r e c t i o n w h e n t h eTURNi s r e c e i ve d .

In t h e fo r w a r d d i r e c t i o n , t h e r o u t e r r e t u r n s fir s t aSTATUSw o r d a n d t h e n aCHECKSUM w o r d . If

t h e c o n n e c t i o n w a s n o t b l o c k e d d u r i n g t h e r o u t i n g c yc l e , a ft e r s e n d i n g t h e s e w o r d s , t h e r o u t e r w i l l b e r e c e i vi n g d a t a i n t h e r e ve r s e d i r e c t i o n a n d w i l l fo r w a r d t h i s d a t a t h r o u gh . If t h e co n n e c t i o n w a s

b l o c k e d , t h e r o u t e r fo r w a r d s aDROPw o r d fo l l o w i n g t h eCHECKSUM a n d r e t u r n s t o t h e i d l e s t a t e . In t h e b a c k w a r d d i r e c t i o n , t h e r o u t e r s i m p l y s e n d s s e ve r a lDATA-IDLEw o r d s b e fo r e s e n d i n g t h e

r e ve r s e d a t a . In o r d e r fo r a p o r t t o b e i n t h e b a c k w a r d d i r e c t i o n , t h e r e m u s t b e a c o n n e c t i o n t h r o u gh t h e r o u t e r s o t h e r e w i l l a l w a ys b e d a t a t o p r o p a ga t e fo l l o w i n g a b a c k w a r d t u r n .

Checksum and Status Information

Th eSTATUS a n dCHECKSUM w o r d s fo r m a s e r i e s o f w o r d w i d e va l u e s w h i c h s e r ve s t o i n fo r m t h e s o u r c e n o d e o f t h e i n t e gr i t y o f t h e c o n n e c t i o n m a d e t h r o u ghe a c h r o u t e r i n t h e p a t h o b t a i n e d

t h r o u gh t h e n e t w o r k . A p o r t i o n o f t h eSTATUSw o r d i n fo r m s t h e s o u r c e a b o u t w h i c h o f t h e l o gi c a l l y

e qu i va l e n t b a c k w a r d p o r t s , i f a n y, t h r o u gh w h i c h t h e c o n n e c t i o n i s r o u t e d . Wh e n t h i s i n fo r m a t i o n a r r i ve s u n c o r r u p t e d a t t h e s o u r c e e n d p o i n t , i t a l l o w s t h e s o u r c e t o i d e n t i fy t h e r o u t e r s t h r o u gh w h i c h t h e c o n n e c t i o n w a s a c t u a l l y r o u t e d . Wh e n a c o n n e c t i o n i s b l o c k e d a t s o m e r o u t e r , t h i s

i n fo r m a t i o n s e r ve s t o p i n p o i n t w h e r e t h e c o n n e c t i o n be c a m e b l o c k e d . Th e r e m a i n i n g b i t s o f t h e

STATUS w o r d , t o ge t h e r w i t h t h e b i t s i n t h eCHECKSUM w o r d o r w o r d s , c a n b e u s e d t o t r a n s m i t a

l o n gi t u d i n a l c h e c k s u m b a c k t o t h e s o u r c e . Th i s c h e c k s u m i s ge n e r a t e d o n a l l t h e d a t a s e n t t h r o u gh t h e c o n n e c t i o n s i n c e t h e l a s tROUTE, i n c l u d i n g t h eROUTEw o r d i t s e l f. Wh e n d a t a i s c o r r u p t e d d u e

t o fa u l t s i n t h e n e t w o r k , t h e c h e c k s u m p r o vi d e s t h e s o u r c e e n d p o i n t w i t h s u ffic i e n t i n fo r m a t i o n t o i d e n t i fy t h e m o s t l i k e l y c o r r u p t i o n s o u r c e .

4.5.4 Making Connections

Wh e n a c o n n e c t i o n i s o p e n e d t h r o u gh a r o u t e r , t h e r e m a y o r m a y n o t b e o u t p u t s a va i l a b l e i n t h e d e s i r e d l o gi c a l o u t p u t d i r e c t i o n . If t h e r e i s n o a va i l a b l e o u t p u t , t h e r o u t e r d i s c a r d s t h e r e m a i n i n g b i t s a s s o c i a t e d w i t h t h e d a t a s t r e a m . Wh e n t h e c o n n e c t i o n i s l a t e r t u r n e d a r o u n d , t h e r o u t e rSTATUS

w o r d r e t u r n e d b y t h e r o u t i n g n o d e i n fo r m s t h e s o u r c e t h a t t h e m e s s a ge w a s b l o c k e d a t t h e r o u t e r . Wh e n e xa c t l y o n e o u t p u t i n t h e d e s i r e d d i r e c t i o n i s a va i l a b l e , t h e r o u t e r s w i t c h e s t h e c o n n e c t i o n t h r o u gh t h a t o u t p u t . Wh e n m u l t i p l e p a t h s a r e a va i l a b l e , t h e r o u t e r s w i t c h e s t h e d a t a t o a l o gi c a l l y a p p r o p r i a t e b a c k w a r d p o r t s e l e c t e drandomly fr o m t h o s e a va i l a b l e .

Th i s r a n d o m p a t h s e l e c t i o n i s t h e k e y t o m a k i n g t h e p r o t o c o l r o b u s t a ga i n s t d yn a m i c fa u l t s

w h i l e a vo i d i n g t h e n e e d fo r c e n t r a l i ze d i n fo r m a t i o n a b o u t t h e n e t w o r k s t a t e a n d k e e p i n g t h e r o u t i n g p r o t o c o l s i m p l e . Wh e n fa u l t s d e ve l o p i n t h e n e t w o r k , t h e s o u r c e d e t e c t s t h e o c c u r r e n c e o f a fa i l e d o r d a m a ge d c o n n e c t i o n b y t h e a c k n o w l e d gm e n t fr o m t h e d e s t i n a t i o n . Th e s o u r c e t h e n k n o w s t o r e s e n d t h e d a t a . Si n c e t h e r o u t i n g c o m p o n e n t s s e l e c t r a n d o m l y a m o n g e qu i va l e n t o u t p u t s a te a c h

s t a ge , i t i s h i gh l y l i k e l y t h a t t h e r e t r y c o n n e c t i o n w i l l t a k e a n a l t e r n a t e p a t h t h r o u gh t h e n e t w o r k , a vo i d i n g t h e n e w l y e xp o s e d fa u l t . So u r c e -r e s p o n s i b l e r e t r y c o u p l e d w i t h r a n d o m i za t i o n i n p a t h s e l e c t i o n gu a r a n t e e s t h a t t h e s o u r c e c a n e ve n t u a l l y fin d a fa u l t -fr e e p a t h t h r o u gh t h e n e t w o r k , p r o vi d e d o n e e xi s t s . Th e r a n d o m s e l e c t i o n a l s o fr e e s t h e s o u r c e fr o m k n o w i n g t h e a c t u a l d e t a i l s o f t h e r e d u n d a n t p a t h s p r o vi d e d b y d i l a t e d c o m p o n e n t s i n t h e n e t w o r k . Ra n d o m s e l e c t i o n a m o n g e qu i va l e n t a va i l a b l e o u t p u t s i s a n e xt r e m e l y s i m p l e s e l e c t i o n c r i t e r i o n t o i m p l e m e n t i n s i l i c o n a n d c a n b e i m p l e m e n t e d w i t h l i t t l e a r e a a n d c o n s i d e r a b l e s p e e d . Ran d o m s e l e c t i o n a l s o r e qu i r e s n o

s t a t e i n fo r m a t i o n n o t a l r e a d y c o n t a i n e d o n t h e i n d i vi d u a l r o u t i n g c o m p o n e n t . 4.6 Network Routing

Ea c h r o u t e r i n a p a t h t h r o u gh t h e n e t w o r k n e e d s t o s e e a d i ffe r e n t r o u t i n g s p e c i fic a t i o n . Si n c e w e r e qu i r e t h e r o u t i n g s p e c i fic a t i o n t o b e i n a fixe d p o s i t i o n i n t h eROUTEw o r d t o a l l o w e ffic i e n t

i m p l e m e n t a t i o n o f t h e p r o t o c o l , t h e d a t a s e e n i n t h i s p o s i t i o n b ye a c h r o u t e r m u s t b e d i ffe r e n t .

Be t w e e n r o u t i n g s t a ge s , t h e b i t s o f t h e d a t a p a t h c a n b e p e r m u t e d s o t h a t r o u t e r s i n d i ffe r e n t s t a ge s s e e d i s t i n c t c o n t r o l fie l d s . Th i s b i t r e o r d e r i n g a l l o w s a s i n gl e r o u t i n g w o r d t o s p e c i fy t h e p a t h t h r o u gh s e ve r a l r o u t e r s . Ho w e ve r , i f t h e n e t w o r k i s s u ffic i e n t l y l a r ge , a l l o f t h e b i t s i n a s i n gl e

r o u t i n g w o r d w i l l e ve n t u a l l y b e e xh a u s t e d b e fo r e t h e fu l l r o u t e t h r o u gh t h e n e t w o r k c a n b e s p e c i fie d . To d e a l w i t h t h i s c a s e ,MRPa l l o w s r o u t i n g s w i t c h e s t o b e c o n figu r e d t o i gn o r e t h e fir s t d a t a w o r d

i n a n i n c o m i n g m e s s a ge b y s e t t i n g aSWALLOWc o n figu r a t i o n b i t . Th i s o p t i o n a l l o w s n e t w o r k s t o b e a r b i t r a r i l y l a r ge . Eve r y t i m e a l l o f t h e r o u t i n g b i t s i n t h eROUTEw o r d a r e e xh a u s t e d , t h eROUTE

w o r d c a n b e d i s c a r d e d a l l o w i n g r o u t i n g t o c o n t i n u e w i t h t h e fr e s h r o u t i n g b i t s i n t h e s u b s e qu e n t

w o r d . In t h i s m a n n e r , t h e fir s tDATA w o r d i n t h e m e s s a ge fo l l o w i n g t h e o r i gi n a lROUTE w o r d i s p r o m o t e d t o b e t h eROUTEw o r d a ft e r t h e o r i gi n a l i s e xh a u s t e d .

Dans le document Robust, High-Speed Network Design for Large-Scale Multiprocessing (Page 71-78)