Other Ways to Collect Routing Data

There are some problems with using raw routing table statistics to construct the routing matrix. In this section, we discuss the problems and ways to overcome them. We used the routing table because it was the only data available. This method has the following problems:

Incomplete Data. When a packet is counted in a router, we only find out where it was headed, not where it came from. If a router connects two similarly-loaded networks to a backbone, there is no way to determine, from the routing table data, which traffic onto the backbone belongs to which leaf network. If the two networks actually talk to completely disjoint sets of networks, the data will not reflect it. This inaccuracy was acceptable in our studies because almost all routers had only one loaded network connected; the others were essentially idle. In addition, we accepted the imprecise routing matrix, where all leaf networks connected to a router were lumped together. This completely avoids the problem.

Multiply-Counted Packets. If a packet passes through a longer path more than just the two backbone routers that are between most leaf networks, it will be counted

additional times at any additional routers. In the Stanford internet, there were only three routers not connected directly to the backbone that would cause packets to be counted multiple times. Two of those routers could not be monitored, and the other was essentially idle.

Lost Data. Route resets cause accumulated packet counts to be thrown away. A solution to this would be for the router to keep all entries and simply mark them as bad. This increases the amount of trash in the routing table, especially in a large, dynamic internet.

No Size Information. Obviously, only packets, not bytes, are counted. Any system, independent of its other attributes, need only keep a byte counter in addition to the packet counter to report actual character throughput. Because this solution can be applied to any monitoring system, it will not be discussed any further.

The “perfect” solution would be to have the routers keep statistics independently of the routing table. Rather than a single list of packet counts, the router would keep one list of counts for each directly-connected network (i.e. one per interface). The table would be an matrix, with equal to the number of interfaces and equal to the size of the routing table.

When a packet arrives at the router, its incoming interface is noted and the source address is checked to see if it originated on the directly-connected network. If it did, then it will be counted when it is routed toward its destination. The counter will be incremented. A routing matrix over the entire internet is obtained by accumulating statistics from all of the routers. This scheme avoids multiple counting because packets are only counted when they arrive at the first router in their path. It gets more complete data because it differentiates between source networks when it counts packets.

The obvious disadvantage is that this scheme adds additional work for the router, which may not be worth the knowledge gained. An alternative will be available shortly when routers that collect detailed accounting information are introduced. One vendor’s latest

router⁵, at least, collects packet and byte counts on a source/destination pair basis. This data can be fed into a variation of the above scheme and be used to accumulate a source-destination matrix over the entire internet.

4.4 Conclusions

Both the annotated network map and the source-destination matrix provide valuable infor-mation about traffic flow in an internetwork. They provide slightly different perspectives, however. The annotated network map provides a picture of how the internet, in its cur-rent configuration, is utilized. The traffic matrix shows where the traffic sources and destinations are in the internet. In essence, it explains why the traffic flows as it does in the map. The distribution of traffic sources and sinks across the internet is not uniform

— a few networks account for most of the traffic and many networks are essentially idle.

Surprisingly, this distribution is very smooth; almost exponential.

5This software version was only recently released, and is not yet in operation at Stanford, so it was not used to collect the statistics reported here. The work would have considerably easier had it been available.

Conclusions

The measurements described in this dissertation have added a great deal to our under-standing of the behavior of the Stanford internetwork, as well as local-area internetworks and their components in general.

Delay through a local-area internetwork is determined almost entirely by the delay expe-rienced by packets going through routers. Most interestingly, Ethernet contention delay, which has been the subject of much research in the past, was not a significant factor in the overall end-to-end delay. We found that the largest component of end-to-end delay was delay through routers, a topic that has received little research. When we measured the router, we found that delay was considerably higher than predicted by a single-service center open queuing network model. Queuing delay, caused by bursty packet arrivals, was not the major cause of this delay, however. This result was surprising, since other research has shown network traffic to be highly bursty, which would be expected to cause more queuing than predicted by the queuing network model, which assumes a Poisson distribution. What we found was that packet arrivals, although not Poisson, do not occur in dense enough bursts to cause excessive queuing.

The discrepancy between actual and predicted delay was caused by size-dependent tasks in the router — long packets could take three times as long to process as short packets.

97

In addition, we found that average packet size varied by factors of 4 or 5 over short intervals. These two facts combine to violate a key assumption of the queuing network model. The simulation, however, produced very accurate results for all of the routers tested when we incorporated the size-dependent delay.

The topology of the Stanford internetwork has changed considerably since its beginnings as a loosely-connected mesh of networks. It has grown much larger in all ways: number of connected subnetworks, number of hosts, and utilization. Because of this increased usage and users’ dependency upon the network for vital services, network reliability is essential. In the early days, a cross-campus link might be down for a few hours without much complaint. There were few users, and most were appreciative of whatever service was available — even with outages, it was far better than previous methods, such as carrying a tape across campus. Now, however, with coursework, research, and administration dependent upon the proper functioning of the network, reliability is very important. Users usually notice outages and report them within a few minutes.

We devised a set of monitoring and analysis tools to study the topology of the Stanford internet. We monitored the topology of the Stanford internetwork for 18 months, as it was restructured from its initial ad hoc design to a more structured design incorporating a redundant backbone. We analyzed the impact of each of the changes on the robustness of the internet and found that, while most changes were innocuous, some had a profound impact on robustness. During this transition, some redundant links were temporarily removed and not replaced for several months, greatly increasing vulnerability to single-point failures.

Surprisingly, the backbone topology, although larger and more structured, is not signifi-cantly more robust than the old topology. There are still network segments and routers that will disconnect large sections of the network if they fail.

As more people use the Stanford network, it has become more difficult to make categorical claims about who’s using what — which users, located where, are using what resources.

It is important to know where the traffic is concentrated in the internet to plan for future needs. We used two techniques to measure and display the traffic patterns. First, we

measured packet rates on the links connecting routers and networks. By annotating a topological map with these packet rates, they give a very clear picture of where traffic is concentrated in the internet. Unfortunately, it is usually not possible to determine ultimate sources and destinations because the traffic blends together in shared links, especially the backbone. Second, we sampled routing table usage counts over the entire internet to construct a source-destination matrix. This method clearly shows the endpoints of traffic, whereas the first shows the paths used.

These measurements have greatly enhanced our understanding of how the Stanford in-ternetwork is used. With this knowledge, we can answer such questions as: where is additional capacity needed; are any routers operating at or near capacity; where are po-tential trouble spots in the network; and where are network users concentrated and what network services do they use.