Ch 3: Network architecture and topology

Source: Sterbenz & Touch 2001, Ch 3 (pp. 79–118), read in full. Page numbers are the book’s printed pages.

Numbering note: a few principle boxes inside Ch 3 carry different IDs from Appendix A. Mesh scalability is “N-II.4” in the chapter and N-II.5 in the appendix; bandwidth aggregation is “N-5B” vs N-5Cb; administrative constraints “N-III.2” vs N-III. The chapter’s closing list also has an “N-4B Redundant Functionality in the Network” that the appendix leaves out. These notes use the Appendix A IDs, as in 01-principles-map.md.

The chapter in one paragraph

Latency along a path is the sum of its parts, bandwidth is the minimum of its parts (p. 116). Propagation is set by geography, so over long distances the only thing the network can do is offer straight paths and few hops. Large networks need hierarchy to keep routing state and diameter under control. Where to put functions (caches, multicast, active processing) is a trade-off among bandwidth, processing and memory, with latency as the constraint.

Topology: meshes beat shared media (pp. 81–82)

• Shared-medium LANs (Ethernet bus, Token Ring and FDDI rings, wireless) give broadcast for free, but every node gets a shrinking share of one channel as the network grows.
• A mesh of point-to-point links moves contention into switches, which can be scaled; when one switch is full you add another. “Virtually all network technologies” are now meshes, including Ethernet, which kept its MAC protocol but dropped the shared medium (p. 82).

Latency (pp. 82–88)

Components of delay (p. 82):

D = Σ d_i over the hops (p. 83). A path that is longer in distance and in hops can still be faster if a node on the short path has long queues (Fig. 3.4).

Speed of light (p. 84): c ≈ 3×10⁵ km/s. Fiber propagates at about 0.7c, copper at 0.6–0.95c depending on the medium. A signal needs about 200 ms to go around the Earth (40,000 km), already twice the 100 ms interactive budget.

Round-trip times from Table 3.1 (p. 84; one-way bandwidth–delay product shown at 1 Gb/s):

• In LANs and MANs the distance term is tiny next to node delays, so topology hardly matters for a 100 ms budget. In WANs, topology engineering is critical (p. 85).
• Bad topology example (Fig. 3.5): a US backbone built as a star around Chicago would send Boston–New York traffic 2,500 km in 12 ms instead of 300 km in 1.5 ms (p. 85).
• Diameter = the longest of all shortest paths, in hops. Traditional IP routers store each packet before deciding, so their hop count must be tightly bounded (N-1Ah, p. 87).
• The book estimates “on the order of 1 to 10 ms” per fast packet switch (p. 87; checked on the page image, the unit really is ms) and sets a design target of about 10 hops for the network diameter (p. 95). Both numbers are WAN thinking from 2001; today’s data-center switches are orders of magnitude faster per hop (No single source).
• Satellites: LEO latency is like a transcontinental WAN, GEO worse than intercontinental (p. 86). Example 3.1 tells the Iridium story (66 LEO satellites, 2,400 b/s channels, shut down in 1999) as a cautionary tale about picking the wrong application (p. 87).

Bandwidth (pp. 88–90)

• R = min(r_i): one slow link caps the whole path, and upgrading anything else is wasted (N-1Ab, p. 88).
• Striping one flow over parallel links needs the skew between links kept low enough to keep packet order, or resequencing at the end; parallel switch planes would also have to switch in lock step. “For this reason, virtually all network links are bit-serial” (p. 89). Sending one flow over several distinct paths is harder still. Using several links between nodes to raise aggregate bandwidth is common and fine (p. 89).
• The highest-bandwidth path is not always the lowest-latency path. In Fig. 3.7 the short path goes through node 2, the fat one through nodes 4 and 5 (p. 90). Network engineering must make sure good paths exist; routing and QoS must pick the right one per application.
• Bandwidth–delay product: a 5,000 km OC-192 (10 Gb/s) link holds on the order of 240 Mb in flight (p. 90). Where retransmission is hopeless, rely on forward error correction (p. 90).

Overlays (pp. 90–94)

• Reasons for overlays: VPNs, secure group sessions, datagram overlays (a permanent mesh of connections so short flows skip connection setup; long flows move to their own reserved connection), optical lightpaths (p. 91).
• An overlay path can be longer than the physical shortest path, so diameter and latency principles apply to it too. Keep overlay levels few: the book’s bad example is a VPN over IP over an ATM PVC mesh over lightpaths over fiber (N-IIo, p. 92).
• Wavelength routing and assignment (pp. 92–94) is optical-network detail; skip unless you work on DWDM.

Scale and hierarchy (pp. 95–110)

• Design parameters (p. 95): size n (end systems), diameter h_max, node degree k, node count s, path diversity, aggregation in the core. Keep h_max around 10 hops so per-hop delays do not dominate.
• Switches with thousands of ports are reasonable, millions are not; k ≈ 1024 was a sensible upper bound in the early 2000s (p. 96).
• Sparse vs dense (Fig. 3.9, 16 end systems): sparse k = 4, s = 11, h_max = 6; dense k = 8, s = 3, h_max = 3. Dense topologies concentrate traffic in the core; extra switches add parallel paths (pp. 96–97).
• Hierarchy keeps link-state databases and flooding bounded. Rule of thumb: clusters of 10–100 nodes with degree around 10 (p. 97). Each node knows its own cluster in full and other clusters only in aggregate (Fig. 3.10, as in ATM PNNI). It helps if the addressing hierarchy matches the routing hierarchy (p. 98).
• Hierarchy also aggregates bandwidth in the core and isolates local traffic at the edge. Example 3.2: ATM core switches looked only at the 12-bit VPI, so tables needed at most 4K entries instead of 2²⁸; MPLS label stacks generalize the idea (p. 101).
• Hierarchy can bound latency: a densely meshed backbone of 2–3 hops plus bounded hops in each lower level keeps the total to tens of hops (pp. 102–103).
• Reality is messier: policy routing (avoid a competitor’s network, prefer a contracted provider) (p. 107), and providers that peer where it suits them, so two nearby east-coast endpoints can be routed through a Midwest peering point (Fig. 3.14, p. 108).
• Example 3.3, “When is a hop not a hop? When it is a POP” (p. 109): routers had only 64–128 ports, so provider POPs grew into small sparse networks of their own, with perhaps 10 hops per POP instead of one. Larger routers fixed that and then strained BGP-4.
• Wireless density control (pp. 103–106) and gateways between different technologies (pp. 108–110): skim.

Resource trade-offs (pp. 110–116)

• Optimize an objective function over the costs of bandwidth, processing and memory, f(B, P, M), then add latency as a constraint (pp. 111–112).
• Example 3.4, multicast (pp. 111–112): sending to n receivers with n unicast flows can put n·r on one link, needs n times the buffers and routing entries, and n times the sender’s processing. A multicast tree carries r on every link with one entry per node. For n-to-n communication, unicast needs n² − n flows.
• Example 3.5, where to cache video (p. 113): moving copies deeper into the distribution tree cuts bandwidth roughly linearly but raises memory exponentially; the sum has a minimum at an optimal depth. A latency bound may push the copies closer to users than that optimum.
• What scaling does (p. 114): bandwidth and processing get cheaper, propagation does not. So latency becomes relatively more important as bandwidth grows, and the bandwidth–delay product (and the buffering it needs) keeps rising.
• Active networking (pp. 115–116), where nodes run injected or provisioned code, is the 2001 research idea; read only as background.

Trading-network lens (my mapping)

Latency is a sum, and only t_q moves. In a colocation, model the path as NIC → switch → switch → exchange handoff and assign each term: propagation (metres of fiber), per-switch forwarding, serialization, queuing. All but the queuing term are roughly constant; queuing is where jitter and microburst loss come from. No single source.

Diameter. The book’s 10-hop target is for a WAN. For the latency-critical path in a trading site the target is one or two switch hops, with layer-1 switches for pure fan-out, and that drives flat designs. No single source.

Straight paths are the product. The book’s Boston–New York-via-Chicago example (12 ms vs 1.5 ms) is the trading-route problem in miniature: for long routes such as Chicago–New Jersey, firms pay for the straightest path they can get. Unverified (route-specific figures not checked yet).

The fastest path is not the fattest (Fig. 3.7). A low-latency radio route has less bandwidth than fiber, so trading networks split traffic: small latency-critical messages on the fast, thin path, bulk traffic on fiber. No single source.

Striping and per-flow hashing. The book’s skew argument is why ECMP and link aggregation hash per flow: all packets of one flow stay on one member link and stay in order. A single market data feed therefore uses one member link, however many there are. No single source.

Bottlenecks and fan-in. R = min(r_i) also applies in time: several feeds merged onto one 10G link make that link the bottleneck during bursts, which is where microburst drops happen. See ../multicast/02-microburst-buffer.md.

Multicast (Example 3.4). One feed to many consumers costs r per link instead of n·r, which is why market data is distributed by multicast. See ../multicast/00-glossary.md.

Overlays hide the path (N-IIo). Tunnels hide the physical route from routing and add encapsulation, so keep the latency-critical path on an underlay you can see and control. No single source.

“When is a hop not a hop?” Know what is inside each hop you pay for: a single “cross-connect” can contain patch panels, a media converter or a provider switch. No single source.

Latency gets relatively more important (p. 114). The book’s 2001 observation is the economics of the low-latency trading race: bandwidth kept getting cheaper, distance did not.

Self-check

1. Name the delay components the book uses in Ch 3. Which one changes with load?
2. Using the book’s 0.7c for fiber, what is the one-way propagation delay over 1,200 km?
3. A path has nine 10G links and one 1G link. What can it deliver, and which principle says so?
4. Why, according to the book, are links bit-serial instead of striping a flow over parallel links? Which modern feature follows the same logic?
5. In Fig. 3.7 the lowest-latency path and the highest-bandwidth path differ. How does a trading network deal with that?
6. Ten receivers behind one link want the same 1 Gb/s feed. What does that link carry with unicast, and with multicast?
7. What did Example 3.3 teach about counting hops?