======================================================================= GIGABUFFER REPEATER PROPOSAL Bernard Daines Packet Engines (509) 922-9190 FAX (509) 922-9185 bernardd@packetengines.com Box 14497 Spokane WA 99214 12119 E Mission Spokane WA 99206 Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Contributors To This Proposal Hon Wah Chin Cisco Systems David Schwartz Cisco Systems Ahmed Nouri Compaq Andreas Bechtolsheim Granite Systems Howard Johnson Olympic Technology Group Kevin Daines Packet Engines Howard Frazier Sun Microsystems Mart Molle UC Riverside Hassan Kamgar VLSI Samba Murthy XAQTI Alak Deb XAQTI Mark Sankey 3Com Wen Tang 3Com Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= In The Beginning. . . +------+ | comp | | | +------+ | | Coax _could not_ support X full duplex T------+-------+-----+----------T (half duplex media) X X | | | | | | +------+ +------+ | comp | | comp | | | | | +------+ +------+ X = Transceiver T = Terminator Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Then Came The Original Repeater +------+ +------+ | comp | | comp | | | | | +------+ +------+ | | | Coax Half Duplex | Coax X +--------+ X T----+-----+-----+--------|Repeater|--------+-----+-----+----T X X +--------+ X X | | | | | DIX | DIX | DIX | DIX +------+ +------+ +------+ +------+ | comp | | comp | | comp | | comp | | | | | | | | | +------+ +------+ +------+ +------+ X = Transceiver T = Terminator Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Then Came The Half Duplex Multiport Coax T-----+-----------------------+----------------------+-----T X X X | DIX | DIX | DIX | | | +------+ +---------+ +------+ | comp | |Multiport| | comp | | | +---------+ | | +------+ | | | +------+ ----- | ----- | DIX | DIX | DIX | | | +------+ +------+ +------+ | comp | | comp | | comp | | | | | | | +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Thinnet Was Still Half Duplex +------+ +------+ | comp | | comp | | | | | +------+ +------+ | | / \ / \ / \ / \ / \ / \ T--- / \ / \ ---T \ / \ Thin Coax / \ / \ / \ ------------- / \ / | | | | +------+ +------+ +------+ +------+ | comp | | comp | | comp | | comp | | | | | | | | | +------+ +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= As Time Passed. . . 10BaseT +------------+ (Bus In A Box) | Repeater | | (hub) | | | Coax bus went to twisted pair star +------------+ Links _could_ support full duplex / | \ _Forced_ half duplex by repeater /100m |100m \100m mechanism and emulation of coax bus / UTP |UTP \UTP +------+ +------+ +------+ | comp | | comp | | comp | | | | | | | +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= More Time Passed. . . 100BaseT +------------+ (Bus In A Box) | Repeater | | (hub) | TX links _could_ support full duplex | | TX _forced_ half duplex by repeater +------------+ mechanism, T4 and continued emulation / | \ of coax bus /100m |100m \100m Still star / UTP |UTP \UTP Still twisted pair +------+ +------+ +------+ Still 100m | comp | | comp | | comp | | | | | | | +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= 10BaseT accommodates Multiple Repeaters +------------+ | | /| Repeater |\ | (hub) | / | | \ +------------+ / / | \ \ /100m |100m \100m / / UTP |UTP \UTP \ +------+ +------+ +------+ T--- / | comp | | comp | | comp | \ ---T \ | | | | | | / \ / +------+ +------+ +------+ \ / +------------+ +------------+ | Repeater | | Repeater | | (hub) | <- 2000m -> | (hub) | | | | | +------------+ +------------+ / | \ / | \ /100m |100m \100m /100m |100m \100m / UTP | UTP \UTP / UTP | UTP \UTP +------+ +------+ +-----+ +------+ +------+ +------+ | comp | | comp | |comp | | comp | | comp | | comp | | | | | | | | | | | | | +------+ +------+ +-----+ +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= 100BaseT Accommodates Multiple Repeaters, Almost ----------------- / \ +------------+ +------------+ | Repeater | | Repeater | | (hub) | | (hub) | | | /\ | | +------------+ 5m +------------+ / / | | \ \ /100m /100m |100m |100m \100m \100m / UTP / UTP |UTP |UTP \UTP \UTP / / | | \ \ +------+ +------+ +------+ +------+ +------+ +-----+ | comp | | comp | | comp | | comp | | comp | |comp | | | | | | | | | | | | | +------+ +------+ +------+ +------+ +------+ +-----+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Now For Gigabit Ethernet We Have Full Duplex Media Full Duplex PHY's Full Duplex MAC's Why Settle For Half Duplex Use? Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= But the problem with gigabit CSMA/CD is the collision domain diameter – perhaps 10 meters Solutions Call it a feature Extend it by extending the carrier event around a minimum packet Localize CSMA/CD theater of operation Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Problem With Carrier Extension For 1000Base-T/F Is Small Packet Performance Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= An Important Point! Motivation For Gigabit Repeater Is Cost vs. Gigabit Switch Not An _Inherent_ Need To Accommodate Half Duplex Media As Was The Case In Original Ethernet Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= There May Never Be Any Half Duplex Media * Two Fibers * Two Coaxes * Multiple Cat-5's Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= The Breakthrough! +-------------------+ | | | CSMA/CD In A Box | | | +-------------------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= The Understanding! CSMA/CD is the access method to the network segment _not_ the access method to the specific link! Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Solution! Link Access Control Avoids Need To Extend Carrier Event Around Minimum Packet For Full Duplex Links Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= 1000Base-T/F Just One Repeater! +------------+ | Repeater | | (hub) | | | +------------+ / | | \ / | | \ / | | \ / | | \ / | | \ +------+ +------+ +------+ +------+ | HUB | | HUB | | HUB | | HUB | | | | | | | | | +------+ +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= +------+ +------+ +------+ +------+ +------+ +------+ | Tx |-->| |------->| |-->| Rx |-->| FIFO |-->| H | +------+ | PHY | Full | PHY | +------+ +------+ | A | | | Duplex | | | L | +------+ | | Link | | +------+ | F R | | Rx |<--| |<-------| |<--| Tx |< ------------| E | +------+ +------+ +------+ +------+ | D P | | U E | | P A | +------+ +------+ +------+ +------+ +------+ | L T | | Tx |-->| |------->| |-->| Rx |-->| FIFO |-->| E E | +------+ | PHY | Full | PHY | +------+ +------+ | X R | | | Duplex | | | | +------+ | | Link | | +------+ | C C | | Rx |<--| |<-------| |<--| Tx |<-------------| S O | +------+ +------+ +------+ +------+ | M R | | A E | | / | +------+ +------+ +------+ +------+ +------+ | C | | Tx |-->| |------->| |-->| Rx |-->| FIFO |-->| D | +------+ | PHY | Full | PHY | +------+ +------+ | | | | Duplex | | | | +------+ | | Link | | +------+ | | | Rx |<--| |<-------| |<--| Tx |<-------------| | +------+ +------+ +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= +------+ | H | | A | | L | | F R | +-------+ +------+ +------+ +------+ +------+ | E | | Tx |->| |------>| |-->| Rx |-->| FIFO |--->| D P | +-------+ | PHY | Full | PHY | +------+ +------+ | U E | /\ | | Duplex| | \/ | P A | +-------+ | | Link | | +-------------------+ | L T | |Link |<-| | | |<--|Link Access Control|<-| E E | |Access | | | | | +-------------------+ | X R | |Control| | | | | | | +-------+ | | | | | C C | | | | | | S O | +-------+ | | | | +------+ | M R | | Rx |<-| |<------| |<--| Tx |<--------------| A E | +-------+ +------+ +------+ +------+ | / | | C | | D | +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Mechanism IPG From Buffered Repeater To DTE Is Filled With Defer Instead Of Idle When FIFO Is Near Full And There Is Contention In The CSMA/CD Theater Of Operation Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Defer Sampled At Preamble Of Packets Coming Into DTE +-----+ +-----+ +-----+ +-----+ IDLE| PKT |IDLE| PKT |DEFER| PKT |DEFER| PKT |IDLE ----+ +----+ +-----+ +-----+ +----- BASIC+-----+ +-----+ +-----+ +-----+ CRS | | | | | | | | ----+ +----+ +-----+ +-----+ +----- +-----------------------+ DEFER | | ---------------------+ +----- EFFECTIVE +-----+ +----------------------------+ CRS | | | | ----+ +-----+ +----- Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= DTE Responds To Defer The Same As DTE Responds to CRS Defer Only ________________________ CRS Not Used By Receive MAC Section Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Minor Mac/PHY Tweak! | Defer To | Jam On | | CRS/Defer | COL | --------------------------------------------- Half Duplex | Yes | Yes | 802.3u --------------------------------------------- Buffered | Yes | No | New! Repeater | | | --------------------------------------------- Full Duplex | No | No | 802.3x Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= ISSUES Link Length (100m - 2km) Buffer size (2K - 10K) Link Access Control (Defer Signaling) Link Access Control (PHY Based) Repeater Core Algorithm (CSMA/CS) Outbound FIFO? Patent Issues? Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Impact Of Frame-Based Flow Control On "Gigabuffer" Repeater (contributed by 3Com) Latency From FIFO High Watermark To Transmit Defer = 2 Max Length Frames + 1 Round Trip Delay + 1 IPG + 1 "XOFF" Frame = ~ 3200 Byte Times Latency From FIFO Low Watermark To Transmit Enable = 1 Max Length Frames + 1 Round Trip Delay + 1 IPG + 1 "XON" Frame = ~ 1600 Byte Times ---> Reasonable FIFO Size = 6 KBytes With High Watermark = 3K Low Watermark = 1.5K Overhead if 802.3x Flow Control = (2 Flow Control (Min Length) Frame = 1 IPG) / (High Watermark-Low Watermark) = 10% Overhead If PHY Based Busy / Not Busy Symbols in IPG = 0% Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Improved Link Access Method For Full-Duplex 802.3z / Gigabit Ethernet (contributed by Granite Systems) 1. Separate the link access method (or mechanism) from the behavior of the network device. 2. Proposed semantics is a fixed size receiver FIFO transmitting a "CSMA/Defer" bit upstream during IDLE. 3. Works with the existing full-duplex MAC. Disconnect the collision wire, and connect the Idle/Defer symbol to a flip-flop that drives the existing carrier sense line. Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Receiver FIFO Size Required (contributed by Granite Systems) Goal is to allow the link to operate at full throughput. This requires FIFO "high-low" watermarks as follows: BUSY: 1 max size packet that causes BUSY event +1 max size packet before BUSY is transmitted +1 RTD (round-trip-delay) IDLE 1 max size packet before IDLE event +1 RTD (round-trip-delay) TOTAL 3 max size packet + 2 RTD 8 Kbyte FIFO allows for 3 packets + 2*1km RTD Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= CSMA/Defer vs Pause Comparison (contributed by Granite Systems) 1. Xon/Xoff is not required in full-duplex implementations 2. Overhead of Xon/Xoff/Pause signaling is: (2 min size packets + IPG) / (high-low watermark) Approximately 15% for 1 Kbyte watermark 3. For buffered repeaters, Xon/Xoff messages must be inserted into output stream for every link state change 4. Pause messages can get lost due to link errors 5. Pause mechanism not suitable for basic link access Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= CSMA/Defer vs CSMA/CD (contributed by Granite Systems) 1. Maintain CSMA/CD mechanism for ultra-low-cost hubs 2. CSMA/Defer avoids performance issues of stretched packets, increased capture effect, etc. etc. 3. CSMA/Defer repeaters can implement a wide range of arbitration schemes, including full speed CSMA/CD 4. CSMA/Defer could avoid the need to specify a Gigabit/second Repeater in 802.3z Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= 802.3u Definitions | | |<------ SLOT TIME ------->| | 512 bit times ( 64 bytes ) | ~-----------------------------~ ~-------------------------------------~ | |S| | |LF| | | | | |S| | IDLE | PRE |F| DA | SA |/L| DATA PAD|FCS| IDLE | PRE |F| DA | SA | |D| | |TD| | | | | |D| | ~-----------------------------~ ~-------------------------------------~ octet count 7 1 6 6 2 |<- 46 - 1500 ->| 4 12 7 1 6 6 | | | |<---- MINIMUM FRAME ----->|<--- IFG ---->| | 64 bytes | 1.6uS | | | | |<--------- MINIMUM PACKET ------->|<- IPG ->| | 72 bytes ( w/ 8 byte preamble + sfd )| .96uS | Legend ----------------------------------------------- * 1 Octet = 1 Byte * L/T FLD = Length / Type Field * Minimum Frame = 64 Bytes * Maximum Frame = 1518 Bytes * IPG = InterPacketGap = .96uS = 12 Bytes = 24 Clocks @ 25MHz * IFG = InterFrameGap = 1.6uS = 20 Bytes = 40 Clocks @ 25MHz Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Effect Of Carrier Extension On Network Throughput Relative to 20 Byte IPG (contributed by 3Com) For Typical Frame Size Distribution, 64 Byte IPG Has 86% Performance 128 Byte IPG Has 74% Performance 448 Byte IPG has 53% Performance Of 20 Byte IPG Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Why Not 802.3x flow Control? * Flow control specified for MAC/MAC point/point links only * Defer bit more efficient for PHY/PHY link access Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ====================================================================== +------+ +------+ +------+ +------+ +------+ +------+ | Tx |-->| |------->| |-->| Rx |-->| FIFO |-->| H | +------+ | PHY | Full | PHY | +------+ +------+ | A | | | Duplex | | | L | +------+ | | Link | | +------+ +------+ | F R | | Rx |<--| |<-------| |<--| Tx |<--| FIFO |<--| E | +------+ +------+ +------+ +------+ +------+ | D P | | U E | | P A | +------+ +------+ +------+ +------+ +------+ | L T | | Tx |-->| |------->| |-->| Rx |-->| FIFO |-->| E E | +------+ | PHY | Full | PHY | +------+ +------+ | X R | | | Duplex | | | | +------+ | | Link | | +------+ +------+ | C C | | Rx |<--| |<-------| |<--| Tx |<--| FIFO |<--| S O | +------+ +------+ +------+ +------+ +------+ | M R | | A E | | / | +------+ +------+ +------+ +------+ +------+ | C | | Tx |-->| |------->| |-->| Rx |-->| FIFO |-->| D | +------+ | PHY | Full | PHY | +------+ +------+ | | | | Duplex | | | | +------+ | | Link | | +------+ +------+ | | | Rx |<--| |<-------| |<--| Tx |<--| FIFO |<--| | +------+ +------+ +------+ +------+ +------+ +------+ Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= This Proposal Does Not Replace Traditional CSMA/CD For Any Half Duplex Media That Is Developed With Whatever Carrier Extension And/Or Collision Domain Desired Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Conclusion * Formalize One Repeater Topology * Standardize Defer Signaling Rules * Standardize Only Minimal Behavior Of Buffered Repeater Leaving Features And Link Lengths To Implementers Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 ======================================================================= Help Needed * Simulation Effort * Patent Issues Collection * Implementation Tutorial For July Packet Engines Incorporated Gigabuffer Repeater Proposal 6/6/96 =======================================================================