B’com Shifts Switch to 12.8 Tbits/s

　　Broadcom is sampling a 12.8 Tbit/second Ethernet switch chip targeting large data centers. The news shows the company continues to set the pace for a growing pack of competitors angling for a piece of one of the most demanding markets in networking.

　　The Tomahawk-3 packs twice the aggregate bandwidth of the Tomahawk-2 launched 14 months ago, both made in the same 16FF+ TSMC process. The company used engineering cleverness to pack 256 56-Gbit PAM-4 serdes in an “incrementally larger” die than the T2 that used as many 25G serdes.

　　In recent years, the Ethernet switch market has attracted seven merchant players. At least one of them, startup Innovium, aims to sample its own 12.8 Tbit/s chip within weeks.

　　Broadcom dominates the market today with a 73 to 94 percent share, depending on how market watchers slice the sector valued at nearly a billion dollars. Its closest rival, Cisco Systems, takes most of the rest with systems using its own ASICs. Juniper, Hewlett Packard Enterprise and Huawei also make Ethernet switch ASICs for their systems.

　　Increasingly the dozen largest data center operators — including the likes of Facebook and Google — build their own switch systems or specify systems built by ODMs. They can drive sales of millions of chips a year but demand maximum bandwidth at minimum cost and power consumption.

　　Rising data traffic drives their need for bandwidth. The advent of machine learning and discrete flash storage servers is further fueling the Web giants’ need for fast networks, Broadcom said.

　　The Tomahawk-3 is geared for the next-generation of their top-of-rack and aggregation switches, delivering up to 128 100GE or 32 400GE ports, the first merchant chip to support 400GE rates. While Broadcom declined to give many specifics about this chip, it did say its delivers 100GE at 40 percent less power and 75 percent lower cost than the prior part.

　　The chip is “a major achievement,” said Bob Wheeler, a senior analyst with The Linley Group.

　　“The new Broadcom has been surprisingly aggressive — it has actually increased its high-end product cadence. They are investing in three different switch architectures — Tomahawk, Trident, and Jericho — leaving few openings for competitors,” Wheeler said.

　　With seven merchant chips in the pipeline and four in-house ASICs in the works, “we will see a record number of unique platforms ship in 2018, not only those based on 50 Gbit/s lanes,” said Alan Weckel, principal of market watcher 650 Group.

　　“There will be over one dozen unique silicon offerings for the data center by the end of 2018, shattering any previous record by a significant amount,” Weckel said.

　　“Back in the day every switch company did their own ASIC, then they all went merchant with really just Broadcom, and now we are back to massive ASIC diversity again,” he added, pointing to the four OEMs designing their own chips.

　　Expect some designs to fall by the wayside, said Sameh Boujelbene, a senior research director at Dell’Oro Group Inc.

　　“I don’t think the market can support this many merchant silicon vendors. I think two or three years from now we probably should see only two or three in that space,” Boujelbene said.

　　Among the merchant players, “only Cavium and Mellanox are really shipping in volume in 2017,” said Weckel. Marvell, China’s Centec and Taiwan’s Nephos are generally aiming at midrange and low-end markets.

　　A year ago, Cavium claimed it had just started shipping its 3.2 Tbit/s XPliant switch chip and had eight OEM design wins including ones at Arista and Brocade, now part of Broadcom. Mellanox’s chips generally support Ethernet and Infiniband but are mainly used in its own systems.

　　A representative for startup Barefoot Networks said it is in production of its Tofino chips that deliver 1.9 to 6.5 Tbits/s with white-box systems shipping since April from Edge-Core, WNC and Inventec. Customer annucements from OEMs and end users will come “in the near future,” it said.

　　“Definitely, Cavium and Barefoot are making most of the headlines in terms of design wins so far. However, we have yet to see whether these design wins are for large-scale deployment or just to put pressure on Broadcom,” said Boujelbene.

　　All the rivals except Nephos, a Mediatek spinoff, buy serdes from Broadcom’s Avago division. “When I met with Nephos, they were definitely laser focused on landing design wins with large cloud service providers such as Amazon,” he added.

　　Nephos came out of stealth mode in March with a 3.2 Tbit/s chip and plans for a 6.4 Tbit/s chip using die stacked with TSMC’s InFO packaging. Innovium announced its plans for a 12.8 Tbit/s chip a week later.

　　Innovium has “active design engagements” with OEMs, ODMs and cloud providers it expects will roll systems in early 2018 about the same time as those using the Tomahawk-3, said Amit Sanyal, vice president of product marketing for the startup. It claims its chip will deliver better “buffering, line-rate programmability, latency, performance/watt and analytics/telemetry” than Broadcom.

　　Wheeler of the Linley Group expects the competition will drive Ethernet switch costs from about $60/port today to about $36/port by 2020.

　　Rochan Sankar, a senior director of product marketing at Broadcom, disagrees. He notes the new chip will pack into a 1U system the capabilities of an 8U system designed by Facebook using 12 Tomahawk-1 chips.

　　“I actually think competition hasn’t been driving a lot of disruption in pricing…we’re delivering leadership economics already,” Sankar said.

　　Despite a reputation for cost cutting under new CEO Hock Tan, the Ethernet switch business is “one of several sustainable franchises where we are leaders…We continue to out-invest all our merchant competitors combined despite a recent influx of interest in Ethernet switching,” he added.

　　Barefoot and Cavium claim a software edge because they support programmable engines that can readily add new protocols as they emerge. Their work is part of a move supported by giants including AT&T and Google to software-defined networks that are easier to manage.

　　Data center giants “are driving their own code and programmable capabilities as close to the server as possible,” said Broadcom’s Sankar. Thus, the company has added programmability to its lower-end Trident and Jericho switches targeting carriers and business users and so-called control-plane networks.

　　At the high end of data-plane nets, big data centers “want performance per cost and per power,” the focus for Tomahawk. “For the top 15 to 20 cloud providers in the world performance and opex/capex efficiency are higher priorities than programmability,” he added

　　Tomahawk-3 packs a Gbit of memory, the most of any Broadcom switch to date. Engineers designed a new block for the chip enabling a shared buffer for RDMA-aware traffic scheduling, a key to the chip’s performance.

　　The design also focuses on reducing what users call tail latency, the time it takes the last packets of a flow to pass through the switch. The chip also packs a laundry list of optimizations for routing, load balancing, telemetry and other functions.