Optimal Networks IT Consulting - We Make Sure You Are Always ON!

This is the first of a two-part series on data center cabling. Read the second tip on network cable types for your data center.

When you walk into most data centers, the first thing admins apologize for is the cabling mess. It should be simple -- just plug in a cable between the server and the network switch. Yet "spaghetti" hangs out of cabinets, blocks the air under raised floors and drapes overhead like tangled snakes. We know how it got this way, but we're not sure why. Is this chaos an inevitable result of ever-changing technology? More importantly, is there any way to do data center cabling correctly?

This two-part series addresses three major challenges of deciding on and implementing a data center cabling infrastructure: topology, quantity and cable type.

Data center cabling topology
Cabling a data center is different from cabling a building. Cable lengths are shorter, performance demands are higher and instead of the two, three or four terminations common to most offices, a single cabinet can require 24, 48, 96 or even more connections in copper and fiber.

Three fundamental methods are used for data center cabling:

1. Point-to-point
2. End-of-row consolidation
3. Top-of-cabinet consolidation (using fiber and/or switches)

Let's eliminate point-to-point cabling right away. That's how data centers have been wired for years, causing messes we still see. Point-to-point means running individual cables or patch cords under the floor, overhead (with or without cable trays) or through the racks and cabinets for every connection needed. Cables are often made on site, or any available patch cord is used. Old cable is rarely removed or labeled, making it very difficult to be traced or even found, resulting in the "rat's nest" familiar to data center workers. Servers and patch panels create enough cable chaos, but when high density network switches are installed in shallow, legacy cabinets, the mess is worse. In short, point-to-point just doesn't work anymore.

Considering end-of-row cabling
End-of-row consolidation, or "area clustering," has been the data center cabling method of choice for several years. In this approach, a rack is designated for patch and consolidation switching, usually at the end of each cabinet row -- sometimes in the middle -- to serve a group of cabinets. Patch panels are installed in each cabinet, and permanent cable is run from those panels to matching panels in the consolidation rack(s). Another set of cables, usually more fiber-intensive, is also installed from the consolidation racks to the core network location, a.k.a. the main distribution frame or area. Think of it as compressing the standard "star topology" of a high-rise building into the data center. Server cabinets are like the offices and consolidation racks are the intermediate distribution frames (IDFs) or telecom rooms (TRs).

End-of-row consolidation also makes adding new hardware easy; just plug in short patch cords from each server to its cabinet patch panel and then plug short cords into the server access switch in the consolidation rack. Only two short patch cords are needed for each connection, and they're easy to install and find later. Similarly, the access switch is patched to the core network via the fiber cable, so the entire installation involves just simple patching. "End-of-row" consolidation minimizes the number of patch cord lengths that need to be stocked, and makes documentation straightforward since every panel is labeled and easy to trace, particularly if patch cords are color coded. Furthermore, because few cord lengths are needed, there's no excuse for leaving old, long, lower-performance cords lying around. This helps avoid using incompatible patch cords that degrade system performance.

Weighing the top-of-cabinet route
Top-of-cabinet consolidation is the newest approach, but can be viewed as a variation of end-of-row. Top-of-cabinet depends on server configuration, so we'll discuss two related options which use the same fundamental data center cabling infrastructure. With a cabinet full of servers, the first option is to install smaller access switches in each cabinet in duplicate if the servers have dual network interface cards. Sometimes, a third lower-grade switch is also installed for out-of-band management connections. The switches are then patched back to the core network via a fiber infrastructure similar to end-of-row's. So the top-of-cabinet principle is the same as end-of-row, but is more readily scalable -- less expensive switches can be added in each cabinet as needed and can even be moved with reconfigurations. A small switch failure also has less impact than when a large switch is used for a whole row.

The alternate approach is used when blade servers include their own chassis switches. In this case, we need only the fiber connectivity back to the core. So the same fiber infrastructure from each cabinet can support either consolidation option. There might even be "hybrid" installations where the entire chassis is still consolidated through the on-board switches, but you still want individual connections from each blade. In short, the common fiber infrastructure can serve any high-speed design.

Picking the most appropriate data center cabling method
Choosing between end-of-row or top-of-cabinet depends on processing density, application throughput requirements and hardware connectivity specifics. In a larger data center, both designs could prove useful in different parts of the facility. In short, there's no single answer, but the popular choice leans toward fiber, and either cabling infrastructure is certainly preferable to the "ad-hoc" connection approach that is the nemesis of so many operations.

The storage network, which has used fibre channel connectivity for years, has generally been cabled similarly to top-of-cabinet. Fiber runs from cabinet patch panels to matching patches for storage director switches in the storage area network (SAN) area of the data center. But this requires a second complete cabling infrastructure from each cabinet -- not particularly cost-effective.

The newest direction is Fibre Channel over Ethernet (FCoE), which enables SAN connections utilizing the same topology and fiber paths as the transport network. FCoE can reduce physical cable quantities, increase bandwidth requirements and consolidate network and SAN management. As FCoE is still emerging, not every manufacturer's equipment is compatible. And, as with the network cable approach, deciding how to cable storage is dependent on the hardware connectivity requirements and other factors. It is likely, however, that FCoE will be supported universally soon.