When I attempted to upgrade my study from 1 to 10 Gigabit networking, I quickly found that there is a world of non-Ethernet based home networking out there. In-fact, Ethernet is barely ever used as the central delivery mechanism for 10Gb, 25Gb and 100 Gigabit networks.
So in this video I run through exactly how the pros (network engineers) tend to achieve high speed networking within a home, office or other building. I also explore how internet-based data is moved longer distances, such as across cities.
If you prefer text over video, please read on for the guide/transcript version of this video.
Video Transcript And Guide
hey everyone, many of us will have started off using WiFi to connect our devices to the internet, but at some point we might have experienced unreliable or slow connections, and so we bought an Ethernet cable and a cheap unmanaged switch, and hooked it all up. BUT if you ever need to move beyond Gigabit networking to achieve 10 Gig, 25 Gig or even 100 Gigabit networking, you’ll quickly find that copper Ethernet cables just don’t cut it anymore – and there’s actually an entire world of non-copper networking cables and equipment out there.
Why Copper Ethernet Cables Fall Short
So it’s probably no secret that some form of fiber is used for higher speed networks, but why won’t “standard” Ethernet cables like this cut it at these higher speeds? Is copper the problem – since Ethernet cables are copper-based, after all? Well that’s the interesting thing. There has been various research and tests done that PROVE that copper cables can actually be faster than fiber – in some cases. Research from Arista shows that copper can have lower latency than fiber, especially at shorter distances. So WHY is copper-based Ethernet considered “too slow” for 10 Gigabit and above networking speeds?
Well, that research actually looks at DAC – direct-attach copper that uses a special form of copper-based cable to transmit internet data over shorter distances. I’ll come back to DAC later but Ethernet cables are DIFFERENT to DAC. Even the latest, highest quality CAT7 cables still ultimately use thinner copper wires than found in DAC, plus Ethernet cables have RJ45 connectors at both ends. If you simply look at an RJ45 connector, you can ‘kinda tell that it’s inherently limited. It doesn’t have much room to terminate the 8 really small wires, and so crosstalk will ALWAYS become an issue as networking speeds increase – especially if we’re talking of 100 Gigabit speeds, for example. The small design will also mean that EMI (electromagnetic interference) could also be more of an issue here.
Finally, copper IS just generally flawed as a method for long-distance data transfer – the signal will get weaker and weaker the further a cable is ran, which is known as attenuation. Both Ethernet and DAC work well at SHORTER distances, but they will struggle at longer distances. So if you’re planning to fit high speed networking throughout an office block, Ethernet just wouldn’t be a viable option.
Exit Light, Enter Night
That’s where fiber cables come in. They transmit data through pulses of light that “bounce” through the thin “glass” strand that actually runs throughout the entire fiber-optic cable. Because light moves very quickly, fiber can be a quick way of moving internet-based traffic from one source to another – such as from an internet street cabinet into someone’s home (which we call FTTP in the UK), or from a private internet carrier to a big data center, for example.
Because they are non-metallic, fiber optic cables are also immune from electromagnetic and radio interference, and while attenuation is always inevitable, there’s much LESS signal drop-off with fiber cables than copper-based cabling (like Ethernet or DAC). As a result, fiber optic is the perfect choice for the “backbone” of a network.
But it does have some flaws too. Fiber is often more expensive than copper cables, and it’s also a bit more brittle. Plus people’s computers don’t have fiber ports (in general), meaning that there often needs to be a ‘conversion’ from fiber to Ethernet SOMEWHERE along the line, which introduces a little bit of latency.
DAC FTW PLS?
I’ll discuss HOW 10 Gigabit (and beyond) networking is achieved in a minute, but firstly I wanted to back up discuss DAC more because I mentioned this at the very start of the video, then I stopped talking about it. Direct-attach copper uses thicker, higher quality copper (than Ethernet), and also the cables (and connectors) are all designed in such a way that there is less crosstalk and signal degradation than with Ethernet. Plus remember that I said that fiber will need to be “converted” somewhere along the line? Well with DAC, it’s already copper-based so there’s no need to have electrical (copper) to optical conversion – leading to lower latencies compared to fiber. What this all means is that for a typical household, it can be ideal to run DAC everywhere because while DAC IS limited to 15 metres or so, it’s unlikely that this will be exceeded in a home environment – especially if you’re just running from downstairs to upstairs, or from one room to another.
However it certainly wouldn’t be possible to use this for transferring loads of internet traffic from one part of a city to another.
How 10 Gigabit Networking Is Achieved
So now we come to HOW 10 Gigabit networking is often achieved. I actually recently upgraded my study to 10 Gigabit via Ethernet (which I show off in another video), and I kinda regret that now because while you CAN use Ethernet for 10 Gig speeds – which is known as 10GBASE-T – it’s often not advised. Copper-Ethernet runs HOT at 10 Gig speeds – my Ubiquiti 10 Gig switch is constantly warm to touch, and the whole set-up consumes a fair bit more power than it would have if I had gone with DAC or fiber-optic. A great post on the TrueNAS forums mentions that Ethernet can use as much as 10 times the power that SFP+ will. Plus 10 Gig Ethernet switches are MUCH more expensive than SFP+ switches. So all-in-all, if you want to achieve 10 Gigabit networking (either at home, or in an office) you’d probably be better off with an SFP+ based solution.
And YES, I realize that I just said “SFP+” three times, but I haven’t yet discussed what this is. So SFP+ is a type of connector, meaning that it’s a rival for RJ45. While many of the network switches we’re used to accept RJ45, when we’re dealing with high speed networks, you’ll actually start seeing SFP ports instead.
SFP+ is rated for 10 Gigabit speeds, and it’s the most typical way that 10 Gigabit networking is achieved. So basically you would have an SFP+ switch, and your key machines (like your servers or workstation PCs) would also have SFP+ network cards via the PCIe slot. So you could THEN run SFP+ cables all around your house, right?
Nope. There’s actually no such thing as an SFP+ cable. It would be a bit like saying an RJ45 cable. You PROBABLY mean Ethernet, but it’s not quite saying the right thing. In this context, when people discuss SFP+ cable, they often mean DAC actually – because direct-attach copper cable can come with SFP+ connector prefitted, meaning you can buy DAC and easily hook it up to your switches and network cards – delivering fast, reliable, low power and low-latency 10 Gigabit networking – which is great.
But that’s not the only way of “getting” SFP+. You can buy transceivers that allow you to plug a cable into them, and then this “converts” the cable to SFP+ – and you can then plug THIS into the network switch. And at 10 Gigabit speeds, you can actually buy Ethernet transceivers – meaning that you could “convert” a CAT7 cable to have an SFP+ connection (for example). The downside here is that – again – Ethernet runs very hot at 10 Gigabit speeds, and it continues to be power hungry too. Simply plugging it into an SFP+ transceiver doesn’t redeem Ethernet, unfortunately.
When Copper Ethernet IS Used For 10/25/100Gb Networks
So the way that network engineers would USUALLY achieve 10 Gigabit networking is by running a mix of fiber-optic and DAC around a home, office or building, and they would mainly use SFP+ switches and other networking gear – using transceivers as needed. But when it comes to standard “end user” machines, a conversion still needs to be made, right? I mean, we can’t install SFP+ cards into every computer. That’s why you often see higher-speed Ethernet switches that have one or two SFP+ ports – these are designed to take a higher speed fiber or DAC connection, and then the other ports are often 1 Gigabit or 2.5 Gigabit Ethernet – where you would then run a standard copper RJ45 cable (Ethernet cable!) to the end user machines, and also to a Wi-Fi router for devices to connect wirelessly in the normal way (of course).
So going back to the title of this video, it’s not 100% true to say that network pros simply don’t use Ethernet (of course) because it is used AT SOME POINT along the line, but it’s certainly not the central, pivotal backbone for high speed networks – to be honest.
Single-Mode Fiber (SMF) Vs Multi-Mode Fiber (MMF)
Awesome, so that’s how the pros would do it WITHIN a building – like a school, home, office or something like that. But what about moving internet data from one building to another, or across a city? While major ISPs would deal with (at least) 100 Gigabit backlinks, but some smaller organisations might only work with 10 Gigabit.
Well we know that copper-based solutions are OUT due to attenuation problems, so that leaves fiber. And in general there’s two types of fiber delivery we need to know about here: multi-mode fiber (MMF) and single-mode fiber (SMF). SMF cables have a fairly narrow core that only allows for a single light mode, whereas (as its name suggests) MMF supports multiple light modes due to its wider core. Logically you’d think that this means that MMF is “better” and, well, it is FASTER – but it also has more light dispersion due to its core design. And as a result, MMF is often used for shorter distances – within buildings itself, or maybe from one office building to a neighbouring office building (but up to around 300 meters but no more) – whereas SSF is best for transmitting internet data from one city to another (for example).
Once the cable run is complete, the only real difference would be the transceiver used – SSF needs 10GBASE-SR transceivers (where SR stands for short range), while MMF needs 10GBASE-LR (or sometimes ER – extended range – ones).
25Gb and 100Gb Networking
So everyone has 10 Gigabit networking and we can relax and go home, right? Well no because – obviously – some organizations will need more than this. 25 Gigabit and 100 Gigabit networks are the most common that you would see beyond the 10 Gigabit level.
To be honest though, everything is ‘kinda the same here – but just at a different scale. Costs start to go up massively, with “data center” switches being needed throughout the network deployment (the network stack) and these aren’t cheap. For example a used Arista 7060CX switch (which supports 100 Gig) costs over £2000 here in the UK. It also needs TWO 745 watt power supplies to handle the massive power load that it will inevitably need – although it’s worth noting that one of these is for redundancy, the device “only” uses a mere 745 watts.
You also still have SFP plugs the majority of the time, but we have moved on from 10Gig SFP+ transceivers to SFP28 and QSFP28 for 100 Gig. You can also get SFP28 and QSFP28 DAC cable for shorter runs of 3 to 5 metres, but it can’t support longer distances than this due to the limitations of copper that we discussed earlier.
So a network engineer at these higher networking requirements would still typically run fiber around a building (especially for longer runs), using 25GBASE or 100GBASE transceivers as needed, to connect to SFP ports, but they would also sometimes use DAC for very short runs.
I find it pretty interesting that how home networking moves away from Ethernet cables beyond 2 and a half Gig speeds, but it kind of makes sense actually. I had various issues when trying to upgrade JUST this study to 10 Gigabit speeds, as I show off in THIS other video if you wanted to check that out.
If you have any questions, feedback or suggestions about this article, please leave a comment below. Please note that all comments go into a moderation queue (to prevent blog spam). Your comment will be manually reviewed and approved by Tristan in less than a week. Thanks!