Flying blind at 100G – the monitoring industry’s dirty little secret.

Posted October 22nd, 2012 by Tim Nichols

According to the latest data from Infonetics, 100G networking is gathering real momentum, and not just amongst the telcos. If you look at the enterprise messaging from any of the big infrastructure vendors they’re all pushing 40G and 100G switching systems that are capable of moving vast amounts of data around the datacenter.  It’s worth noting as well that 100G is not just limited to the data center; ESnet (Internet 2) earlier this year announced they’re connecting research facilities together over the public Internet at 100G to help researchers move large amounts of data around.

100 Gigabits of data (or 12.5 Gigabytes) is a lot of data, that’s for sure. But how much is it in reality? Using some very crude math, in 1 second, a fully loaded 100G link should be able to transmit any of the following:

4 Full length Avengers movies
4167 Lady Gaga songs
41667 Kindle ebooks
178571 Simultaneous ‘Breaking Bad’ NetFlix streams
1250000 simultaneous skype phone calls
89285714 ‘LOL’ text messages


100G is not for everyone for very obvious reasons, but big banks, research institutions, telcos, service providers and organizations serving the media and entertainment sector appear to be the early winners. The economics of 100G networking are actually very compelling when contrasted to 10 Gbps networking. Over optical fibre a 10 Gbps link consumes a whole glass strand and uses one connector;  100G delivered using SR-type 10 Gbps optics  consumes 10 whole strands bonded together but only one connector. LR/ER type 100G technology is even more efficient, consuming one strand of glass and one connector, making it ten times more efficient. So if you’re moving a LOT of data around then 100G is a perfectly rational choice.

Unfortunately, deploying 100G and living with 100G are two different things. From an IT ops perspective, 100G network segments are just like any other network segment; they need to be monitored, analyzed and recorded so that issues can be detected and investigated before end users get involved.

Unlike 10 Gbps network segments which can be matched to a 10 Gbps monitoring port on an IDS or analytics platform directly there is no such thing as a 100G monitoring system. Nada. Niet. Nuffin. The dirty truth here is that the monitoring industry has been caught napping. Today, anyone operating a 100G network is flying blind. You’d never drive your car blindfolded, so why would you run your network that way?

In fact it’s not the first time this situation has arisen; back in 2008 when 10 Gbps arrived with a vengeance most organizations only had 1 Gbps monitoring infrastructure and IT ops teams faced a similar problem. The answer back in 2008 came in the form of a layer 1 matrix switch, which Gartner tidily named Network Packet Brokers earlier this year. NPBs solved the problem almost over night by ingesting 10 Gbps of traffic and load balancing it out over multiple 1 Gbps ports which could be connected to existing 1 Gbps capable infrastructure. Over the last 5 years this market has evolved to become a $250m industry helping organizations access, filter, load balance and duplicate their 10 Gbps and 1 Gbps network traffic.

So the question is…. “why hasn’t history repeated itself at 100G”  Well, in due course it most likely will, but there’s a couple of reasonably major issues that the monitoring and NPB vendors have to step up and deal with.

Firstly, 100G is a LOT more complicated to deal with than 10 Gbps. Today’s NPB market is based on commodity merchant silicon (re-purposed from that found in standard Ethernet switches) which is perfect for the basic task of moving 10 Gbps traffic around, but isn’t going to scale to meet the demands of 100G. It’s going to take a different architectural approach potentially using a different technology to meet the 10X increase in throughput and it’s appears to be beyond the scope and capability of most, if not all of the current set of NPB vendors

Secondly, there’s a fundamental problem in the 10 Gbps tools market that’s going to bite in a 100G world. The ugly truth is that very few, if any of the monitoring and security tools on the market today that support 10 Gbps ports are actually capable of operating at 10 Gbps for any sustained period of time. 10 Gbps performance might range from 1 Gbps to 5 Gbps, but rarely gets anywhere close to 10.  At some point above 2 Gbps traditional software based packet capture technologies on which most monitoring and security tools are built start dropping packets.

The issue of packet loss is being largely ignored by most organizations and vendors today, but will become front of mind for when it comes to 100G monitoring because the types of companies that buy 100G really care about this stuff and will ask the tough questions that most people have been dancing around for the last 5 years. Maybe 100G will finally force the issue of 10 Gbps packet loss?