Interop Las Vegas – 72 billion packets of network traffic. And a hangover.

Interop Las Vegas is over for another year, and as we recover from the excess that comes with every event that happens in Vegas, we’ve had a chance to look at the data captured from the pair of EndaceProbe appliances deployed on InteropNet. And they tell a fascinating story about what folk do when they’re not listening to vendor pitches!

For anyone not familiar with InteropNet, it’s the network that provides the 18,000 show visitors and 285 vendors that exhibit with connectivity. Given the nature of the show, it’s become a showcase in its own right for the very latest networking, security and monitoring products. We’re proud to have been invited to deploy our EndaceProbe appliances in the network analysis and forensics product category.

Our EndaceProbe appliances, with 10Gb Ethernet (10GbE) interfaces and 64TB of local storage, were deployed so that they could see, capture and record every packet on the network. Between Tuesday at 4:00 p.m. and noon on Thursday, the EndaceProbe appliances recorded an incredible 72 billion packets. The dropped packet counter on the EndaceProbe recorded zero packet loss, so when we say that 72 billion packets traversed the network, we really mean 72 billion packets traversed the network and we captured every single one to disk. Those 72 billion packets translate to:

  • 68GB of metadata that can be used to generate EndaceVision visualizations.
  • 6.1TB of packet data that can be retrieved through EndaceVision in a few short clicks for a full payload investigation in packets, or Wireshark.

Users of the network consumed more than 130GB of iTunes traffic (7th highest on the list of application usage) and 100 GB of bit torrent (10th highest on the list). Whether vendors should be taking this as an insight into how interesting their presentations are is an interesting question in its own right!

On the network itself, the average bandwidth utilization was just 350 Mbps, however, what’s interesting – and what few organizations understand given the lack of fidelity in their monitoring tools – is that the network was regularly bursting to 8Gbps and had frequent spikes of 10Gbps (measured in the millisecond range).

The ability to see traffic spikes at such a low level of resolution is critical for understanding the behavior of the network and planning for the future. With the wrong tools, you could easily be mistaken to thinking that a 1Gbps link would be sufficient to handle InteropNet traffic. But you’d be very wrong indeed….

An interesting anecdote from the NOC that highlights the power of EndaceVision came from an escalation inside the NOC that the show wifi was slow. In a few clicks, we were able to show that the problem was coming from a single user (Silvio, we know who you are!) who decided to download more than 300GB of data over the network and saturate the resource.

So, until next year, we bid Las Vegas farewell and head home for a well deserved rest.

 

Why you should never trust a fox to guard your hen house

Based on high-profile announcements from a number of vendors over the last few months, there appears to be general agreement that recording your network traffic in order to help analysts and engineers diagnose and troubleshoot network and security problems is a good idea.

With this frame of reference set, it’s worth taking a bit of time to think about HOW you should go about recording network traffic to maximize your chances of connecting an analyst to the network history that he or she needs to investigate a particular problem or ‘event.’

Thoughts on Obama’s executive order on cyber security.

It was reported yesterday that US President Obama issued an executive order on cybersecurity ahead of his State of the Union speech.  One more bit of evidence that it’s important to know what’s on your network.

A couple of interesting notes from the text of the order:

Critical infrastructure and utility cyber security update

As a global supplier of searchable network intelligence infrastructure, Endace has long been a strategic partner to many large government, telco and enterprise organizations.  More recently we have seen a number of design wins in other critical infrastructure areas, most notably with utilities’ Industrial Control Systems (ICS).  As a general rule, Endace network intelligent network recorders are being deployed by electric utilities to solve critical issues relating to

  • Grid operations
  • Security
  • Regulatory compliance

Network traffic retention best practice

How long should I store packet captures?  How much storage should I provision to monitor a 10Gbps link?  When is NetFlow enough, and when do I need to capture at the packet level?

These are questions network operations managers everywhere are asking, because unfortunately best practices for network data retention policies are hard to find.  Whereas CIOs now generally have retention policies for customer data, internal emails, and other kinds of files, and DBAs generally know how to implement those policies, the right retention policy for network capture data is less obvious.

The good news is that there are IT shops out there that are ahead of the curve and have figured a lot of this out.

The future of network visibility, as defined by Emulex Corp.

A few days on and the dust has started to settle on the announcement that Emulex plan to acquire us lock, stock and barrel. Since the news broke there has been a great deal of speculation about what it all means and there have been various attempts to explain the rush of M&A activity in the space. Tom Nolle from CIMI Corp penned a thoughtful piece last week looking for the deeper meaning in it all that’s worth a read. He has some interesting perspectives for sure. The reality here, in our humble opinion, is that the market for monitoring, management and analysis tools is simply an area where customers have real unmet needs, particularly in the 10Gbps space, and it is driving industry consolidation to achieve market leadership.

Hardware v’s software – Riverbed’s acquisition of Opnet.

Now the dust has settled over Riverbed’s acquisition of Opnet it’s worth taking a step back and considering some of the expert opinions that have been voiced since the deal was announced. NetWork World published a fascinating article on the 1st of November that canvassed the opinions of a number of respected analysts. The full article can be found here.

Before considering any industry analyst opinion, it’s worth bearing in mind that Wall Street made its views on the acquisition very clear. It’s difficult to see a 20% drop in share price as anything other than a vote of little confidence in the deal.

Thanks, Extrahop, for the validation

We’ve written before about the difference between prevention, detection, response and root-cause.  In a nutshell: in-line devices like firewalls and WAN optimization boxes prevent bad stuff from happening; fault management, APM and SEM tools detect when bad stuff happens despite your best efforts; and then when a detection tool alerts that something is unusual, IT professionals have to do something about it – with a quick response, ideally followed by a full root cause analysis & permanent corrective action.

Some issues that are alerted by detection devices are obvious; but many are not.  So a very common approach to resolving the issues raised by detection tools is to take a guess at what the cause might be, then make a change that you hope will fix it.  If the problem goes away, you then have to assess whether it went away because of your fix or just by coincidence.  Chronic/intermittent issues are particularly difficult to deal with this way.

Flying blind at 100G – the monitoring industry’s dirty little secret.

According to the latest data from Infonetics, 100G networking is gathering real momentum, and not just amongst the telcos. If you look at the enterprise messaging from any of the big infrastructure vendors they’re all pushing 40G and 100G switching systems that are capable of moving vast amounts of data around the datacenter.  It’s worth noting as well that 100G is not just limited to the data center; ESnet (Internet 2) earlier this year announced they’re connecting research facilities together over the public Internet at 100G to help researchers move large amounts of data around.

Why you need to know EXACTLY which applications end users are playing with.

Do you know which applications are running on your network? Well, if you’re anything like the 40% of the Fortune 500 customers we recently surveyed then there’s a good chance that you don’t. According to the survey, not only do large organizations NOT have a good grasp on the applications end users are playing with, but at least 53% of organizations of them also have IT policies that preclude the use of certain applications. It doesn’t take an astronaut to figure out that somethings very wrong here, so we won’t labor the point.