For anyone not familiar with InteropNet, it’s the network that provides the 18,000 show visitors and 285 vendors that exhibit with connectivity. Given the nature of the show, it’s become a showcase in its own right for the very latest networking, security and monitoring products. We’re proud to have been invited to deploy our EndaceProbe appliances in the network analysis and forensics product category.

Our EndaceProbe appliances, with 10Gb Ethernet (10GbE) interfaces and 64TB of local storage, were deployed so that they could see, capture and record every packet on the network. Between Tuesday at 4:00 p.m. and noon on Thursday, the EndaceProbe appliances recorded an incredible 72 billion packets. The dropped packet counter on the EndaceProbe recorded zero packet loss, so when we say that 72 billion packets traversed the network, we really mean 72 billion packets traversed the network and we captured every single one to disk. Those 72 billion packets translate to:

• 68GB of metadata that can be used to generate EndaceVision visualizations.
• 6.1TB of packet data that can be retrieved through EndaceVision in a few short clicks for a full payload investigation in packets, or Wireshark.

Users of the network consumed more than 130GB of iTunes traffic (7^th highest on the list of application usage) and 100 GB of bit torrent (10^th highest on the list). Whether vendors should be taking this as an insight into how interesting their presentations are is an interesting question in its own right!

On the network itself, the average bandwidth utilization was just 350 Mbps, however, what’s interesting – and what few organizations understand given the lack of fidelity in their monitoring tools – is that the network was regularly bursting to 8Gbps and had frequent spikes of 10Gbps (measured in the millisecond range).

The ability to see traffic spikes at such a low level of resolution is critical for understanding the behavior of the network and planning for the future. With the wrong tools, you could easily be mistaken to thinking that a 1Gbps link would be sufficient to handle InteropNet traffic. But you’d be very wrong indeed….

An interesting anecdote from the NOC that highlights the power of EndaceVision came from an escalation inside the NOC that the show wifi was slow. In a few clicks, we were able to show that the problem was coming from a single user (Silvio, we know who you are!) who decided to download more than 300GB of data over the network and saturate the resource.

So, until next year, we bid Las Vegas farewell and head home for a well deserved rest.

With this frame of reference set, it’s worth taking a bit of time to think about HOW you should go about recording network traffic to maximize your chances of connecting an analyst to the network history that he or she needs to investigate a particular problem or ‘event.’

On the one hand, there’s a school of thought that recommends deploying an integrated detection AND recording solution (any vendor that gives you a dashboard of alarms to worry about and the ability to drill down to the traffic of interest falls into this integrated category). Some vendors in this category record all the traffic, and others just record short segments of traffic that pertain to events that they have detected.

The other school of thought, and the one that we subscribe to, advocates using a dedicated and physically separate recording infrastructure solution that continually records every packet with 100% accuracy. For sure, two appliances are always going to be more expensive than one, but the benefits of separation far outweigh the risks of integration. Here’s why:

Detecting bad things in modern high-speed networks generally involves parsing huge amounts of network traffic with rules and correlations and other clever algorithms. It’s essentially a software problem solved by software companies that don’t really care about hardware.

There are two primary factors that determine whether or not a detection system ‘alarms’ on a problem or not:

• Did it see the traffic that contained the problem?
• Did it have the right rules to generate an alarm in the first place?

If the answer to either of those questions is ‘no,’ then you’ve got a false negative. Any operations manager will attest that false negatives are the true enemy of both network and security operations departments because they don’t bite till later…

Now consider a situation where your integrated detection and recording solution is generating false negatives because it’s dropping packets at the interface. With 14 million packets per second on a fully loaded 10 Gbps link, the chances of a software-based detection system dropping packets is, in fact, extremely high. In this scenario, you’ve got no alarm AND (a false negative) no packet history, because the packets that were dropped at the interface didn’t made it into software or onto disk. Now when end users complain about the problem, engineers have no history to work with and are back to square one, scrambling for log files and anything else that might give them some insight into what happened.

False negatives that result from poor detection rules are marginally more palatable as the network traffic should be available for post-event forensics when someone raises the alarm. That is, unless of course you’re relying on a system that just records network traffic in response to a detected alarm – in which case you’re right back at square one.

Now consider a physically separated solution with a best-in-class detection engine deployed alongside a best-in-class network. If the detection engine fails to alarm for any reason, it doesn’t matter quite as much, as a 100% accurate record of every packet exists on a separate appliance. It may not be quite as quick to get to the packets of interest on a separate appliance, but the benefits of knowing that the history actually exits far outweigh the additional time required to find it.

The other benefit of separation is that the recorded traffic can be treated as a shared resource that can be mined by multiple different teams and can be used in conjunction with multiple detection systems. It’s also worth noting that in the network operations domain, most of the events that engineers deal with are not generated by detection systems at all, but from end users calling in to complain about outages, poor performance, etc. With this knowledge, the case for dedicated and physically separate recording appliances grows even stronger.

The reality is that ‘detection’ and ‘recording’ are very different technical competencies and, for the reasons outlined shouldn’t be mixed under any circumstances. Large organizations have historically invested in best-in-class technologies and should recognize recording and detection as two very different ‘classes’ of technology.

So…. if you’re relying on your detection vendor to provide engineers with network history in a single integrated appliance, it’s quite possible that you have what amounts to a fox guarding your hen house. And when all the lights go out, the one thing you don’t want is a fox in your henhouse.

A couple of interesting notes from the text of the order:

• It addresses both the public and the private sector.  The focus is on “critical infrastructure” which is defined very broadly.
• It requires agencies to “ensure the timely production of unclassified reports of cyber threats to the U.S. homeland that identify a specific targeted entity.”  Great recognition here that really understanding what happened yesterday is important to making sure we can protect against it tomorrow.

Some commercial enterprises with mission-critical networks are ahead of the curve on this, but many are not.  It’s rare that the government gets out ahead on technology issues – CIOs and CISOs who are not at least as far along as this Executive Order need to be playing catch-up.

• Grid operations
• Security
• Regulatory compliance

We felt that some current stories and reports concerning this industry would be of interest to our readers.

First, the recent US DHS (Department of Homeland Security)  ICS-CERT report (http://www.us-cert.gov/control_systems/pdf/ICS-CERT_Monthly_Monitor_Oct-Dec2012.pdf) is an excellent “canary in the coal mine” document and appropriately makes the case for focusing more  attention on the state of cyber security for ICS in general, and particularly for critical infrastructure elements, such as the electric grid. (We see an increasing need for this type of cyber security planning in many industries today – including industrial control.)

Two articles last week caught our attention, focusing on different aspects of the document . First, CNN (http://money.cnn.com/2013/01/09/technology/security/infrastructure-cyberattacks/index.html) looked at  the report’s statistics that showed  increased numbers of attacks on ICS networks, especially on the energy sector, while Threat Post (https://threatpost.com/en_us/blogs/shodan-search-engine-project-enumerates-internet-facing-critical-infrastructure-devices-010913) led with a private survey that discovered internet-connected SCADA (Supervisory Control and Data Acquisition) and ICS systems to assess the potential problem, then cross-referenced  the ICS-CERT report to look at  some specific instances. A subsequent post (https://threatpost.com/en_us/blogs/malware-infects-two-power-plants-lacking-basic-security-controls-011413) then examined two successful attacks on systems controlling power generation mentioned in the ICS-CERT report.

Of course, ICS security is a complex issue. Utilities must continuously assess and re-assess risks and threats, and use those assessments to guide ongoing investment across four key areas

• Detection,
• Prevention,
• Response
• Root cause analysis

It seems clear — and is looking clearer each day — that no matter how much (or how little) you invest in threat detection and prevention, attackers will get through. While this might appear to be a BGO, or Blinding Glimpse of the Obvious, until relatively recently, doubling down on the investment in these areas was pretty much the only option on the table.  But that is no longer true. Working off of a complete and accurate data capture record, our visual network traffic search engine provides foundational event response/containment and permanent remediation capabilities for these networks.   Such a balanced approach of prevention, detection, response and root cause also aligns nicely with the layered defense-in-depth methodology of the  DHS CSSP Recommended Practice (http://www.us-cert.gov/control_systems/practices/documents/Defense_in_Depth_Oct09.pdf).

As we have previously discussed (http://blog.endace.com/2012/09/exposing-the-biggest-security-challenges-facing-large-enterprises-today/), security operations teams face numerous obstacles, even in enterprises that are not encumbered by 30-year use-life expectancies, the uniquely challenging world of  many utilities today. These include  lengthy response times caused by lack of network visibility and an overwhelming number of  tools and alarms. We feel that the logical course is not to double down on technologies that, at best, will provide nominal improvements to failure rates and response times, but to look to new  types of investments  that not only leverage existing tools, but enable security and event response professionals to be significantly more productive.

Pike Research, in their Utility Cyber Security report, highlight a few possible “best practices” for utilities to adopt, including

• Control network isolation when possible (and DMZ where not – we would add network recording to that suggested best practice for such critical connections)
• Application whitelisting (which is enhanced by having DPI capability in your network recording fabric (standard with Endace))
• Security event logging and  correlation across both infrastructure (network headers) and application data (i.e., you need full packet capture with search and other advanced capabilities if you want timely event reconstruction, root cause analysis and response).

Root cause analysis and event response are clearly complex, layered tasks. A network recording system is a fundamental enabler for addressing both. Look for our whitepaper with more on these layers soon. Both in reports and our daily engagements, we see that intelligent network recording can enhance security practices and reduce time to response.

While some utilities and their vendors may be waiting either for regulatory action and/or a catastrophic event closer to home than Stuxnet or Shamoon, we believe now is the time to act.  Network traffic recording of the likes of Endace’s make even APT (Advanced Persistent Threat) attacks such as Stuxnet more problematic to perpetrate as the final stage of an APT is to cover tracks and remain undetected, which is difficult if the attack footprint has been recorded and time stamped.

Fortunately some US and global utilities are leading the way. Some are adopting advanced network recording fabrics which greatly enhance the efficiency of their security operations team and reduce risk exposure, while others focus on reducing time to resolution for grid (SCADA, etc.) operations, improving safety and reliability, and still other teams benefit by enhancing compliance. Another effort worth noting is led by Southern California Edison (http://resnick.caltech.edu/programs/seminars/Gooding_Grid2020.pdf), which is developing a central cyber security services capability to help share more advanced security infrastructure across formerly “siloed” control systems.

So, while the news still speaks of far too many vulnerabilities, we see the tide starting to shift. Leaders are making more balanced investments in root cause analysis and event response, and these investments are improving critical infrastructure safety and reliability while reducing risk.

Do you think utilities and other critical infrastructure operators have sufficient security event response, or are moving towards that goal at an appropriate pace? Can we secure legacy systems in place, or will we have to replace them and/or wait out their use-lifetimes? We’d like to hear your thoughts, stories and best practices…

For more information contact enquiries@endace.com

These are questions network operations managers everywhere are asking, because unfortunately best practices for network data retention policies are hard to find.  Whereas CIOs now generally have retention policies for customer data, internal emails, and other kinds of files, and DBAs generally know how to implement those policies, the right retention policy for network capture data is less obvious.

The good news is that there are IT shops out there that are ahead of the curve and have figured a lot of this out.

Background

To begin with, it’s important to clarify for your own organization what the goals are for network history.  Some common answers include:

1. Respond faster to difficult network issues
2. Establish root cause and long-term resolution
3. Contain cyber-security breaches
4. Optimize network configuration
5. Plan network upgrades.

You may notice that the objectives listed above vary in who might use them: stakeholders could include Network Operations, Security Operations, Risk Management, and Compliance groups, among others.  While these different teams often operate as siloes in large IT shops, in best-practice organizations these groups are cooperating to create a common network-history retention policy that cuts across these siloes (and in the most advanced cases, they have even begun to share network-history infrastructure assets, a topic we discussed here).

Some of your objectives may be met by keeping summary information – events, statistics, or flow records for example – and others commonly require keeping partial or full packet data as well.   A good retention policy should address the different types of network history data, including:

• Statistics
• Events
• Flow records – sampled
• Flow records – 100%
• Enhanced flow records or metadata (such as IPFIX, EndaceVision metadata, etc.)
• Full packet data – control plane
• Full packet data – select servers, clients, or applications
• “Sliced” packet headers – all traffic
• Full packet data – all traffic.

Generally speaking, the items at the top of the list are smaller and therefore cheaper to keep for long periods of time; while the items at the bottom are larger and more expensive to keep, but much more general.  If you have the full packet data available you can re-create any of the other items on the list as needed; without the full packet data you can answer a subset of questions.  That leads to the first principle: keep the largest objects (like full packet captures) for as long as you can afford (which is generally not very long, because the data volumes are so large), and keep summarized data for longer.

Next, you should always take guidance from your legal advisor.  There may be legal requirements arising from regulation (PCI, Rule 404, IEC 61850, etc.), e-discovery, or other sources; this article is not meant to be legal advice.

Now that said, in the absence of specific legal requirements that supersede, here are the best practices we’re seeing in the industry.  Working the list from bottom to top:

Packet data for all traffic: 72 hours

• Full packet data or “sliced” packet headers? The choice here will depend on how tightly controlled your network is and on what level of privacy protection your users are entitled to.  For highly controlled networks with a low privacy requirement, such as banking, government or public utilities, full packet capture is the norm.  For consumer ISPs in countries with high privacy expectations, packet header capture may be more appropriate.  General enterprise networks fall somewhere in between.
• Whichever type of packet data is being recorded, the goal consistently stated by best-practice organizations is a minimum of 72 hours retention, to cover a 3-day weekend.
• For the most tightly-controlled networks retention requirements may be 30 days, 90 days, or longer.

Packet data for control plane & for select traffic: 30+ days

Control plane traffic can be extremely useful in troubleshooting a wide variety of issues.  It’s also a type of traffic that is owned by the network operator, not the customer, so even networks that don’t record all traffic should keep history here.

Traffic types of interest include for example:

• Routing protocols (OSPF, IS-IS, EIGRP, BGP; plus  protocols like RSVP, LDP, BFD, etc. in carriers)
• L2 control plane (ARP, spanning tree, etc.)
• ICMP
• DHCP
• DNS
• LDAP, RADIUS, Active Directory
• Signaling protocols like SIP, H.225.0, SCCP, etc.
• GTP-C in mobile networks

In addition to control plane traffic, in every network there are particular servers, clients, subnets, or applications that are considered particularly important or particularly problematic. For both control-plane and network-specific traffic of interest, organizations are storing a minimum of 30 days of packet data. Some organizations store this kind of data for up to a year.

Flow records @ 100%: 120+ days

• Best-practice organizations record either enhanced metadata (such as that collected by EndaceVision), or at least basic NetFlow v5/v9/IPFIX.
• This flow data is useful for a wide variety of diagnosis and trending purposes.  Although a few router models can generate flow records on 100% of traffic, best-practice is to separate this function onto a separate probe appliance connected to the network via tap, SPAN or matrix switch.  The probe appliance both offloads the router/switch and also enhances flow data with DPI / application identification information.
• Best-practice here is to store at least 120 days of flow data.  (We have seen organizations that keep 100% flow records for as long as seven years.)

Samples and summaries: 2 years or more

• sFlow or sampled NetFlow, using 1:100 or 1:1000 packet samples, can be useful for some kinds of trending and for detecting large-scale Denial of Service attacks.  There aresignificant known problems with sampled NetFlow, so it’s not a replacement for 100% flow, but it does have usefulness for some purposes.
• Summary traffic statistics – taken hourly or daily, by link and by application – can also be helpful in understanding past trends to help predict future trends.
• Because this data takes relatively little space, and because it is mostly useful for trending purposes, organizations typically plan to keep it for a minimum of two years.
• One point to remember in maintaining history over periods of a year or longer is that network configurations may change, creating discontinuities.  It’s important to record every major network topology change or configuration change alongside your traffic history data, so you don’t compare incomparable data and draw the wrong conclusions.

Average vs Peak vs Worst-case?

One challenge faced in sizing network-history storage capacity is the fact that well-designed networks run well below 100% capacity most of the time, but in times of stress (which is when network history is most valuable) they may run much hotter.  Should you size for 72 hours of typical traffic, or 72 hours of worst-case?

The best-practice we’ve seen here is to make sure your network history system can capture at worst-case rate, but has enough storage provisioned for typical rate.  The reasoning here is that when the network gets very highly loaded, someone will be dragged out of bed to fix it much sooner than 72 hours, so a long duration of history is not needed; but that person will want to be able to rewind to the onset of the event and will want to see a full record of what was happening immediately before and after, so having a system that records all traffic with zero drops is crucial.

Here’s an example to make it concrete:

Suppose you have a bi-directional 10Gbps link that averages 1Gbps in each direction over a 24-hour period, and 3Gbps in each direction over the busiest hour of the day.

Then 72 hours of full packet storage at typical load would require

(2Gbit/sec x 72 hours x 3600 sec/hour / 8 bits/byte)

= 6480000 Gbytes, or about 64 terabytes.

Under worst-case load, when recording is most important, it could run at the full 10Gbps, which would fill storage 10 times as fast.  The good news is: best-practice here says you do not need to provision 10x the storage capacity, but you should be using a capture system that can record at the full 10Gbps rate with zero loss.  That means that in a worst-case scenario your storage duration would be more like 7 hours than 70; but in that kind of scenario someone will be on the case in much less than 7 hours, and will have taken action to preserve data from the onset of the event.

Of course, the same considerations apply for other types of network history: systems need to be able to process and record at the worst-case data rate, but with reduced retentionduration.

Other considerations

The above discussion slightly oversimplifies the case; there are actually two more important considerations to keep in mind in sizing storage for network history.

First, most recording systems will store some metadata along with packet captures, and this adds some overhead to the storage needed – typically around 20%, though it may vary depending on the traffic mix and on the recording product you use.

Second, while we say above you should provision storage for typical load, most organizations actually use projected typical load, extrapolating the traffic trend out to 18-36 months from design time.  How far ahead you look depends on how often you are willing to upgrade the disks in your network recording systems.  A three-year upgrade cycle is typical, but with disk capacity and costs improving rapidly there are situations where it can be more cost-effective to provision less storage up front and plan to upgrade every 24 months.

Implementing the policy

When organizations first take on the challenge of standardizing network-history retention policy, they nearly always discover that their current retention regime is far away from where they think it needs to be.

Typically we have seen that implementing a best-practice retention policy happens in six phases:

1. Create the “idealized” policy describing where you want to be, without regard to current state
2. Inventory the current state and identify how far off it is from the ideal
3. Set targets for 3-6 months, 12 months and 24 months
4. Over the 3-6 month horizon, take low-hanging fruit by reconfiguring existing systems to optimize for the new policy, and identify what new technologies will be needed to achieve the chosen retention policy
5. Over the 12-month horizon, pilot any new technologies that may be required to achieve the long-term policy
6. Over the 24-month horizon, roll out these technologies network-wide.

Summary checklist

• Bring together stakeholders to develop a common network-history retention policy
• Understand everyone’s objectives
• Check with legal advisor
• Choose what types of data will be kept for what purposes
• Set idealized retention goals for each
• Inventory current state and gaps
• Close the gaps over 24 months.

Does your organization or industry have additional or different best practices?  Share them with us at BestPractice@endace.com.

For some members of the analyst community Emulex’s announcement was unexpected, but in reality it shouldn’t have been. Today Emulex is responsible for 30% of the word’s 10G ports and has a publicly stated desire to extend their Ethernet story beyond connectivity into related markets.  Over the last few years they have acquired and partnered with a number of smaller companies that have helped them to create an expanding and differentiated connectivity and management product portfolio, and adding Endace is a  logical next steps to grow their Ethernet  engine.

Because we share market leadership in our respective areas of 10, 40 and 100G ethernet networking we see problems and indeed opportunities before many of our competitors, and we both recognize the need for end to end visibility through the data center. As organizations become more dependent on their networks Emulex’s bet – and indeed ours – is that dedicated visibility infrastructure will become a standard feature in every row of data center equipment. As we’ve successfully demonstrated over the last few years, the logic and the business case for intelligent network recording is both rational and compelling. Over time organizations are becoming ever more dependent on network connectivity for application delivery and business continuity, which is driving demand for insurance against unplanned downtime and service affecting problems.  Together we believe that  a fully searchable source of 100% accurate network history is exactly the right solution to deliver the required levels of service and meet SLAs.

Beyond our ability to grow our current business faster with more investment and an expanding global reach, what’s really exciting about the opportunity to work with Emulex is ability to reach into the server and create more visibility into the application.  For all but a few niche vendors, visibility into virtualized server architectures is tricky, but as virtualization continues to grow (Emulex expect 70% of the converged adapters that they ship next year to end up in a virtualized environment) it’s likely to emerge as a key battlegrounds as monitoring vendors seek to sell more complete ‘end-to-end’ application visibility solutions. As network guys we’re equally at home on the LAN and the WAN, but we’re all at sea in the server cluster, but with access to management data from the server end point we believe there’s a new generation of data center visibility solutions just waiting to be invented that will fundamentally change how organizations connect, monitor and manage their networks.

For sure it won’t happen overnight, but we’re excited by what it will mean to be part of the Emulex family.

Before considering any industry analyst opinion, it’s worth bearing in mind that Wall Street made its views on the acquisition very clear. It’s difficult to see a 20% drop in share price as anything other than a vote of little confidence in the deal.

In the Network World article Andre Kindness (Principal analyst at Forrester) makes some interesting comments. He says “Honestly, I think it’s brilliant. … The reality is, hardware companies have to transition into hardware/software companies, and that’s the only way they’re going to live,” he continues. ”I think the problem is that people don’t realize [monitoring's importance] … even within IT, people still don’t put monitoring at the top. The problem is that everyone follows the shiny ball, and the shiny ball right now is cloud, or software-defined networking — monitoring hasn’t been sexy,” according to Kindness.

As a company playing in the monitoring space we’re in violent agreement with Andre, at least about the need for network visibility and the market’s ability to get easily distracted by new shiny things. Where we feel the need to take odds is with his comment about the need for hardware companies to become software companies to stay relevant. To us this is anathema and is symptomatic of the broader markets lack of understanding about network monitoring.  Anyone that believes that truly actionable network visibility is a problem that can be solved in software is in our opinion fundamentally wrong.

As hardware guys to the core we’re fairly extreme in our views, but at the same time our customers speak volumes. We provide the monitoring, response and root cause foundation for some of the largest organizations in the world from government agencies to telcos, banks and manufacturing companies. What unites our customers is an almost zealotry belief that network monitoring at speeds in excess of 2Gbps is a problem that can ONLY be solved with purpose built hardware. Countless customer bake-offs have demonstrated that in real-world 10Gbps environments software based solution simply don’t cut it.

Back to Riverbed – in the past we’ve run into Opnet in customer engagements and have seen the product in action. At a dashboard level it’s undoubtedly very powerful, but just like all the other software-based vendors out there, their solution struggles to scale to meet the demands of true 10Gbps environments, despite what it might say on the tin. Our mantra is very simple – if you’re going to deploy solutions for Prevention, Detection, Response or Root Cause analysis the only thing that matters is that the analysis engine is seeing every packet. If it’s not seeing every packet, you cannot trust it.

So, Andre on this occasion we need to agree to disagree.  For monitoring companies to stay relevant in a 10Gbps world they absolutely have to BECOME hardware companies, or at least embrace the need for dedicated appliances to underpin their software, because the only way software can effectively scale to 10G is to start sampling network traffic and samples simply doesn’t provide the levels of accuracy and insight required for effective network performance management, response and root cause in critical network environments.

Some issues that are alerted by detection devices are obvious; but many are not.  So a very common approach to resolving the issues raised by detection tools is to take a guess at what the cause might be, then make a change that you hope will fix it.  If the problem goes away, you then have to assess whether it went away because of your fix or just by coincidence.  Chronic/intermittent issues are particularly difficult to deal with this way.

Endace customers, by contrast, don’t have to guess or run experiments – they can consult their network history, which includes full packet capture, metadata, indexing and search capability – and prove conclusively why an issue happened, so it can be definitively resolved.  Adding a network history layer improves time to resolution (TTR) often by 50% or better, which reduces downtime, increases SLA compliance, and reduces load on network & security operations teams.  Endace intelligent network recorders are deployed by organizations who recognize that “stuff happens” and who want to be prepared to respond when it does.

Now Extrahop, an APM company, has acknowledged this point by introducing packet capture into their product.  They quote Gartner analyst Will Cappelli in their press release saying “packet capture is a tried-and-true method of analyzing the root cause of network and application issues.”  They also say in their release that “IT teams often must wait for the problem to occur again before they can capture the packets needed to pinpoint the problem.”  Both of which are 100% dead-on right.  Best-practice shops avoid the second issue by doing continuous capture at important aggregation points, so they have data when they need it.

Where they’re mostly wrong, though, is in claiming that packet capture requires engineers to “spend hours if not days digging through gigabytes of data to find the problem.”  That’s probably true with some products out there.  But with EndaceVision, our network history navigation and search software, analysts can navigate through days worth of network history in seconds.  Which is important, because the real value of network history comes from the ability to zoom in and out, and pivot across different dimensions, such as: “what was this same user doing yesterday?” or “was anyone else speaking on this same network port around the same time?”

We’re really glad that Extrahop has jumped on the packet-capture bandwagon.  Once you discover the frustration of their poor-man’s packet capture, which shows a tiny amount of context with darkness all around, we invite you to take the MAX challenge — and see how much better your life can be with full visibility.

100 Gigabits of data (or 12.5 Gigabytes) is a lot of data, that’s for sure. But how much is it in reality? Using some very crude math, in 1 second, a fully loaded 100G link should be able to transmit any of the following:

100G is not for everyone for very obvious reasons, but big banks, research institutions, telcos, service providers and organizations serving the media and entertainment sector appear to be the early winners. The economics of 100G networking are actually very compelling when contrasted to 10 Gbps networking. Over optical fibre a 10 Gbps link consumes a whole glass strand and uses one connector;  100G delivered using SR-type 10 Gbps optics  consumes 10 whole strands bonded together but only one connector. LR/ER type 100G technology is even more efficient, consuming one strand of glass and one connector, making it ten times more efficient. So if you’re moving a LOT of data around then 100G is a perfectly rational choice.

Unfortunately, deploying 100G and living with 100G are two different things. From an IT ops perspective, 100G network segments are just like any other network segment; they need to be monitored, analyzed and recorded so that issues can be detected and investigated before end users get involved.

Unlike 10 Gbps network segments which can be matched to a 10 Gbps monitoring port on an IDS or analytics platform directly there is no such thing as a 100G monitoring system. Nada. Niet. Nuffin. The dirty truth here is that the monitoring industry has been caught napping. Today, anyone operating a 100G network is flying blind. You’d never drive your car blindfolded, so why would you run your network that way?

In fact it’s not the first time this situation has arisen; back in 2008 when 10 Gbps arrived with a vengeance most organizations only had 1 Gbps monitoring infrastructure and IT ops teams faced a similar problem. The answer back in 2008 came in the form of a layer 1 matrix switch, which Gartner tidily named Network Packet Brokers earlier this year. NPBs solved the problem almost over night by ingesting 10 Gbps of traffic and load balancing it out over multiple 1 Gbps ports which could be connected to existing 1 Gbps capable infrastructure. Over the last 5 years this market has evolved to become a $250m industry helping organizations access, filter, load balance and duplicate their 10 Gbps and 1 Gbps network traffic.

So the question is…. “why hasn’t history repeated itself at 100G”  Well, in due course it most likely will, but there’s a couple of reasonably major issues that the monitoring and NPB vendors have to step up and deal with.

Firstly, 100G is a LOT more complicated to deal with than 10 Gbps. Today’s NPB market is based on commodity merchant silicon (re-purposed from that found in standard Ethernet switches) which is perfect for the basic task of moving 10 Gbps traffic around, but isn’t going to scale to meet the demands of 100G. It’s going to take a different architectural approach potentially using a different technology to meet the 10X increase in throughput and it’s appears to be beyond the scope and capability of most, if not all of the current set of NPB vendors

Secondly, there’s a fundamental problem in the 10 Gbps tools market that’s going to bite in a 100G world. The ugly truth is that very few, if any of the monitoring and security tools on the market today that support 10 Gbps ports are actually capable of operating at 10 Gbps for any sustained period of time. 10 Gbps performance might range from 1 Gbps to 5 Gbps, but rarely gets anywhere close to 10.  At some point above 2 Gbps traditional software based packet capture technologies on which most monitoring and security tools are built start dropping packets.

The issue of packet loss is being largely ignored by most organizations and vendors today, but will become front of mind for when it comes to 100G monitoring because the types of companies that buy 100G really care about this stuff and will ask the tough questions that most people have been dancing around for the last 5 years. Maybe 100G will finally force the issue of 10 Gbps packet loss?

At Endace we regularly get involved with customers that are adamant that they know what’s running over their networks. To make the point that they’re wrong (and they usually are) we ask them to name an application that’s shouldn’t be present and we go off and look for it. More often than not we’re able to find it somewhere which we see as proof, if ever it was required, that users will always find a way to make IT work for them. For organizations that have adopted a BYOD policy the problem of application proliferation are amplified as users tend to have all sorts of odd things running on their phones, tablets and laptops. Today there are more than a 1000 different applications that all have a unique network fingerprint.

‘Application awareness’ is a buzz-phrase banded about by vendors in the firewall, network performance and network security markets with gay abandon. Application awareness infers that a vendor’s product has an understanding of what’s happening at the application layer. But not all application aware solutions are equal…. Historically, figuring out and indeed managing application layer traffic was reasonably straightforward because applications communicated on different port numbers, and by looking at the port number associated with a packet you could pretty much figure out what it was. But in today’s world, where a growing number of apps are web-based (and thus all communicate on the same port) the old methods for application detection simply don’t work. Being able to distinguish between Peoplesoft and World of War Craft at line rate 10 Gbps requires a sophisticated blend of DPI (Deep Packet Inspection) port matching, heuristics and horse power. It’s not quite an artform, but it’s pretty close.

But why care? With bandwidth to burn, surely people can do what they like, right? Wrong. In fact very wrong. Here’s a list of 5 really good reasons why you need to know, as accurately as possible, which applications are in use on your network right now.

1. Applications carry different security risk profiles which need to be understood by IT teams. Application X exposes an organizations to a higher level of exploitation risk than application Y and thus the use of application X needs to be monitored carefully to ensure that security is not compromised. Call it compliance, call it best practice, it doesn’t matter. It’s the right thing to do
2. Applications expose organizations to different levels of risk of data loss, which is subtly different to point 1. Dropbox and Skype aren’t by definition ‘insecure’ but they do make it very easy for end users to move sensitive data out of the business almost invisibly, bypassing traditional DLP tools
3. Some applications are patently illegal. End users downloading pirated content from Pirate Bay on a corporate wireless LAN is a recipe for trouble. And ignorance is no defense, so beware!
4. Different applications consume bandwidth in different ways and in different proportions. Application performance issues are often related to bandwidth congestion, so knowing which apps are consuming the available resources at any given point in time is a vital input into the troubleshooting process and can dramatically shorten time to resolution
5. Some applications are by definition more ‘productive’ than others. A corporate network full of YouTube and World of War Craft can be a sign that employees aren’t perhaps using their time as productively as they might be and that some kind of intervention might be required.

So, there’s a number of good reasons why you need to know what’s happening on your network.  But of course, knowing is one thing and enforcing is something quite different. The reality is that for most organizations there’s two pieces to the application awareness puzzle: An application-aware network monitoring / recording tool such as an EndaceProbe AND an application aware firewall. The combination of the two technologies enables organizations to not only build an understanding of network usage usage, but also deliver the necessary enforcement / prevention capability.

So how much do you really know about your application mix? If you’re interested in what else other organizations do and don’t know about their networks you can download our 2012 network visibility survey here