There’s no doubt that the Layer 1 matrix market is running hot  right now, and there’s good reason -Layer 1 switches help to solve a real problem that just about every large enterprise in the world has. Today, organizations need more network visibility than they’ve got access points, and the layer 1 guys solve this problem by  duplicating traffic captured from taps or spans and sending it off to a growing number of downstream analytical and security tools for crunching. There’s of course  a bit more to it than that, as the ability to aggregate and dis-aggregate traffic is a useful layer 1 trick that enables organizations to extend the life of Gigabit tools and maximize the return on 10Gigabit tool investments by ensuring that every port is fully loaded.

So on the surface at least the market for Layer 1 looks good and the spreadsheets for projected growth have all the lines pointing in the same direction, which are the right conditions for M&A. But there is a problem with this market and it’s related to the downstream tools that these Layer 1 switches are feeding. Like it or not, most of these systems are unable to handle the streams of traffic being thrown out them today (which is even before they’ve been loaded up with aggregation), which means that they are, for all intents and purposes, broken.

Functionally, Layer 1 switches are responsible for suppling a stream of packets monitored from the network to a downstream tool, such  as an analytics package for example. The analytics package then ingests the packets using it’s own I/O, analyzes them and potentially stores some or all of them to disk for later retrieval before populating a dashboard with some kind of intelligence about the performance and /or behavior of the network. The problem is that most of these downstream tools are in fact unable to consume all of the packets that they are being presented with without missing a few here and a few there… If you’re serious about network visibility then a ‘few dropped packets here and a few dropped packets there is a big deal – particularly when ‘a few’ could be as many as 50%

Publicly available research suggests that there are in fact very few tools that actually perform as advertised – particularly in the 10Gbps space.  If a car has an advertised top speed of 140MPH then you expect it to be able to drive it at 140MPH, not 100…Likewise, if a system has a 10Gbps port on it, you expect it to be able to handle ful 10Gbps of traffic (multiplied by however many ports there are) without missing anything at all.

So, although the layer 1 guys are meeting a short term need for access to the network, there’s a strong argument that they are actually persisting and exacerbating a more systemic problem which is only getting worse as network get more and more loaded. Until organizations accept the fact that the tools that they’ve got in the ground are broken and start to look at strategic alternatives that really do perform as advertised they will continue to suffer the effects of network blindness – which is exactly what they bought tools in the first place to avoid.

Below you’ll find a series of links to some of the most poplar use case videos along with a brief description of what each one does. If you have any use cases that you want to see us have a go at then please email enquiries@endace.com and we’ll see what we can do to oblige. If you come up with a good use case we’ll send you some Endace chotsky.

Using EndaceVision to investigate anomalous network behavior

http://www.youtube.com/watch?v=kKS3CqlRaW4

In this user submitted use case we show EndaceVision can be used to solve a mysterious network traffic spike that has been occurring on the network and then configure alarms to alert next time the issue occurs.

Using EndaceVision to investigate a poor video quality call

http://www.youtube.com/watch?v=F3K4gJnkkYQ

In this video we show how EndaceVision can be used to rapidly analyze an under-performing video conference call by selecting into the time range the event occurred, visualizing the event and then investigating the potential causes for its low quality.

Using EndaceVision to see and investigate web apps on a network

http://www.youtube.com/watch?v=lWtqHilArMY

In this video we use EndaceVision to investigate what web applications are running on the network by visualizing traffic broken down by application. We then apply filters for port 80 and 8080, allowing us to view all of the web applications. From here we filter on Mafia Wars and then switch view to source IP to see who was using accessing that particular application at that time.

Using EndaceVision to rapidly isolate packets of interest

http://www.youtube.com/watch?v=gbf1dJ9iiWw

In this video we use EndaceVision can be used to reduce a massive 10TB network traffic file to a 135MB selection containing only the packets associated with an RDP connection we were looking to investigate. From here we download the packets directly into Wireshark for a full decode which gives us the full story.

Using EndaceVision to apply alarm overlays

http://www.youtube.com/watch?v=IpqcivR10ZE

I In this video we show how preconfigured alarms can be used as an overlay and be applied to a bandwidth over time graph to spot anomalies within a network, and visualize how they correlate with other network events.

Using EndaceVision to Share and Collaborate on Network Events

http://www.youtube.com/watch?v=Ej5SyNrwuJ0

This video demonstrates the various ways to share and comment on a visualization of a network event between different members of a network operations team. EndaceVision’s flexible framework helps to streamline the workflow and get to the right information to the right person in real time.

Answer: The network guy, of course.

Sound familiar? If you work for a Managed Services Provider then there’s a better than average chance that you’ll have experienced this many times. In fact, anyone responsible for running any network will probably recognize the scenario. In a complex multi-vendor environment figuring out what’s wrong and who’s responsible is really really hard.

It’s kind of convenient to blame ‘the network’. Network problems are notoriously difficult to diagnose, they are frequently transient and can often be subjective. One person’s slow can be another person’s fast and so on. Add into this mix the fact that routers and switches are now so complex and support so many features that the chances of any one single person knowing how it’s supposed to function are slim to say the least. In networking there’s always an element of grey as to whether something is a ‘feature’ or a ‘fault’ and that’s easy for other people to play on.

As we enter an age where the network and IT are converging into one amorphous mass the Ops Teams responsible for managing service delivery are inevitably going to become ‘all powerful’. While IT remains mission critical but massively complicated we will see a new  ’Age of Operations’ emerge and, in an Age of Operations, the tools that help diagnose problems (and prove innocence along the way) are going to be invaluable.

The need for rapid, effective and highly accurate visibility into both the network and the IT infrastructure it supports was crystalized for us in a recent customer engagement:

An MSP was responsible for providing the network to support a medium sized (~10,000 seat) VDI implementation. They were a core part of a multi-vendor team all responsible for delivering their respective parts of the project. Everything had come together well and the implementation was launched on time, but within days users were reporting unacceptable response times from the network. The team came together to trouble shoot the problem and laid the blame squarely at the feet of the MSP. In the absence of any network visibility the MSP dutifully responded by doubling the bandwidth and then quadrupling the bandwidth to try and fix the problem, but it simply didn’t make any difference. So eventually they called us.

By dropping an EndaceProbe (running EndaceVision) onto the network they were able to almost instantaneously see that the network was behaving exactly as it was supposed to behave and that the problem was related to an unexpected user-behaviour. For whatever reason end user community had developed a culture of plugging their own USB keys into their PC’s to view and share photos and music etc. Such a behavior is fine in a fixed host environment but a disaster in a VDI environment, as the virtual desktop footprint (which is downloaded and uploaded every time a user logs in) balloons and absorbs all the available network resources.

If you are that network guy and you’re forever carrying the can, maybe it’s time you looked at improving your Mean Time To Innocence, as life’s hard enough fixing the problems that really are your own without trying to fix problems that belong to someone else.

The first step towards developing an informed policy decision around BYOD is to really understand the issue.

Organizations perceive a risk from an open BYOD policy because:

• They fear that employees’ personal devices will ‘infect’ the network and expose the organization to the threat of cyber attack from the inside out
• They fear  employees will download sensitive data to their machines and slip out the back door
• They fear that employees will spend all day watching NetFlix, become inefficient AND clog up the network with streaming video traffic along the way

These are real risks for sure and organizations are investing in all manner of hugely complex NAC and DLP platforms to help them manage the risk, but are they missing the point? What’s the upside of playing a more open game and is it worth the risk?

Let’s take a step back for a moment and look at the bigger picture. Contrast an employee bringing their iPad to work and connecting to the Internet to check his Gmail account or check Facebook against the every day occurrence of an employee taking his corporate issue laptops home on a Friday night, spending the weekend surfing dubious websites on his home wifi network and then bringing his laptop back to work on Monday morning.

Given that corporate web policies cannot be easily enforced when a machine is disconnected from the corporate network there’s arguably a greater risk from ‘Bring Your Corporate Laptop Back To Work On Monday’ (BYCLBTWOM?) than there is from an iPad hitting the network during a lunch break. In the same vein, anyone using a USB key to transfer files from a corporate machine to a home machine (for weekend homework) could equally be accused of BYOD, as anything dangerous on their home computer could easily end up inside the LAN the second they plug the USB in.  If organizations are worried about getting attacked from the inside there’s arguably far more open doors that should be closed before they start worrying about their BYOD policy.

It’s equally important to understand the potentially positive impact on the organization of embracing an open BYOD policy. By allowing employees to use their own devices in the office they can often be more efficient, they can derive greater pleasure from their roles, work harder and become better, more satisfied employees – which is exactly what most organizations strive for every day. HR teams spend millions of dollars every year dreaming up creative ways to develop staff morale, but may well be missing some really obvious low hanging fruit.

Some organizations fear that by condoning BYOD that their IT help desks will be flooded with random support calls about Windows XP configuration issues, but the reality is that the people that want to bring their own devices to work are typically the most tech savvy employees and the most self sufficient. For the most part, most people are perfectly happy with their corporate issue IT and would just like to connect their iPhone to the wireless LAN to reduce their data overage.

Organizations around the world have adopted a range of different positions on BYOD from total bans to more liberal, laid-back approaches. The challenge with a hard-line policy is actually enforcing it, as it’s technically really hard to stop people connecting their personal devices to the network. Sternly worded emails from the IT and HR departments about the ‘harsh consequences of being caught with non-corporate issue IT’ only go so far… In a recent customer engagement with a large organization in Europe who had a hard-line policy on BYOD we were able to show them ‘substantial’ download traffic on their network originating from Apple’s iOS upgrade portal. In a 100% Microsoft environment that was definitely unexpected!

As with all things, it is possible to find a middle ground that gives employees access to high speed internet (which is by and large all they’re after) in a highly controlled manner, without giving them access to all of the network resources. For most organizations this involves setting up wireless access via a DMZ. Sure, it won’t satisfy everyone and there will still be abuse, but by meeting employees half way there’s half a chance that the risks can be managed down to acceptable levels.

If you’ve taken the hard line approach, or you just want to check that employees aren’t abusing your good-will gesture, then you’re going to need some tools to help you see what’s actually going on inside your network. EndaceVision has some useful features that can be applied to trigger alarms when traffic that pertains to ‘unauthorized devices’ appears anywhere on the network. It can also be used to monitor the quantity of NetFlix and YouTube on the Network which is often a good indication of BYOD abuse. In our experience most organizations have very little idea what’s really going on inside their networks and are genuinely shocked when they get the real picture.

There’s a strong argument that BYOD is a battle that organizations will ultimately lose and should, with certain conditions, be happy to concede. The line between home and work is getting more and more blurred and hard line policies on BYOD may be counter-productive. In the future organizations that elect to run with hard line policies may find themselves struggling to attract talent. It’s clear that Gen X and Gen Y employees won’t tolerate rubbish IT and they will rule the world (eventually) so the faster you get to a place that satisfies the few that want to connect their personal iPad to the Internet during office hours, the better for everyone concerned. Just make sure you’ve got the right visibility tools in place first to help you manage the risks.

Networks are getting faster, IT is migrating to the cloud, applications are sharing the web and people are bringing their own devices to work. These factors, coupled with the fact that the bad guys are playing a smarter game than they’ve ever played before, combine together to have a profound impact on the way that organizations must start behaving. If you are responsible for protecting a mission-critical network, here are eight things that you need to think about:

1. Record network traffic for the purposes of forensics
You will never make sense of a security breach without a complete record of every last packet after the fact. Event and log management can indicate some type of breach occurred, but without having all the data to reconstruct the precise activity of the session, your company will not be able to determine if the attackers merely got onto a system, versus having gotten away with sensitive data.

2. Use recorded traffic for retrospective threat detection
The fact that your IDS or IPS didn’t alert you on an attack at the first pass doesn’t mean that there wasn’t one there. It simply means that your rules engine didn’t know about it. If you record traffic, you can re-run it through your network security systems the next day or the day after with updated rule sets. With network recording you reduce the risk of being caught out by a zero-day attack.

3. Get visibility into the application layer
You can’t truly know where you are vulnerable until you have complete visibility into exactly what’s traversing the network in real time. Different applications have different risk profiles, and you need to know which applications are present on your network and who is using what. With more and more applications sharing a common port, the ability to distinguish between applications at layer 7 is critical.

4. Don’t oversubscribe your systems
When resources are constrained and space is limited, there’s a natural tendency to push more traffic through systems than they can actually handle. What a security system—such as an IDS or IPS—says it can handle and what it can actually handle are frequently two different things. It’s essential that you understand what throughput your systems can handle before they start missing important events and exposing you to unnecessary levels of risk.

5. Take into account the demands of tomorrow, today
As you make strategic decisions about which systems that you’re going to use to protect your organization, make sure you think about the way your network is changing. For many large organizations, 40Gbps networking will become a reality inside the next system refresh cycle (3-4 years). When your core infrastructure upgrades, will your tools be able to keep up? To avoid the need to retrain your teams and switch hardware vendors, work with vendors that can show you a 40 and 100Gbps roadmap today.

6. Use a common data source
Network security—and monitoring tools for that matter—all rely on captured packets to generate intelligence. One way to improve your security posture is to ensure that all your tools are sharing the same source of 100 percent accurate traffic, either by putting them all behind a single accurate source of packet capture OR by co-locating them on a common platform.

7. Think about your rules
Your network and your traffic are uniquely yours, and for that reason it’s critical that the rules you choose to run on your network security systems are relevant to you. By understanding your traffic profile and insisting on a network security platform that enables you to choose your rule supplier and write your own rules where necessary, you can dramatically improve your posture.

8. Take an Inventory of your business and security requirements
Compare your business and security requirements to what’s actually happening. For most companies there’s a significant delta between what the business side and security side want to capture and analyze on the network, versus what is actually being captured and analyzed. Ask yourself the following questions:

• Is the business getting information it needs to be secure?
• Which security solutions are leaking data or not getting the whole picture?
• What hardware and software are due for replacement?
• Are there consolidation opportunities?

By dedicating more attention to your network visibility efforts—combined with security practices—your organization will be in a better position to not only protect, but correct network anomalies.

As any engineer will attest, the most difficult problems to trouble shoot – the ones that absorb 90% of the available resources – are the intermittent ones. The problems that are there, aren’t there, are sometimes there are the ones that are just plain irritating. All over the world days, if not weeks, of  man effort are wasted chasing intermittent faults. The traditional strategy employed by most organizations is to wait for a user to report a fault once (or maybe twice…) then ship in a probe of some description to capture a trace which can then be analyzed to help diagnose the problem. And ten years ago, when networks ran sub-gigabit and the network was important, but not mission critical, this was probably OK. But ten years have passed and things have changed – a lot.

On mission critical networks, where the network IS the business, or the organization is critically dependent on the network for operational continuity, the concept of pervasive capture is catching fire. Why? Because pervasive capture, accompanied by the right visualization tools, is the key to reducing time to visibility. Here’s a recent example from the team that really helps to bring the concept of Time to Visibility to life.

A service provide (who shall remain nameless) was having a nightmare. At the same time every day something happened that caused their billing system to fall over. Now if you’ve ever worked for a carrier, the ONLY thing that matters is that you can take payment. If your billing system is down it means that your store is closed and in carrier land, if your store is closed people just go to a store that is open because everyone’s selling the same thing. At the same time every day it all went dark for almost exactly an hour and the ops team had no idea why. Gradually, over a period of two weeks, through the most painful process of elimination imaginable, they worked out that the billing outage was caused by a DDOS attack from an overseas source using an IP connection request flood (which was generating CDRs) and were able to block the attacker’s IP and resume normal service.

To fix the problem It took a team of four highly paid engineers 2 weeks working pretty much full to identify the source of the problem and eliminate it. In the mean time, the business not only lost significant amounts of revenue but more importantly, it took a battering in the media where the real damage (if you’re a telco) gets done. Not ideal

So here’s the same scenario replayed through a different lens, where the organization had deployed a pervasive packet capture fabric.

A wireless carrier who shall remain nameless was having a nightmare. At the same time every day  something happened that caused their billing system to fall over. Through a network of interconnected Endace network monitoring and recording systems the carrier was able to visibly see (using EndaceVision) that there was a significant bandwidth spike on the connection between the carrier’s wireless network and the internet. The ops guy in charge at the time was instantly able to isolate a sample of traffic from the traffic spike and, through a simple drill-down, establish that it was a single IP address causing the problem. Decoding the packets in Wireshark showed that the foreign IP was establishing connections to the network that created CDRs that flooded the billing system and caused it to fall over.  Within 10 minutes the IP address was blocked at the firewall and normal service was resumed. Total outage time? Less than 10 minutes.

The example illustrates that there’s actually two metrics at play here – ‘time to visibility’ and ‘cost of blindness’.  Take a moment to think about your time to visibility and your cost of blindness…and wonder whether it’s time to take another look at pervasive packet capture…

To really understand the case for network recording we need to take a brief trip down memory lane and look at how organizations unlucky enough to have had their own ‘CNN moments’ during 2011 fared during their time under the media spotlight – think Sony, Zappos, Epsilon and, of course, RSA. One thing that becomes very clear very quickly when you look at each of the cases is that they were all, without exception, PR disasters. Millions of dollars got wiped off market caps and consumer confidence in organizations got destroyed.

For a large listed corporation a security breach is arguably one of your PR department’s worst nightmares (after major oil leaks if you happen to operate in the oil business perhaps). PR 101 says that in the event of a disaster (and let’s be clear, which ever way you look at a data breach it is a disaster) ‘fess up’ as fast as possible, beg forgiveness, hope that it’s not a quiet news day and ‘front up’ with a plan to make it right. As a text book strategy that’s all well and good, but it does pre-suppose that organizations actually know what happened. Which more often than not, they don’t.

In the event of a breach, if you’re lucky, you’ll find the (metaphorical) broken window pane and the brick on the floor, but the chances are you’ll have little or no idea what stock got stolen because your shelves are virtual. Sure, your log files are useful to a point, but will never give you the whole story. To manage the damage you have got to be very certain on exactly what got stolen and you’ve got to be fast and certain. The faster you come clean and the more certain you are the smaller the damage footprint is. The media will claim that the public has a right to know how many of your records (because they are really their records) were stolen, how sensitive were they and how long ago it happened. The longer it takes for you to come clean, the worse it gets. The ugly truth here is that from a PR perspective at least, the only answer to the question ‘what did you lose?’ worse than ‘we lost everything’ is ‘we really have no idea what we lost’.

So, if you subscribe to the view that you’re already breached (and the smart money says that you should) what’s your answer going to be when Wolf Blitzer asks you the question “what did you lose?’ If you haven’t got a good answer then maybe it’s time that you took another look at monitoring and recording technologies?

As followers of our story will know, for the last ten years we’ve been the platform-of-choice for organizations seeking to protect, monitor and measure their networks, but to date we have been reliant on software applications from third parties to deliver the application layer. For the first time EndaceVision breaks that third-party dependency and provides proven 100% accurate Endace network visibility all the way from the wire to the screen and back again, so it goes without saying that this is a big step for us. As hardware guys, the beauty of being able to write your own application is that you get to truly optimize every aspect of the application. EndaceVision leverages a unique blend of hardware and software processing to create the fastest, most accurate and most elegant architecture of any visibility solution today. Having the right solution architecture has always been important to us because EndaceVision, and the underlying visibility infrastructure that powers it, is designed specifically to meet the demands of 100Gbps networks, which is quite literally a whole different ball game.

EndaceVision is the product of many customer interviews, market research and competitor analysis – as well as a healthy dose of gut instinct for what the market needs. As platform guys our mission historically was always about capturing, recording and presenting packets to third party applications with our world-famous accuracy and performance. As we transition into a fully fledged solution vendor we’ve become much more sensitive to our customer’s solution needs, what their pain points are and what problems they are trying to solve. To be honest, it’s been hugely enlightening.

It became clear very early on that creating a new solution and, in the process a new technology category, was going to require us to think about more than just technology. Our research quickly highlighted that the tools our customers are using today to manage and protect their networks are flawed on multiple levels. The tools they’ve got don’t give them them information resolution they need, are extremely hard to work with  and they don’t scale well. We’ve talking about the case for scale at length in other blogs, but the issues of information resolution and practicality are worth exploring in a little more detail.

Information resolution is about what you can see. Traditional network monitoring tools provide visibility from layer 1 to layer 4 of the OSI stack. In yesterday’s world, layer 4 visibility was enough; you could pretty much figure out what was going on and what was going wrong without seeing any further up the OSI stack than the transport layer. But in the last 5 years IT has gone through a radical transition. Application delivery has gone virtual and business critical apps have migrated to the web, so visibility to layer 4 simply isn’t enough anymore. Organizations must now be able to see all the way to the application layer (layer 7) and be able to distinguish between Facebook traffic and Salesforce traffic in order to effectively manage, optimize and troubleshoot the network. EndaceVision offers visibility all the way to Layer 7 and is able to recognize up to 600 different applications in real time and comes with a cast-iron commitment from Endace to continually update application signatures.

The practicality (or usability) of a network visibility solution encompasses lots of different design elements. For example, in a world of virtual desktops deploying thick clients for local host processing is really hard for organizations as it requires work-arounds to be put in place and ‘edge-cases’ to be created. To get round this problem we’ve designed EndaceVision to run in any standard web browser. Being browser based brings with it a range of other advantages as it means that experts can be brought in to an investigation regardless of where they are in the world without them needing to come to site. The shift to the browser is already showing itself to have significant business benefits in both performance and deployability.

The other side of practicality is usability. What we discovered when we sat with experienced users is that they already have well established processes in place, and what they don’t want is new tools that change their processes. What they want are tools that work within the boundaries of their existing processes, making them faster, more efficient. The good news is that most of the processes in use across the customers we looked at were very similar and we were able to develop a unique and elegant workflow within EndaceVision that helps them identify and investigate problems and get to root cause in the shortest possible time. The workflow is flexible enough to adapt to the different circumstances we see but does not over-engineer the problem. There’s no doubt that simplicity is the key to effective network visibility.

We’re really proud of what we’ve achieved with EndaceVision and we’d like to publicly express our thanks to the customers (who at their own bequest have asked to remain nameless) who let us into their private world’s and gave us the benefit of their experiences. Without them, EndaceVision would not be what it is today.

Every expert worth their stripes knows that at a given network speed a NIC-based monitoring system will start to drop packets and as a result, the applications that they feed will start to go blind. But the reality is that software applications also have their own failure point beyond which they can’t process the packet flow – regardless of how accurate or complete the stream may or may not be. It turns out how your chosen software application is written has a profound impact on it’s performance at higher network speeds.

At Endace we spend a lot of time helping customers to see into their networks for a wide variety of different reasons and in the course of our work have tested a lot of different software applications for their respective break points.

You see, Endace Systems are able to provide a 100% accurate feed of traffic to an application hosted locally on them (at speeds well in excess of 10Gbps) and thus we’re able to very accurately determine at what point the applications starts to miss important events in the traffic flow.

In our experience testing a broad range of ISV software apps (from VoIP Quality of Experience to DLP), there’s as much chance of the software application failing at 3Gbps as there is the NIC based hardware. So, with this information how do you scale a monitoring application scale to meet the demands of even moderately loaded 10Gbps segments without them falling over?

Option 1 is to throw hardware and processing cores at the packet capture problem and cross your fingers that the app can keep up. Sound familiar?

Option 2 is to use an EndaceProbe. Purpose built packet capture solves the problem of 100% accurate capture (at speeds up to 100Gbps!). The problem of application scaling is solved using a combination of hardware and software based load balancing within the EndaceProbe. Multiple instances of the relevant application are loaded into the Endace Application Dock and fed a stream of packets that never exceeds 80% of the apps capacity to consume them.

As the resource of the Endace System are saturated (which depends on the compute demands of the application) then EndaceProbes are daisy-chained to provide the additional power required to process the packets. Some smart hardware filtering which reduces packets down to just the packets that matter means that hardware footprint is reduced to the absolute minimum.

So there’s some choices – either limit your choice of applications to those that you know scale to 10Gbps (and there are no where near as many of them as you think) OR continue to use the applications that you trust, but use some clever hardware tools to make sure that they can keep up with your network demands.

A new class of packet based network monitoring and recording solutions are emerging that enable companies running high-speed and ultra-high-speed networks to address the issue of network blindness, a condition that exposes organizations to a raft of operational, legal, compliance and reputational risks. With the cost of network downtime measured in millions of dollars per hour, knowing what’s going on inside the network isn’t just a nice to have, it’s critical.

Today’s 10 Gigabit networks are so complex that there’s invariably duplicate traffic from badly configured switches and routers consuming bandwidth without being noticed, resulting in everything from videoconferencing falling over to failure of critical business applications. Installing a packet-based monitoring and recording fabric enables organizations to alleviate network blindness and gain visibility into network congestion issues.

It’s clear that ultra-high-speed networking is on the horizon in many industries. In a recent survey of 100 organizations in North America, 71% said they have made the transition to 10Gbps networking. The companies that participated included tier-two telcos, online service providers, retailers, manufacturing companies, health service providers and gaming companies, all with annual revenue of at least $10 billion. In addition, 43% of the organizations surveyed said they have plans to adopt 40Gbps or 100Gbps networking.

According to the senior networking, operations and security professionals surveyed, many of their incumbent network monitoring and security vendors are unable to reliably manage higher network speeds. In fact, 47% of the respondents believe they are missing potentially significant network events due to failing or under-performing systems. Another 65% of the organizations do not record network traffic for forensic analysis of network events, and 43% percent reported experiencing “significant difficulties” investigating and remediating network events.

Other findings of note:

- 33% of organizations reported experiencing some kind of data loss in the previous 12 months.

- 39% were unable to accurately identify what was lost.

- 42% admitted to having been the victim of a cyberattack in the past 12 months.

- 67% of those victimized by an attack admitted to having serious problems investigating the attack.

There are a plethora of 10 Gbps-capable monitoring tools available, but most of them start to get a nasty case of network myopia as network speeds hit 3Gbps. What they claim to be able to do, and what they actually do, are turning out to be quite different things. The challenge they have is that they are unable to get packets off the wire fast enough to figure out what’s really going on. The interrupt rates of standard NICs overwhelm CPUs, causing packets to be dropped. Therefore, there is a need for dedicated and purpose-built packet capture hardware.

In the past, simple visibility to Layer 4 was enough, but that’s no longer the case as so many applications are now Web-based. Without visibility into the application layer, organizations can’t distinguish between Skype and SAP or Dropbox and FarmVille. With visibility into the application layer organizations can really able to start to see what’s happening and who’s responsible.

Packet-based monitoring is the only way to get really high levels of granular visibility into the network. When compared to sampled NetFlow-based tools or traditional SNMP polling, the difference in information resolution is an order of magnitude greater. It’s only by recording packet level information that organizations can go back in time to perform forensic investigations into events. But remember, the output of a packet-based tool is only ever going to be as good as the quality of the input, and without every packet, almost any analysis that you do is pretty much pointless.

Tools that accurately record network traffic (as part of an integrated security solution) have proven themselves to be an essential way of providing post-security attack forensics, including the ability to understand what data may have been lost and how. Security teams and network operation teams need to isolate packets for forensic investigation and deal with rising expectations for network uptime and performance.

In order to select technologies that will scale to meet their needs as they move to 10Gbps networks and beyond, organizations need to start asking the network monitoring and network security solution vendors a different set of questions:

• How far (up the OSI) stack can you see?

• How fast can you capture and record packets before you start losing them?

• How can I access the raw packets that have been captured?

• Do the packets leave the data center when I access them (big compliance risk)?

To be acceptable, vendors need to be able to prove their claims around network performance (the standard metric is dropped packet counts at different network speeds).

Today, a new breed of distributed packet-based network monitoring and recording fabrics are emerging to help organizations solve the problem of network optimization and real-time anomaly detection and forensic examinations. They are reducing both the amount of time it takes to investigate any given network issue (meantime to resolution) and reducing the average skill set required to do so.

One-hundred percent packet capture-based network monitoring and recording platforms provide a common network management infrastructure that allows users to chop and change between tool sets quickly and easily, depending on the issues they need to address. Packet-level recording is only going to become a bigger deal as the true impact of recent security breaches is felt and SEC legislation is imposed to force organizations to come clean. What’s your plan?