To make any sense of this question it’s important to understand what causes a microburst and what impact they can have on your business systems.

Microbursts are a problem for organizations when the spike in network traffic overwhelms the capacity of systems in the data path to handle the load. When systems (like trading systems for example) are overwhelmed they typically drop or delay the flow of packets pointed at them. When trades are measured in microseconds, dropped or delayed packets have the potential to cost organizations serious momey. For firms engaged in high frequency trading knowing the true capacity of systems in the data path and being able to spot bursts that potentially compromise those systems is a big deal and worthy of serious investment.

Microbursts are not just isolated events on a particular link. They are caused by congestion on multiple downstream devices that feed a single upstream device. Essentially a ‘glut’ of packets all arrive at the same point in the network at exactly the same time. Put another way, think of a 3 lane highway merging into a single lane at rush hour – the burst cannot exceed the total available bandwidth, but it can saturate it for a period of time.

With that clear, how big does a spike have to be and how long does it have to last before it should be called a microburst?  The answer to that question now depends on the capacity of the worst system in the data path. A microburst should be defined as any traffic spike that causes that system to fail. So in reality everyone’s network should have its own unique microburst definition.

If you’re looking for a guide, based on our industry experience a sustained burst of traffic that saturates 75% of line capacity over a period of 100 microseconds is enough to cause problems for some under-powered systems. Assuming that you can work out what your unique definition of a microburst is (and that’s normally through a process of trial and error comparing trace files at different line rates) how do you go about alarming on threshold breaches that could potentially cause packet loss? The answer is of course that you buy some kind of microburst detection solution.

Vendor solutions in this market can generally be put into one of three groups

• Group one are vendors that use sampled network statistics (NetFlow) collected from routers and switches in the data path to look for bandwidth over time spikes. Being based on sampled data these systems offer relatively low levels of visibility and typically struggle to provide visibility into bandwidth utilization below 1 second increments.

• Group two are the vendors that manufacture the routers and switches that sit in line. These solutions see network traffic at a bit level (rather than at a packet level) and calculate bandwidth utilization from true data throughput speeds. The problem that these systems have is that they don’t have the necessary processing power to generate a truly accurate calculation of bandwidth over time.  In many instances they simply generate NetFlow statistics which they pass off to other systems which really put them in Group 1.

• Group three are vendors, like Endace, that use passive packet capture solution to calculate the volume of traffic based on the packet time stamps and packet sizes. This approach provides far and away the best mathmatical framework for detecting microbursts if (and it’s a big IF) the packet time stamps are of a sufficiently high resolution and they really have captured every packet. In our experience you should take neither of these things as read.

Once you’ve got all the timestamped packets the process of plotting bandwidth utilization is fairly straightforward… Every 100 microseconds sum all of the bits that have transited the wire and plot them on a line graph. It’s essential to make this an exercise in addition rather than multiplication, as taking a short traffic sample and multiplying the data volume out to provide a view over a longer period adds unacceptable inaccuracies into the equation – particularly as network speeds head towards 10Gbps.

If you’re worried about the impact that microbursts might be having on your critical systems here are 4 questions to ask any vendors that may be pitching you their wares.

1. Over what time period do you calculate a microburst? (anything less that 100 microseconds is unlikely to give you the answers that you’re looking for)
2. How do you guarantee the accuracy of time stamps and to what resolution do you offer time stamping? (look for an answer in the region of +/- 8 nanoseconds)
3. What sort of alarms and threshold monitoring do you provide and at what level of granularity?
4. What tools do you provide to allow me to extract the packets of interest so I can compare them with the output of my business systems?

If you’re interested in seeing how we monitor for microburst feel free to get in touch and we’ll gladly show you.

The basic premise of the PC World argument is pretty straightforward – cloud computing has the potential to deliver huge operational cost savings to organizations IF (and this is the big IF) they can get a handle on the bandwidth demands. Their argument is that cloud implementations are failing because organizations aren’t listening to the needs of the network and, as a result, are finding that their apps are failing to deliver acceptable levels of user experience, their back ups are timing out and their databases are getting out of synch.

Inside the cloud bandwidth requirements relate to connectivity within the virtualized environment and end-user connectivity to the cloud through the access network. For a cloud implementation to be successful both must be insynch, which requires very high levels of network visibility.

The article looks closely at Intercontinental Hotels’ cloud experience which involved a fundamental re-architect of their network so data could be quickly reachable and that their global data centers could stay insync. For them bandwidth was, and remains, a critical success factor.

Interestingly very few enterprise applications are designed to operate in a cloud environment and are found to require significant amounts of bandwidth to accommodate the amount of ‘chat’ that goes on (locally and remotely) between the various database, storage and application servers. As part of the article, Theresa Lanowitz (analyst at Voke) asks why more organizations don’t include bandwidth considerations in their cloud strategies.

So the question is, why aren’t organizations doing more to manage bandwidth and how can they ensure that they don’t get application melt down when the cloud comes on line and critical business apps start contending with Monday morning Facebook video requests?

Here are 3 insights aggregated from our recent experiences in the cloud:

1)     Very few organizations know what’s really on their networks to start with and, as a result, are stumbling around in the dark when it comes to planning and forecasting bandwidth requirements. Understanding what users are really doing, how bandwidth is really being consumed and how application use really changes by day and hour is key.

Now there are of course tools out there that provide network visibility, but it’s typically at the wrong level of the stack (layer 3 or 4) and if your network is carrying traffic at speeds in excess of 2Gbps there’s a good chance it’s only giving you half the picture.

If you’re planning a cloud move and want to know what you’re dealing with before you start then you need trusted to know what’s happening at layer 7 (regardless of line rate) which is exactly where we create value through visibility.

2)     Figuring out what’s going to happen once the cloud goes live is something of an art. It’s also absolutely critical.  Get it wrong and it all grinds to a halt, get it right and no body notices you’ve just cut the IT budget in half.

Sure, there are a number of specialist test and measurement tools on the market that can help to artificially create peak traffic loads (the article notes vendors Ixia and Spirent in this regard) but in our experience the only way to figure what’s really going to happen is take a recorded copy of your own network traffic and use that (replayed at different rates) to find out how things are going to interact. Your network is as unique as your fingerprint and there’s no substitute for testing with real network traffic.

3)     Once your cloud is up and running its essential to have highly accurate visibility tools in place at strategic points across the infrastructure and access network to monitorbandwidth and application usage to ensure that bandwidth usage doesn’t exceed known performance parameters. With the right tools the time taken to troubleshoot problems can be slashed from hours or days to minutes. From our experience it only takes a relatively subtle shift in the network application usage profile to have a substantial impact on bandwidth requirements.

On the whole, we’re in violent agreement however, what PC world has failed to acknowledge is just how hard it is for organizations to get the necessary levels of visibility at speeds exceeding 2Gbps – which is exactly where most large organizations are today.

While 10 gig has taken off, the macro-economic climate has changed dramatically, and as IT budgets have tightened organisations have quite legitimately started to look for ways to save money and delay spending on what they perceive to be ‘non-essential’ items. One such ‘pop strategy’ is to attempt to extend the life of existing gigabit network monitoring tools. To help facilitate this need a market for intelligent layer 1 matrix switches has sprung up out of seemingly nowhere.  Amongst a wide range of claims, the switches that this community is marketing promise to ‘extend the life of  1Gbps tools by allowing them to be leveraged in 10 gig environments through ‘sophisticated filtering, disaggregation and load-balancing’. Prima facie the logic is sound – spend $50K now and delay an investment of $250k for a year or so, then redeploy the $50K hardware elsewhere and you’re laughing. But is it really that straightforward?

The answer is of course ‘no’ – but to understand why, you need to first accept three incontestable facts about 10 gigabit networking.

• A 10Gbps network WILL burst up to 10Gbps some of the time, regardless of it’s ambient traffic level
• Network monitoring tools typically deliver the vast majority of their value in times of high network stress
• The output of any network monitoring tool is only as good as the quality of it’s input. If it’s only seeing half the input….

If you accept those things then for every 10 Gigbabit link that you want to monitor you’re going to need to have 10 uncontested Gigabit monitoring ports available all of the time to cope with the bursts. And of course that’s where the problems start because no one does, because the economics simply don’t work. Organisations provide maybe 5 gigbabit ports for every 10 gigabit link,  but that completely misses the point.

So what happens in practice? Well – the infrastructure team become instant heroes because they’ve successfully delayed a significant capital investment by applying a sticking plaster to an ugly problem. And then there’s a serious network performance issue in the core that netops need to investigate. So they log into the network monitoring tool and begin the process of trouble shooting only to find, surprise surprise, that half they’ve only got half the story. And reality bites.

The truth is that trying to sweat gigabit tools in a 10 gig environment – particularly critical environments where down time and outages cost real money – is a false economy.  It might look good on the balance sheet, but in practice it will cost far more than it saves as operational management costs escalate and network performance becomes less and less predictable.

Where the network is critical there is no substitute for true 10 gigabit infrastructure. And when we say true 10 gigabit infrastructure we mean infrastructure that is truly capable of handling line rate 10 Gigabit traffic without dropping a single packet along the way. In exactly the same way that you trying to extend the life of one gigabit tools is a false economy, buying ten gigabit visibility tools that can’t really do 10 gigabit is an equally false economy for all the same reasons.

It’s time for the vendor community to wake up before 100 Gig bits them in the rear….

This year’s stand out speaker was  Major General Jonathan Shaw, head of the defence cyber operations group at the MoD. The MoD have taken responsibility for allocating the UK governments 650 million pound investment in national cyber security infrastructure and are thus under the national microscope.

Speaker presentations from the conference can be found here. The audio from the event can be found here

The report suggests that 2012 is likely to be a ‘pivotal year’ for banks and investment firms as they try to stay ahead of the IT security curve. At the heart of the report is a recurrence of the idea that companies need to work on the assumption that they are already infected and learn to live with it. This concept isn’t new, but it certainly seems to be gathering a following. Exactly what this means and exactly how organisations are supposed to learn to live with it is a little unclear and will no doubt be the subject of much discussion during 2012. From our perspective, it highlights the need to monitor what’s leaving your network as accurately and as diligently as what’s trying to get into it and to put appropriate network recording capabilities in place.

Of the 10 threats listed below most are reasonably well understood. The one that stands out is the last point 10 (increased scrutiny from the regulator). It’s clear from recent very public incidents during 2011 that organisations haven’t been as transparent as they perhaps should have been – possibly because they don’t have good answers to some very basic questions. Whatever reason, the fact remains, people have a right to know when their data has been compromised and if the industry best practice of network recording isn’t adopted universally, then the SEC needs to step in and mandate the requirement.

Booz Allen identify the Top 10 threats to be:

1. The exponential growth of mobile devices drives an exponential growth in security risks. Every new smart phone, tablet or other mobile device, opens another window for a cyber attack, as each creates another vulnerable access point to networks.
2. Increased C-suite targeting. Senior executives are no longer invisible online. Firms should assume that hackers already have a complete profile of their executive suite and the junior staff members who have access to them.
3. Growing use of social media will contribute to personal cyber threats. A profile or comment on a social media platform – even by the CEO’s son or sister — can help hackers build an information portfolio that could be used for a future attack.
4. Your company is already infected, and you’ll have to learn to live with it – under control. Security should remain a priority, but today’s risks and threats are so widespread that it will become impossible to have complete protection – the focus of cyber security tactics increasingly must be to analyze, detect and expunge threats inside your system.
5. Everything physical can be digital. The written notes on a piece of paper, the report binder and even the pictures on the wall can be copied in digital format and gleaned for the tools to allow a hacktivist-type of security violation, and increasingly this will be a problem.
6. More firms will use cloud computing. The significant cost savings and efficiencies of cloud computing are compelling companies to migrate to the cloud. A well designed architecture and operational security planning will enable organizations to effectively manage the risks of cloud computing.
7. Global systemic risk will include cyber risk. As banks and investment firms continue on the path to globalization, they will become increasingly inter-connected. A security breach at one firm can create negative ripple effects that greatly impact systemic risk in financial markets.
8. Zero-day malware (malicious software) and organized attacks will continue to increase. Like a vicious, insidious virus that mutates, the tools of cyber criminals adapt and change constantly, rendering the latest defenses useless. Firms need to be prepared to adapt quickly as well to zero-day malware and the tactics of organized crime and foreign adversaries that are increasingly used today.
9. Insider threats are real. The accidental insider breach will continue to be the primary source of compromise for the Advanced Persistent Threat (APT) and other attacks. Organizations need to focus on security awareness training and internal monitoring to detect intentional and accidental insider access.
10. Increased regulatory scrutiny. Recently, the Securities and Exchange Commission introduced guidelines that require companies to report incidents that result, or could possibly result in, cyber theft or a risk of compromised data considered material.

First, The Ponemon Institute published a piece of research that explores the true cost of a data breach, looking specifically at the impact on brands and company reputations. We’ve long been of the opinion that organisations radically underestimate the real cost of data leakage, and as a result tend to under invest in the tools to prevent, identify and remediate security issues, so it was fascinating to see an organisation attempt to quantify ‘true cost’.

Researchers surveyed 843 leaders from a wide range of organizations. The first task was to estimate the economic value of their organisations’ corporate brand or reputation – responses ranged from a value of less than $1 million to greater than $10 billion. Next researchers set out to understand what the impact on the brand would be from a variety of different data loss scenarios.  The results are quite staggering.

• The average fall in the value of the brand ranged from $184 million to more than $330 million
• As a percentage of their annual gross revenue, the economic value of reputation and brand ranged from less than 10% to greater than 5 times revenue
• The average was a 31% decline.

Other interesting insights from the survey include:

• The most damage comes from the loss of customer-sensitive information
• On average organizations had experienced 2.7 sensitive data loss events in the last two years
• Depending on the scale of the breach it could take longer than a year to recover and restore reputation and brand image.

The report concludes that “The findings further demonstrate how devastating a data breach can be for an organization and how important it is to reduce the risk of such an incident.”  When compared to the real cost of a breach, the cost of the technology that can help to provide visibility into these issues and minimize the impact of them simply pales into insignificance.

The second study that came out this week was from our friends at Symantec in the form of their 2011 State of Security Survey which is again a great piece of content.  Symantec surveyed 3,300 organizations from across the full spectrum of sizes and verticals to understand how their views on cyber security were changing year on year. The report drew 4 key insights:

1. Cyber security is important to business and most important fears relate to cyber attacks followed by IT incidents caused by well meaning insiders and internally generated threats
2. The drivers of security are changing with mobile computing and social networking topping the list of concerns. 49% of attacks site hackers as the primary cause of problems
3. 71% of organisations experienced attacks in the last 12 months with 92% of respondents reporting losses from these attacks. 20% of respondents admitted to the loss of customer data (identified by Ponemon’s research as far and away the most brand – damaging. 20% of businesses lost at least $195,000 as a result of cyber attacks which is obviously a far cry from the full cost as defined by Ponemon’s analysis
4. Organisations are (finally) recognising the need to both have preventative technologies in place, but also have forensic capabilities that enable them to go back in time and work out what happened when it all goes wrong. The bad news is that less than 50% of organisations feel that they are on top of things, which leaves a lot of room for improvement (and a lot of opportunity for hackers).

The conclusions from the two studies are pretty obvious -

• Organisations are radically under-estimating the real cost of cyber attacks. It’s clearly time to recalibrate
• Organisations still have much to do if they are to protect themselves against an every changing threat landscape and must be able to look for security attacks from the inside-out as well as from the out-in
• Organisations must be prepared to invest more – and be willing to invest in both reactive and retrospective tools.

The article highlights the case of an international hedge fund that discovered (after a phone call from a friendly stranger) that they had at least 15 compromised PCs within their organisation and had absolutely no knowledge or visibility of the highly sensitive information that was leaking. The reality is that any company that has valuable intellectual property is a target for cyber attacks.

The article goes on to review the TechAmerica and RSA security summit (July 13 and 14) which focused on Advance Persistent Threats (something close to RSAs heart). The conclusion of the conference ended up being very straightforward – organisations need to start from the assumption that they have been breached and invest time in proving to themselves that they haven’t. Not the other way round.

The report highlights 3 areas where organisations (large and small) should be focusing their attentions

• Closing the exposure window and limiting damage through efforts to compartmentalize systems, stop sensitive data egress and go back to the core principles of IT security such as ‘least privilege’ and ‘defense in depth.’

• Understanding what digital assets are important to protect, where they reside, who has access to them and how to lock them down in the event of a breach.

• Preserving, aggregating and reviewing data to detect a potential intrusion but also for post-event forensics.

Cisco’s Gavin Reid said organizations that don’t have a good record of internal network activity stretching back months or even years have little chance of understanding the breadth of an APT attack after it occurs.

“Without that information, there is very little victims can piece together to understand what came in, what went out, and who else was involved,” Reid said.

At Endace we’ve long been of the opinion that effective network security (and network performance management for that matter) is a function of highly accurate, network-wide packet-capture and analysis. Some of the largest corporations in the world trust us to help them get visibility into their 10Gb/s, 40Gb/s and even 100Gb/s networks. So we know a bit about the subject.

From our perspective there are three things that organisations must deploy if they are to build effective network security

• High performance IDS with a full and complete rule set from multiple different sources which is dynamically managed by a team of people that know what they are doing

• Effective data archiving practices (100% accurate packet capture guarantee goes without saying) to ensure that engineers can go back in time and look for anomalies in historical traffic

• Highly efficient investigation tools for finding needles in haystacks (which is a huge issue at 10Gb/s plus)

If you believe that you have something worth stealing then you simply cannot afford not to put the appropriate tools in place. In reality, assuming you’ve been breached is the only safe way to behave. Because the chances are you have….

We were represented by Kevin Formby who explains the role Endace plays in the video below.

It’s clear that there’s real concern on the front line that even large well-funded corporations in the Fortune 500 list simply don’t have the infrastructure in place to deal with relatively straightforward security breaches and are blind to relatively basic network security attacks.

Highlights of the survey include:

• 84 percent of respondents have concerns about their incumbent vendors’ abilities to manage 10Gb/s throughput environments

• 47 percent of respondents believe they are missing potentially significant network events due to failing or under-performing systems

• 78 percent of organizations recognize “strong correlation” between network security and their ability to comply with PCI and ISO27000

• 65 percent of organizations surveyed do not record network traffic for the purposes of forensic analysis of network events

• 33 percent of organizations  reported having experienced  some kind of data loss in the last 12 months with 39 percent being unable to accurately identify what was lost

• 42 percent of organizations admitted to having been the victim of a cyber-attack in the last 12 months, with 67 percent of those admitting to serious problems investigating the attack

The full report can be downloaded from here:

ESM 6.0 is an important release and, for the first time, sees SNORT move from being a native feature of OSm (our Operating System for Monitoring) into a virtual container on the Endace System. This means that ESM (which includes the SNORT image) will sit alongside third-party applications hosted in the Endace Application Dock. It of course retains its place as a core feature of our Application Suite which is included as part of the base configuration for all Endace Systems.

Over the last couple of years there has been much debate inside the community about the performance impact associated with ‘virtualising’ SNORT (as opposed to a running it natively) and the engineering team here in New Zealand have worked extremely hard to optimise our implementation. We are extremely pleased with the results and can announce that the impact on SNORT performance is officially ‘negligible’.

Moving SNORT into a virtual container brings with it a number of important benefits for customers using our IDS within their Monitoring and Recording Fabrics:

• It decouples our IDS release schedule from our OSm releases schedule, which allows us to be more agile and flexible with our IDS product roadmap
• With ESM in a virtual container customers will be able to manage the version(s) of SNORTthey are running more dynamically, which enables them to stay in tune with the SNORT community
• It enables different rule sets to be assigned to different virtual containers in the system, thereby allowing different polices per VLAN to be supported.

In this latest release we’ve also (re)introduced a number of key reports. These include:

• Top applications detected
• Top destination IPs by signature
• Top source by signature
• Top source and destination  by signature
• Alerts by source TCP port
• Alerts by source UDP port

Our network intrusion detection is starting to gain thanks to the open and flexible nature of the application – and of course the fact that our IDS sensors don’t just do IDS. They also support…. Netflow generation, Analytics, Data mining, Forensics and anything else that you want them to support!