Taming the Big Data Beast with Endace Network Recorders

It was bound to happen sooner or later. Big data, the darling buzz words that have transcended the boundaries of the tech industry and proliferated every water-cooler conversation from Silicon Valley to Shanghai took a bit of a beating recently. You see, there has been a little scandal that has percolated through every global media outlet out there and stems from allegations that the National Security Agency (NSA) has been secretly compiling and storing people’s personal web and phone records (without their consent, of course), leading to a much larger debate around the validity of such extensive data-gathering operations in general.

However, like any accused criminal that has to yet to be proven guilty, unless all the facts are gathered and the jury hears both sides of the story, it would be premature to pass sentencing. It’s important to look past the negative publicity that ultimately detracts from all the positive aspects of big data and the equally positive impact it will invoke on our society as a whole.

Big Data – It’s Here to Stay (So learn to love it)

Here is the blunt truth – Big data will seep into every crevice of our lives and the inherent proliferation of big data is a requirement that every organization needs to look at in order to maximize their long-term viability. In fact, regardless of how big data is defined, 58% of respondents to a searchstorage.com poll say their companies are dealing with it, and 41% have even bought — or plan to buy — some new gear to grapple with big data.  This aligns with a survey Endace commissioned that stated that half of all respondents (54%) said that over the next two years, big data would increase their organization’s need for network bandwidth in the data center by more than 50% – data will continue to explode, and organizations need to prepare themselves by expanding their network bandwidth to make their data center ‘big data’ ready.

The data speaks volumes. According to market research firm IDC’s Digital Universe Study[1] 2.8 zettabytes of data would have been created and replicated in just 2012 alone!  With the amount of data in the world increasing at exponential rates, analyzing that data and producing intelligence from it becomes very important. Aside from the ability to keep more data than ever before, we have access to more types of Big Data sources including the following examples which are run by notoriously bandwidth-hungry applications.

• Science and Research (NASA, Human Genome Project)
• Government Agencies (NSA…duh!)
• Social Media Analytics (Facebook, Twitter, Instagram)
• E-Commerce Analytics (Google, Amazon, Netflix, eBAY)
• Healthcare (Insurance agencies, diagnostics)
• Transactional Data (email, office documents, device generated data, web logs)
• Audio and Video (Digital content and imaging, on-demand video)

The Network Recording Market – Solving the Big Data Conundrum

“Network recording” is a feature of products that falls under network performance management (NPM), application performance management (APM), and the network security forensics markets. The latest industry data from Frost & Sullivan in their new report:  “Analysis of the Global Network Recording Market,” finds that the network recording market earned revenue of $345.5 million in 2012 and estimates this to reach $2.04 billion in 2020[2].

Key factors driving the spike in NRM deployments include IT departments adopting longer retention times for saved data (six months or longer) which, in turn, requires expansion of their existing networked data storage. With mandated retention policies like Sarbanes-Oxley to account for, end-users are also required to capture every data packet that is transacted, and retrieve historical data for analysis without any potential data packet loss.  Tracking data leaks, security breaches, and capturing data at high speed rates without dropping even a single packet is vital for the successful functioning of the network.

As you can imagine, the rapid, unprecedented data growth associated with big data deployments stretch any organization’s storage architecture, capability and manageability limits. The increasing volume, speed and wide swath of unstructured business information challenges not only the current limits of monolithic storage solutions, but also those small-to-medium sized businesses (SMBs) as well. It’s for these very reasons that Intelligent Network Recording (INR) devices are a pivotal component of big data deployments with their ability to capture, record and store valuable data.

Endace: We Know a Thing or Two about Network Recorders (and big data)

Our portfolio of network visibility and network recording products are trusted by large organizations all over the world to measure, monitor, analyze, protect and troubleshoot some of the fastest and most complex big data and cloud deployments on earth.

In the aforementioned Frost & Sullivan report, they concluded NRM is roughly split between 10Gb (53%) and 1Gb (47%) recording deployments and nearly all the market growth will be in 10Gb speeds and above. Primary drivers for market growth are the following:

• Increased need to analyze application and service performance
• “After-action” security forensics
• Increased connection speeds being deployed (esp. 10GbE)
• Proliferation of and big data/cloud services.

On the flip side, the primary restraints on market growth are the following (real or perceived):

• Difficulty to store large (“meaningful”) amounts of network recording data for long (“meaningful”) amounts of time
• Difficulties in analyzing largely unstructured data
• Lack of customer understanding of NRM product features

In order to make Endace successful in NRM, we have addressed the primary restraints mentioned above head-on by employing the following strategies:

The “Network Search Engine” is the critical/differentiating attribute of NRM

• The concept of a “Network Search Engine” is the critical attribute of NRMs.  Finding the right data is critical to analyzing it and because the amount of data is increasing exponentially (both due to increased data speeds and the desire to retain more data), the “search and index” problem should be the critical differentiator in whether recorders can be used effectively to find the “needle in the haystack.”
• Endace prioritizes data packets to include network flows and “search analytics” (DPI, alerts, etc.) in order to “fuse” data together from various tools and provide an integrated view of the threat/problem environment.  This also includes concepts such as SDN-enabled search, VM-enabled search (utilizing Endace connection capabilities), and application-aware search (connecting the output of partner tools into the search “stream”).

Adding storage capacity to network recorders should be inexpensive

• The Frost & Sullivan study demonstrates that capacity is seen as one of the key drivers for buying standalone recorders and Endace understands by planning to sell low-cost storage to “back-end” existing products and ultimately provide significantly more recording capacity at a (relatively) low price.
• Existing customers who have fully deployed EndaceProbe Network Recorders will now have an investment protection story by scaling only as their storage capacity requirements dictate when and where they want it.

Partner with leading providers of analytics tools

• Endace understands that consolidation has occurred in the NPM/APM markets and is indicative of customer buying trends and we offer customers a “complete solution” to stay competitive with the integrated solutions.
• Endace has significant alliance/partnership efforts in place with “standalone” analysis tools (Splunk, Dynatrace, etc.) and we market aggressively with them to ensure our partners know how to sell and integrate these solutions as demonstrated by our partnership with Arista Networks.

Educate the audience

• Endace makes it a priority to educate the market on the business impacts/benefits of using NRM, and how it fits into both our overall Endace product portfolio and the NPM/APM markets
• Endace provides key decision-makers with the data they need: quantitative results from Endace deployments (e.g., “We are saving $50M per year from our Endace deployments”); and customer references and success stories across different verticals and target markets.

Target specific vertical customers and vertical application markets

• Endace targets specific markets and partners (financial, government, service/cloud providers) and applications (credit card processing, VOIP, VDI, HFT, leakage detection) where the need for NRM is significant.

In conclusion…

It ultimately comes down to whether you want to take advantage of the inherent business benefits big data provides or let it take advantage of you and get a few more grey hairs in the process. By employing EndaceProbes and visibility infrastructure products you can manage your big data deployments by accelerating response times for network and security issues, maximize IT risk management and improve operational efficiency to capitalize on network uptime.

EndaceProbes are designed from the ground up to capture, index and record network traffic with continuous 100% accuracy, regardless of network speed, or traffic type. They are powerful, efficient, and a critical infrastructure element in every modern data center environment.

Oh, and in case you are wondering, in the end, the jury came to the conclusion that big data was framed for a crime it did not commit and found not guilty.