Big Data and Incident Detection- A Promising Future, but Mind the PResent

By Matthew Neely, SecureState Director of Strategic Initiatives

The Hype

Recently there’s been a lot of talk in the media and at security conferences about using big data tools to detect incidents.  As an information security geek, the prospect of using big data tools like Hadoop and NoSQL to plow through piles of IDS alerts and logs is exciting.  However, as a management consultant who needs to help SecureState’s clients develop implementable and supportable solutions my thought on this is “meh” for most of my clients.

The problem is that most people still haven’t spent enough time working on the basics of information security to be ready to tackle these advanced projects.

Focus on Security Basics

In my experience, most people are still struggling with setting up, tuning and maintaining out of the box log management and monitoring products. So before you look at using big data tools to manage and monitor your security events, make sure you at least have the basics covered in your security strategy:

• Understand any regulatory requirements for monitoring and log management and make sure you are meeting or exceeding them.
• Document logging requirements and apply these requirements to all systems.
• Properly tune the monitoring system so it alerts on high priority events and produces minimal false positives.
• Have enough trained staff to properly monitor alert and review logs when needed.
• Have a documented Incident Response plan.
• Have a properly trained and staffed incident response team.
• Perform Incident Response Tests annually to verify your ability to properly respond to incidents.

Accomplishing the items on the list above can be daunting, so it’s also important to understand the limits of your internal team and get help when needed. You can also consider outsourcing your event management and response to a Managed Security Service Provider (MSSP).

If Not Now, When?

If you’ve accomplished everything on the list above, then is time to look at using these new tools? We’ll it depends on the priorities of your business and security program. For most people, I would still recommend holding off until turnkey big data solutions are created or log management, and SEIM products offer out of the box integration with these tools. From looking at the current research in this area I estimate we’ll see big data analysis features coming to commercial SEIMs in two to three years.

Until then, if you are a security geek by all means experiment and play with these tools. They have a lot of potential. Just make sure they don’t distract you from performing the basics.