Security Log Collection for Cloud Solutions

July 13, 2016 by

Frank Siemons

Something all Information Security Controls have in common is the data output they produce in the form of logged events and alerts. With an increase in the size of an organization or an increase in security levels, the size of this data and its storage requirements will also rapidly grow. Traditionally organizations purchase more and more relatively cheap storage to process and archive logs. In some jurisdictions, the retention requirement of log data can be years, so it is easy to imagine the sheer size of these logs after such an extended period.

The recent migration of many services to Cloud Service Providers has created a few challenges for organizations trying to deal with these large amounts of data which can now be located externally, within that same Cloud Platform. Fortunately, many of these CSP's have been quite active in this field, and some exciting new opportunities have opened up as well.

Learn Cloud Security

Get hands-on experience with cloud service provider security, cloud penetration testing, cloud security architecture and management, and more.

Start Learning

Analyze in the cloud

An organization with 1000 staff members and an average network size, can easily generate 100 Gbytes of logs in a single day. If most of that organization's environment is hosted within a cloud Platform, analysis of that amount of data at the organizations local site with, for instance, a SIEM solution is nearly impossible. How would that data get synchronized fast enough to allow for real-time analysis? There is also a potential for an attacker to delay or halt that data stream by for instance generating a large amount of log data (DDOS) resulting in a temporary lack of security monitoring. The most viable solution here is to monitor and analyses the log data directly within the Cloud Platform. A possible hybrid solution is to have a SIEM application or a simple log analysis application running on a cloud-based server and feeding some of the more interesting, correlated or filtered data back to the organizations on-premises (local) environment. As mentioned, CSP's have systems in place to allow customers to configure these solutions based on their requirements.

Microsoft has released a whitepaper for their Azure platform covering items such as Azure Deployment Monitoring and Windows Event Forwarding. Amazon provides similar options, and most CSP's allow customers to deploy their own SIEM or Splunk related services without issues.

Download from the cloud

Security log data can be downloaded from providers on a regular or an ad-hoc basis, even if this is a significant amount. This data can then be fed into an on-premises SIEM solution such as Alienvault or ArcSight for local analysis, and if needed, correlation with other event feeds. A regular download can be based on a (scripted) API connection. This could be scheduled for instance daily or so frequent, that it will appear as if the data is effectively synchronized continuously. This method is often used to obtain data for cloud-based security products as well, such as Cloud Antivirus solutions and Intrusion detection systems. As mentioned, bandwidth usage and the potential for the data feed to be interrupted, limiting security event visibility should be taken into account when planning for this setup. For compliance reasons or deep incident investigations, sometimes a bulk of months of data is required. Due to the sheer size of that data, a download would not be feasible. Cloud Service Providers can usually assist with a customized, fitting solution as well. Amazon, for instance, has developed "Snowball" which is a Petabyte sized, secure data transport solution designed to get large volumes of data into and out of their AWS cloud. Other providers have similar options available because these bulk data requests are not uncommon.

Upload to the cloud

Some organizations are not looking to download security data from the cloud; they need to upload it to the cloud environment instead. This could be the case for instance if there is a SIEM product in their cloud environment. As mentioned, this is possible because some organizations simply produce much more security log data in their cloud environment than they produce locally. That locally produced log data will need to be uploaded to the cloud for analysis and correlation.

It could also provide a reliable form of off-site storage, for compliance or Data Redundancy purposes. An attacker can target security log data and have a secure, off-site copy is an information security best-practice. Organizations planning to upload a once-of large amount of data to a Cloud Service Provider will run into the same issue mentioned earlier with regards to bulk data downloads: the amount of time for the upload and the required bandwidth sometimes makes this impossible. Shipping the data on secure hardware will then be the only solution.

SIEM as a Service

Dedicated third party cloud-based Security Operations Centre (SOC) providers are also gaining popularity. Loggly is an example of an organization that allows customers to upload their security log data. The Loggly SOC monitors and analyses that data and alerts the customer where needed. This setup is starting to gain the title "SOC as a Service" or "SIEM as a Service" (SaaS). There are more and more of these SaaS providers available every year, such as AlertLogic and Proficio and it is likely this trend will continue to grow further. Using a SaaS provider means organizations do not need to setup their own, highly skilled 24/7 Security Operations Centre at great expense. It is important to take into consideration, however, that the required bandwidth, service availability and possibly compliance and local regulations mean that this solution is not the best option for every organization.

Learn Cloud Security

Get hands-on experience with cloud service provider security, cloud penetration testing, cloud security architecture and management, and more.

Start Learning

Conclusion

The challenges with security log data that cloud customers have had to deal with over the last years have mostly been addressed, with a wide range of solutions now available. Most of these solutions create a hybrid-like cloud configuration where part of the data resides locally, and part of the data resides in the cloud. That data can be and should be synchronized in one form or another by using the relatively easy upload and download options out there. Where data size becomes a challenge, discussions with the Cloud Service Provider of choice could lead to a customized solution that better fits the requirements. The recent introduction of the SIEM as a Service shows that the cloud security field is still very dynamic, and many more exciting developments in this space can be expected in the years to come.