General security

Top 10 Open-Source Security Tools Released by Tech Giants

We have always wondered how tech giants have been able to keep their security so tight? Do they use the same tools that are available for the rest of us? Alternatively, they have allocated a small portion of their massive resources dedicated to coming up with something different? Finally, we have our answers. Many tech companies such as:

Netflix

Google

AOL

Facebook

Twitter

Etsy

have created open-source tools which are typically a result of their needs. Let's see some of the tools released by these companies:

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

RAPPOR

RAPPOR is a tool released by Google that stands for Randomized Aggregatable Privacy-Preserving Ordinal Response is a privacy tool that collects crowdsourced statistics from end-user client software without compromising user's privacy. RAPPOR is built by using Python and R, and its source-code can be found at https://github.com/google/rappor

As of now, RAPPOR has only been tested on Linux platforms. RAPPOR has a few dependencies which can be installed by:

$ ./setup.sh

followed by building the code by:

$ ./build.sh

and finally, to run the demo:

$ ./demo.sh

The demo strings together the Python and R code to:

• Generate simulated input data with different distributions
• Runs it through the RAPPOR privacy-preserving reporting mechanisms
• Analyses and plots the aggregated reports against the true input

The output is written to an HTML file that can be accessed by any browser.

Conceal

Conceal is a set of JAVA APIs for Android. Released by Facebook, Conceal is used to perform encryption and authentication of data on large files in public locations such as SD cards in a fast and memory efficient way. It was designed to allow developers to performs said functions on old Android versions with low memory and slow processors. Its source code can be found at https://github.com/facebook/conceal

It uses a 256-bit key from v 1.1 as compared to the 128-bit key used I the versions before. Facebook personally uses Conceal to store image files on SD cards. Instead of implementing any crypto, Conceal uses specific cryptographic algorithms from OpenSSL. It also uses fast modes like AES-GCM and HMAC-SHA1 by default. By shipping selected number of encryption algorithms from OpenSSL, conceals and makes the OpenSSL library much smaller (85KB) than as when being used wholly.

Image is taken from http://facebook.github.io/conceal/

Skyline

Skyline is a real-time anomaly detection system that enables us to monitor hundreds of thousands of metrics without the need to configure a model for each and every single one like in Nagios. It primarily consists of two services:

• Horizon Agent: The Horizon Agent is responsible for listening to new data-points and keeping the time series clean and up-to-date in Radis.
• Analyzer Agent: The Analyzer Agent is responsible for analyzing every metric for anomalies.

Skyline is released by Etsy and made in Python. Its source code can be found at https://github.com/etsy/skyline

However, Skyline is no longer actively maintained.

Image is taken from https://github.com/etsy/skyline

Google Rapid Response

GRR (Google Rapid Response) is an incident response framework to help with remote live forensics. It is an alternate solution to FireEye/Mandiant's MIR incident response platform. Built using Python, GRR is a cross-platform application that works on Linux, MacOS, and Windows.

It has a client-server architecture which means that a Python client is installed on the target machines and a server is used to communicate with them.

Client Features

• Cross Platform
• Live memory analysis using open-source memory drivers for Linux, OS X, and Windows via the Rekall memory analysis framework
• Powerful search and download capabilities for files and Windows registry
• Secure communication infrastructure designed for Internet deployment
• Client automation update support
• Detailed monitoring of client CPU, memory, IO usage and self-imposed limits

Server Features

• Basic reporting infrastructure
• Basic system timelining features
• Fully scriptable IPython console access
• AngularJS web UI and RESTful JSON API
• OS-level and raw file system access using the SleuthKit (TSK)
• Enterprise hunting support
• Automated scheduling of recurring tasks
• Fast and simple collection of hundreds of digital forensic artifacts
• Fully fledged response capabilities for handling most incident response and forensics tasks
• Asynchronous design allows future tasks scheduling for clients, designed to work with a large fleet of laptops

GRR's repository can be found at https://github.com/google/grr

Secure Headers

Secure Headers is an open-source tool released by Twitter. It is a Ruby Gem that allows for the automatic application of headers related to security, including Content Security Policy (CSP) to prevent Cross-Site Scripting (XSS), HTTP Strict Transport Security (HSTS) for protection against Firesheep attacks and X-Frame-Options (XFO) to prevent clickjacking.

Its repository can be found at https://github.com/twitter/secureheaders

Additional headers include:

• X-XSS-Protection: Cross Site Scripting heuristic filter for IE/Chrome
• X-Content-Type-Options: Prevent content type from sniffing
• X-Download-Options: Prevent file downloads opening
• X-Permitted-Cross-Domain-Policies: Restrict Adobe Flash Player's access to data
• Referrer-Policy: Referrer Policy draft
• Clear-Site-Data: Clearing browser data for origin
• Public Key Pinning: Pin certificate fingerprints in the browser to prevent man-in-the-middle attacks due to compromised Certificate Authorities.

OSquery

OSquery is an open-source, cross-platform Operating System instrumentation framework. It offers a low-level operating system using a system that represents operating system attributes such as loaded kernel modules, running processes, and open network connections in SQL tables that can be queried and monitored.

It essentially exposes an operating system as a high-performance relational database using which we can use SQL-based queries to explore the data in the operating system.

The repository of OSquery can be found at https://github.com/facebook/osquery

Note: The Facebook Bug Bounty Program also includes OSquery.

OSquery can be downloaded from https://osquery.io/downloads/

Sketchy

Sketchy is an open-source task based API used for taking screenshots and scrapping text from websites. It is a Python based tool released by Netflix back in 2014.

Its repository can be found at https://github.com/Netflix/sketchy

Sketchy has a capture module that contains all the information associated with screenshots, scrapping, and storing HTML files from an URL. Screenshots, text, and HTML can be stored locally or in an S3 bucket.

It used PhantomJS with Lazy-rendering to ensure Ajax heavy sites are captured correctly. It also uses Celery task management system enabling users to scale Sketchy accordingly and manage time intensive captures.

Scumblr

Scumblr is an open-source web application released by Netflix that allows periodic syncs of data sources (such as GitHub repositories and URLs) and performing analysis on the identifies results. Scumblr helps streamline security through an intelligent automation framework to help identify, track and resolve security issues faster.

Its repository can be found at https://github.com/Netflix/Scumblr

Sync Tasks

• Route53 DNS: Sync FQDNs from Route53 DNS
• GitHub: Sync results from GitHub repositories
• Manual Result Upload: Specify a new line delimited list of results you would like to sync to Scumblr

Security Tasks

• Curl: Execute curl commands to identify vulnerabilities or issues against Scumblr results
• Bandit: Perform static code analysis against Python projects
• GitHub Search: Search GitHub for secrets, anti-patterns, and vulnerabilities in your repositories
• Brakeman: Perform static code analysis against RoR (Ruby on Rails) projects

Search Tasks (legacy)

• Twitter
• YouTube
• Facebook
• Google
• iTunes Store
• eBay
• Google Play
• Reddit
• RSS Feeds
• Certificate Transparency

Security Monkey

Security Monkey is an open-source Python-based tool released by Netflix back in 2014. It is essentially a monitoring and security analysis tool for Amazon Web Services and (EC2) and Google Cloud Platform (GCP). It provides a single UI to browse and search through all accounts, regions, and cloud services. It remembers the previous state of the machine and can show exactly what changed and when. It also includes components for monitoring, developing, and executing actions based on policy rules, notifying users when audit rules are triggered and storing configuration histories for forensic and audit purposes.

Its repository can be found at https://github.com/Netflix/security_monkey

Security Monkey can be installed using Docker as well: https://github.com/Netflix/security_monkey/blob/develop/docs/docker.md

Moloch

Moloch is an open-source network forensic tool designed to analyze traffic. It enables large-scale IPv4 packet capturing, indexing and storage with a web interface to browse, search and export the PCAP data. AOL released Moloch in 2012.

Its repository can be found at https://github.com/aol/moloch

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

Moloch augments the current security infrastructure to store and index network traffic in standard PCAP format. You can also use the exported packets is other tools such as Wireshark during the analysis. Moloch is also protected by using HTTPS with digest passwords or by using authentication providing web server proxy.