Threat hunting with SaltOpen
The SaltStack platform is an open-source and Python-based configuration management software. Using Salt, developers and system administrators can better manage their infrastructure and deploy code and changes.
In this article we’ll be running SaltOpen, the open-source version of Salt Enterprise, to see how both the master and minion installations can be used to perform various threat hunting functions for threat hunting teams in Cybersecurity departments.
Overview of Salt
Salt is made up of six modules that can influence its functionality and management. These modules determine the actions that the Salt user intends to carry out. They include:
- Execution modules: These represent functions that are available for direct execution from the remote execution engine
- State modules: These make up the back end for the Salt configuration management system
- Grains: These detect static information about a system
- Render modules: These make it possible to render information to the Salt state system
- Returners: These manage arbitrary return locations
- Runners: These are master-side convenience applications which are executed by the salt-run command
Initial setup and first run
Installation of Salt Open is a pretty straightforward process; a good guide to it can be found here. In our case, we’ll be running the master and minion on Ubuntu 18.04 systems. Once you have installed the salt-master on your master and salt-minion on your minions, you can start them both using the following commands respectively.
Salt master configuration
sudo systemctl start salt-master
The following screenshot shows the result of running the command above.
We need to be able to reach our minions. To do this, we edit the salt master config file to contain the location of the salt minion. This is done by editing the file /etc/salt/master to add the line ‘interface: 192.168.100.30’, where the IP address is the location of the salt minion.
Salt minion configuration
sudo systemctl start salt-minion
The screenshot below shows the result of running the command above.
We also need to configure the minion to be able to find the master. This can be done by editing the minion config file found at /etc/salt/minion.
In our case, we edit the file to include the line ‘master: location’ (where location is where the master is located). Our master is located at 192.168.100.29, so we feed that into the file.
We then also edit the salt minion ID to hold a descriptive name of the salt minion. This is done by editing the file /etc/salt/minion_id and adding salt-minion-01 as our minion ID.
Next, we need to edit the hosts file to contain a DNS record for the salt master. This is done by editing /etc/hosts and including the line ‘192.168.100.29 salt’.
Now we can print the master-key fingerprint by running the command
sudo salt-key -F master.
The result is shown below:
We then copy the master.pub key above, feed it into the master_finger section within the file /etc/salt/minion in the salt-minion and restart the salt-minion service.
We then head back to the master and issue this command: sudo salt-key -L. We’ll receive the following:
To accept the key above, we’ll issue the command sudo salt-key –accept=’salt-minion-01’.
The output is shown below:
Now when we list the keys using ‘sudo salt-key -L’, we should be able to see the accepted key (minion), as shown below:
At this point, the master and minion are connected, and the keys can be verified by issuing the following commands respectively:
Within the master: sudo salt-key –finger salt-minion-01
Within the minion: sudo salt-call –local key.finger
The output should be a key that matches on both hosts.
Testing the connection
To ensure that you can issue commands to the minion, the following command can be run on the master:
salt salt-minion-01 test.ping
You should have a similar result to the following:
Threat hunting with SaltOpen
Threat hunters can make significant use of SaltOpen to interrogate minions in order to support their hunting activities. The way this is done is by making use of salt commands which take the form below:
salt section module.function
In the command above, “section” can be replaced with a minion ID to refer to a specific minion or can be replaced with “*” to mean all the minions. Module refers to execution modules and function refers to the functions that can exist within each module.
What follows is a hypothetical scenario that can be used to inform the queries we make to the minion in order to demonstrate a bit of the capabilities of Salt.
Intrusion scenario — Ubuntu Linux server breach (salt-minion)
On March 12th, 2019, our Intrusion Detection team discovered unauthorized access on our JBoss production server. The server was being used by the adversary to launch scans within the internal network according to our SIEM logs. Fortunately, our effective and quick response ensured that the intrusion was stopped before significant damage could be done. What follows is the discovery and analysis of the malicious actor’s actions within the JBoss server (salt-minion), as collected using Salt (salt-master).
To begin our analysis, we wanted to determine whether our compromised system was still up or had been formatted. We made use of Salt Runners, using the command sudo salt-run manage.status, as shown below.
We wanted to examine the possibility of locally-stored exfiltrated data by first querying the disk usage within the breached server. Our assumption was that this server could have been used to store files looted from other systems from the network.
The command used was sudo salt salt-minion-01 disk.usage.
We were able to corroborate the results above with our disk usage snapshot taken a week before and did not find suspicious disk usage, which would have prompted us to check for large files.
However, the adversary had installed Nmap, which was being used to conduct unauthorized scans within our network.
We examined the server for suspicious files under commonly targeted sub-directories. The command used was sudo salt salt-minion-01 cmd.run ‘ls -l /etc’.
We then checked for suspicious accounts within the /etc/passwd file. Our interest was searching for accounts with UID 0 and GID 0. It seemed that the root account was being used to perform the unauthorized activities.
We wanted to check whether the attacker had installed any rootkits. We therefore installed chkrootkit and did a rootkit scan.
We couldn’t discover rootkits, so we decided to examine the running processes for suspicious ones.
We examined the breached server for any suspicious listeners and discovered a Netcat root listener, as shown below. The attacker used this listener to issue commands remotely from the Internet.
We also discovered a malicious cron job using the command sudo salt ‘salt-minion-01’ cron.list_tab root.
The attacker had installed a malicious tool that would inform him/her of any new live subdomain assets that we would roll out into production.
Further audits did not expose any unauthorized actions. However, we discovered that the adversary had managed to perform unauthorized access by exploiting a JBoss vulnerability using an open-source tool — jexboss.
The table below summarizes the threat hunt by listing the identified indicators along with the attack phases. The indicators listed below are what we discovered as being performed by the adversary.
|Reconnaissance||Unauthorized access from log files|
|Exploitation||JBoss jmx-console misconfiguration leading to root|
|Installation||/usr/bin/nmap + /root/sublert|
|Actions on Objectives||Unauthorized access and scanning of internal network|
We have configured and made use of SaltOpen, which is the open-source alternative to Salt Enterprise, to perform some threat-hunting activities in order to learn about adversary activities on a hypothetical attack scenario. However, what we have covered is quite minimal compared to the capabilities of Salt. We encourage you to take a look at the documentation here to access a much wider set of modules and functions that are available within the salt stack.
- Salt Cookbook, Packt
- Salt in 10 Minutes, SaltStack
- saltstack/salt, GitHub
- How to Install Salt / Saltstack Master & Minion on Ubuntu 18.04 LTS, Computing for Geeks