Hackers are currently attacking vulnerable SaltStack systems – here’s how

The software SaltStack is a commercial, open source management tool that allows system administrators to automate the orchestration and configuration of their servers. That means with its high speed data connectivity and fast communication, the software can ensure that specific packages are installed and certain services are running, as well as even being able to…

The software

SaltStack is a commercial, open source management tool that allows system administrators to automate the orchestration and configuration of their servers. That means with its high speed data connectivity and fast communication, the software can ensure that specific packages are installed and certain services are running, as well as even being able to query and execute commands on individual machines and remote nodes.

Each server runs an agent, known as a minion, that publishes reports and receives update instructions from a master. A huge number of these minions are persistently connected to their publisher server in order to listen to instructions and messages from the master. When needed, they also connect to their request server to send results to, and request files from, the Salt master.

Image: https://docs.saltstack.com/en/getstarted/system/communication.html

The vulnerability

On 15th April 2020, an internet wide scan by F-Secure disclosed that over 6,000 Salt masters were publicly exposed and at risk of compromise from a vulnerability that was detected in SaltStack back in March. The vulnerability allowed attackers to bypass authentication controls, connect to the request server, and publish arbitrary control messages, meaning a malicious actor could read and write files anywhere on the master server system and gain full remote command execution as a root user.

The vulnerabilities were categorized as two separate CVEs (CVE-2020-11651 and CVE-2020-11652), the first being listed as an authentication bypass, unintentionally exposing unauthorized network clients to gain functionality, and the second being directory traversal, where incorrectly sanitized inputs allow unconstrained access to the entire file system of the master server.

The exploit

F-Secure stated that they would not be providing a Proof of Concept (PoC) exploit code so as not to harm any SaltStack users that were slow to patch. Instead, they left the exploit as an exercise and suggested that any competent hacker would be able to create exploits for the vulnerabilities in less than 24 hours. Here’s how we did it.

The code

It was time to start writing some code. This is mostly from the Python ZeroMQ documentation.

We got this response.
b'\xa8bad load'

Let’s take a look at the Salt Docker output.

[CRITICAL] Could not deserialize msgpack message. This often happens when trying to read a file
not in binary mode. To see message payload, enable debug logging and retry. Exception: unpack(b)
received extra data.
[ERROR ] Bad load from minion: ExtraData: unpack(b) received extra data.
[WARNING ] /usr/lib/python3.7/site-packages/salt/loader.py:772: DeprecationWarning: dist() and
linux_distribution() functions are deprecated in Python 3.5
ret = funcs[key]()

A couple of things jumped out straight away. A deserialize error was occurring, but first, we wanted debug mode in order to see the payload and get more data. We killed the running container and edited our Docker file.

FROM saltstack/salt:2019.2.0
EXPOSE 4506
EXPOSE 4505
CMD salt-master -l debug

We rebuilt and ran it again, which revealed a lot more output in the Docker logs. Another run of the payload provides even more details.

A quick google helped us to identify the “msgpack deserialization failure” message; the top results were for the SaltStack Github repository! We discovered from here that MessagePack is like JSON, but “faster and smaller”. There was a Python library available, so we installed it to see what happened next.

We were still receiving a bad load message, but looking back at the SaltStack logs, we found this.
[ERROR ] Bad load from minion: KeyError: 'enc'

What happens when we set the message to include an ‘enc’ key?

msg = msgpack.packb({"enc": "world"}, use_bin_type=True)

[ERROR ] payload and load must be a dict. Payload was: {'enc': 'world'} and load was None

The system now wanted to load:
[code]msg = msgpack.packb({"enc": "world", "load":{}}, use_bin_type=True)[/code]
...and then got really upset.

It killed our socket! We had to find out what it wanted. The stack trace told us.

Thankfully, because SaltStack is open source, we were able to look up the source code on Github and find the source code for master.py. As we were working on an older version, our line numbers didn’t exactly line up with the latest version. So, a quick ctrl+f for _handle_payload found our function, pretty close to where we thought it would be.

We’ve got a lot of information here, including an example payload, which meant we could start constructing our valid payload. However, we know _prep_auth_info() is our vulnerable method, so we quickly searched for this in the repository.

Perfect! A test case that seemed to be for our exact requirements. This search also divulged some test cases against specific CVEs, so at this point, we had everything we needed.

Why wasn’t it working? This was supposed to be an authority bypass, but we were given an authorization error. It started to click when we unpacked the raw message.

We jumped onto the Salt master, used cat to read the root key, and compared.
/var/cache/salt/master # cat /var/cache/salt/master/.root_key
qxWqaRIfKk/MoBDKkz29HAQh49XNnDu8aNPuLoSTSL1FLlUtizE9gigkS9MJuNeXAHoEIwxvlZ0=

This is a valid root key! We now had the ability to compose and send any command we wanted to the master or any of the minions. 

Here’s our working PoC to read and write files.

The fun bit

The second part of the CVE suggests we can use _send_pub to run a job on every minion. There’s an example in the same test_clear_funcs.py that shows us how to construct a payload.

This isn’t targeting the master. We needed to get a minion connected in order to fully test the system. Once again, this proved easy enough to achieve with a little bit of Docker.

Over at the master, we approved the minions key by running salt-keys -A and tested connectivity with salt '*' test.ping. With everything now connected, we could test our exploit payload. Fingers crossed!

Success! Sending this payload to the Salt master resulted in the job being executed on the connected Minions. We could see that the file was created in the /tmp/ directory and was owned by the root user; this means we had full control of every minion at the highest privilege level.

To make the jobs appear legitimate and avoid any issues with duplicate job IDs that may block multiple runs, we added some code to generate correct IDs.

Trying the same techniques on the master resulted in failure. There were only a limited set of functions exposed to us that we could use. These wheel functions do expose read and write privileges to the Salt master, however, which we could abuse to gain additional access to the master. Examples of this could include reading SSH keys or other sensitive files, writing scripts to root profiles, or crontabs to be executed at login or reboot.

Now, find out how to identify whether you were infected by this vulnerability or not with our blue team-style report on the CVE.

If you'd like to get hands on with these vulnerabilities and try out the techniques discussed above in a safe and secure environment, head over to Immersive Labs Lite to try it for free.

We help businesses to increase and evidence human capability in every part of cybersecurity.

Legal