An evening with SVD-2022-0607

Back in June, along with the release of Splunk 9.0, Splunk dropped several security advisories. I’m spending a little time digging in on SVD-2022-0607. Come along with me as we learn together.

The first thing of interest to me about this one is … we’ve been here before. Go back to https://www.duanewaddle.com/splunk-pass4symmkey-for-deployment-client-deployment-server/ and read the update from Martin:

Discussing this feature with Martin Müller, there are some limitations here.  The [broker:broker] authentication for pass4SymmKey only protects the DS “control channel” API endpoint.  There are other API endpoints, like the app download endpoint, that will not require pass4SymmKey.  What this means is that an attacker who knows the names of your serverclasses and apps will still be able to download those apps from your DS without authentication.  See Splunk Ideas https://ideas.splunk.com/ideas/EID-I-391 which discusses this.

The good news I guess is that this is fixed? Now we’ll need to review the docs suggested by the SVD in terms of how to implement the fixes. Also go back and read my old blog post as a refresher – there’s some useful stuff in there. At first glance, I don’t think the docs are accurate / sufficient – I’ll have to do some docs feedback on that.

The Setup

In my test environment I have a DS and 2 UFs. The DS (ds) is running Splunk 9.0.2. One UF (uf9) running Splunk 9.0.2, the other (uf8) is running Splunk 8.2.9. To make understanding what is going on a little easier, I have disabled TLS on the DS REST API as follows in server.conf:

[sslConfig]
enableSplunkdSSL = false

Now I can use tcpdump to help understand the protocol between the UFs and DS, something tells me this will be important. On the DS I will enable the exact settings from the docs page in restmap.conf.

[broker:broker]
authKeyStanza=deployment
requireAuthentication = true

[streams:deployment]
authKeyStanza=deployment
requireAuthentication = true

On the UFs I won’t do anything special but set them up to use the DS in deploymentclient.conf, but in HTTP mode:

[target-broker:deploymentServer]
targetUri= http://ds:8089

The Test

Now we will set up a trivial app (testapp) and serverclasses.conf for it, and watch what happens on both uf8 and uf9. First from splunkd.log on uf9:

11-16-2022 03:38:43.840 +0000 INFO  HttpPubSubConnection [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Running phone uri=/services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5

11-16-2022 03:38:43.845 +0000 INFO  DeployedApplication [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Checksum mismatch 0 <> 15970752176927723036 for app=testapp. Will reload from='http://ds:8089/services/streams/deployment?name=default:test:testapp'

11-16-2022 03:38:43.849 +0000 INFO  DeployedApplication [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Downloaded url=ds:8089/services/streams/deployment?name=default:test:testapp to file='/opt/splunkforwarder/var/run/test/testapp-1668569894.bundle' sizeKB=10

11-16-2022 03:38:43.849 +0000 INFO  DeployedApplication [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Installing app=testapp to='/opt/splunkforwarder/etc/apps/testapp'

11-16-2022 03:38:43.859 +0000 INFO  ApplicationManager [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Detected app creation: testapp

11-16-2022 03:38:43.868 +0000 WARN  DC:DeploymentClient [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Restarting Splunkd...

Now in the splunkd_access.log on the ds:

10.89.0.6 - splunk-system-user [16/Nov/2022:03:37:43.837 +0000] "POST /services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5 HTTP/1.1" 200 407 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms

10.89.0.6 - splunk-system-user [16/Nov/2022:03:38:43.842 +0000] "POST /services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5 HTTP/1.1" 200 471 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 2ms

10.89.0.6 - splunk-system-user [16/Nov/2022:03:38:43.848 +0000] "POST /services/streams/deployment?name=default:test:testapp HTTP/1.1" 200 265 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms

This is pretty reasonable. We see the “phone home” event on both the UF and the DS, and the “download” from the DS and the install to the UF. The app is out there and all is well. But on uf8 running Splunk 8.2.9, things are not looking so great:

11-16-2022 03:51:17.044 +0000 INFO  HttpPubSubConnection [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Running phone uri=/services/broker/phonehome/connection_10.89.0.5_8089_uf8.dns.podman_6f42c58a52ce_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0

11-16-2022 03:51:17.049 +0000 INFO  DeployedApplication [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Checksum mismatch 0 <> 15970752176927723036 for app=testapp. Will reload from='http://ds:8089/services/streams/deployment?name=default:test:testapp'

11-16-2022 03:51:17.051 +0000 WARN  HTTPClient [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Download of file /opt/splunkforwarder/var/run/test/85d323c966bd0bee failed with status 401

11-16-2022 03:51:17.051 +0000 WARN  DeployedApplication [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Problem downloading from uri=ds:8089 to path='/services/streams/deployment?name=default:test:testapp'

11-16-2022 03:51:17.051 +0000 ERROR DeployedServerclass [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - name=test Failed to download app=testapp

Oof. HTTP 401 errors coming back from attempts to download the app. But, this makes sense! We enabled authentication on the DS side for streams:deployment above, and Splunk has already told us that Splunk < 9.0.0 as a deployment client does not send pass4SymmKey authenticated requests for that endpoint. So this is not really a surprise – we are just setting the stage. Let’s see what’s in the pcap! From uf8 (Splunk 8):

POST /services/streams/deployment?name=default:test:testapp HTTP/1.1
User-Agent: Splunk/8.2.9 (Linux 5.15.0-52-generic; arch=x86_64)
TE: trailers, chunked
Host: ds:8089
Accept-Encoding: gzip
Content-Length: 0

From uf9 (Splunk 9):

POST /services/streams/deployment?name=default:test:testapp HTTP/1.1
User-Agent: Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)
TE: trailers, chunked
Host: ds:8089
Accept-Encoding: gzip
Content-Length: 0
x-splunk-lm-nonce: 1f6cc1fd7ef379df64d6709823d383d9
x-splunk-lm-timestamp: 1668569923
x-splunk-lm-signature: LIKvYvPhmg9vkgSh/qMyuX0XxrhEeT3Sc5cr/o3Fqvlqp6oDUFaesPwiOXM=
x-splunk-digest: v2,VgAXgTboCvGPJh8qIdg+MhDy18W1UkAMPPb3Uh75JWwIiu0j/zkub8adNs7t/E8JZdzozwHGGwRdOoCRgMWl0Q==

There’s some new request headers here. The pass4SymmKey is not DIRECTLY in the HTTP headers, but there’s definitely some new stuff here. With enough time, we might be able to figure out exactly what these headers are trying to say or how to validate them. But, the key thing here is that the key code change in Splunk 9.0.0 for SVD-2022-0607 was to add the pass4SymmKey authentication headers to this API call. Once that was done and your environment has rolled out Splunk >= 9.0.0 to every DS client, then the Splunk Administrator can enable the DS to require the headers via restmap.conf. To keep this story going, I’m going to upgrade uf8 to Splunk 9.0.2.

Mysterious Missing pass4SymmKey

If we look back at our DS, we had configured restmap.conf to look for pass4SymmKey in the deployment stanza of server.conf. The btool command will confirm this, but it will also confirm that no pass4SymmKey is set there.

[root@33e28c67e142 local]# /opt/splunk/bin/splunk btool --debug restmap list broker:broker | egrep authKeyStanza
/opt/splunk/etc/system/local/restmap.conf   authKeyStanza = deployment

[root@33e28c67e142 local]# /opt/splunk/bin/splunk btool --debug server list deployment
/opt/splunk/etc/system/local/server.conf [deployment]

So what pass4SymmKey is it using? Thankfully the README/restmap.conf.spec gives us a hint for the DS-side of the connection.

If no pass4SymmKey is available, authentication is done using the pass4SymmKey in the [general] stanza.

We could test this, but I’m going to take it at faith and do what seems to be the “right thing.” I’ll add a pass4SymmKey under the [deployment] stanza in server.conf.

[deployment]
# myfirstpass4symmkey
pass4SymmKey = $7$TZxP0ci6l/e88I7IcRGJuhJR/N1eLgNuUlv3wQqbi29ueVps/9D8SSuSLOMTka8ZAkks

After we do this and restart, both of our UFs are now failing to connect because they don’t have the right pass4SymmKey. There are HTTP 401 responses to their attempts to connect in the splunkd_access.log. I can add that pass4SymmKey to each of them using a [deployment] stanza. Once I do that they’re both able to connect.

Thinking about this some, this means that if I do nothing more than follow the procedure in the Splunk Docs above, my environment may be configured in a way that all of the UFs are authenticated to the DS using the default pass4SymmKey of “changeme“. The docs do suggest that setting a pass4SymmKey is a necessary prerequisite to this, but it turns out a default one for the [general] stanza is already set. A Splunk Admin has to be careful when implementing this in order to not present a thin veneer of “security”.

Also, this complicates UF deployment a bit. If I do configure a robust, non-default pass4SymmKey on the DS then my UFs need to know it before they can ever connect to the DS. Because the DS is usually how UFs are configured, this is a circular dependency. This means that my UFs need to be pre-seeded with the correct pass4SymmKey. This is extra work for the admin as part of deployment. For example, on Windows, the MSI does not include “deployment pass4SymmKey” as one of the supported arguments for the MSI. For Linux, the additional effort is probably not too large because Linux usually requires extra effort to configure the UF to phone home to a DS anyway. But, this definitely adds a new complexity to be handled as part of the UF deployment.

Changing the pass4SymmKey

So by now, we’ve ideally solved the deployment problems. We’ve installed Splunk 9.0.x UFs everwhere, we’ve pre-seeded them with a robust pass4SymmKey using the [deployment] stanza, and we’ve enabled authenticated access to our DS using restmap.conf. Of course, now that we’re done someone accidentally does a git commit with our pass4SymmKey in it and pushes it to a public GitHub repo. We’ll need to rotate the pass4SymmKey. This is a problem. If we first change the pass4SymmKey at the DS, that will cut all of the UFs off until we visit each one to give it the new value. If we start visiting UFs first, then we’ll cut them off one at a time until all have been visited. Neither of these are great.

We could turn off authentication until all of the UFs have been updated, and then re-enable it. That might work, but if we went to the effort of enabling authentication then this might not make sense. We added authentication for a reason, and just arbitrarily turning it off just to rotate a password seems less than ideal. Surely we didn’t enable authentication just to put a check mark in a compliance mandated security theater spreadsheet?

It turns out that Splunk anticipated this in restmap.conf, but you only see the hint of it from the spec file. In the authKeyStanza key, you can list multiple server.conf stanzas that contain pass4SymmKey values. So we can do this on the DS:

[root@33e28c67e142 local]# cat restmap.conf 

[broker:broker]
authKeyStanza=deployment, deployment2
requireAuthentication = true

[streams:deployment]
authKeyStanza=deployment, deployment2
requireAuthentication = true


[root@33e28c67e142 local]# /opt/splunk/bin/splunk btool server list deployment
[deployment]
pass4SymmKey = $7$TZxP0ci6l/e88I7IcRGJuhJR/N1eLgNuUlv3wQqbi29ueVps/9D8SSuSLOMTka8ZAkks
[deployment2]
pass4SymmKey = $7$+N40XRehCqEaLsWuaVQntroku8ekkskZ7IqG2xSkTt6Ejkal5/rdb5bsBswF0IIgXyPGyHgdKHv2cShtTkwb2X8xZg==

[root@33e28c67e142 local]# /opt/splunk/bin/splunk cmd splunkd show-decrypted --value '$7$TZxP0ci6l/e88I7IcRGJuhJR/N1eLgNuUlv3wQqbi29ueVps/9D8SSuSLOMTka8ZAkks'
myfirstpass4symmkey

[root@33e28c67e142 local]# /opt/splunk/bin/splunk cmd splunkd show-decrypted --value '$7$+N40XRehCqEaLsWuaVQntroku8ekkskZ7IqG2xSkTt6Ejkal5/rdb5bsBswF0IIgXyPGyHgdKHv2cShtTkwb2X8xZg=='
mysecondkeybecausethefirstwasleaked

Now, the DS supports either one of these pass4SymmKey values. We can put either of them on a UF and do a transition from one to the other. Check out uf8 and uf9 using different pass4SymmKeys:

[root@888d73cf5fde local]# /opt/splunkforwarder/bin/splunk btool server list deployment
[deployment]
pass4SymmKey = $7$FhVu9cYZfT0eAZX42cOYtfNJDIV5WIDUsi1dtczbFKhbQud5EiksI/+rMNO+o0pjfPIbQU76XkNgQI2tFAckG1tm4w==

[root@888d73cf5fde local]# /opt/splunkforwarder/bin/splunk cmd splunk show-decrypted --value '$7$FhVu9cYZfT0eAZX42cOYtfNJDIV5WIDUsi1dtczbFKhbQud5EiksI/+rMNO+o0pjfPIbQU76XkNgQI2tFAckG1tm4w=='
mysecondkeybecausethefirstwasleaked


[root@6f42c58a52ce testapp]# /opt/splunkforwarder/bin/splunk btool server list deployment
[deployment]
pass4SymmKey = $7$RLJg9Y2qsPjYkHZmXdiZ9GYHXBDPU2xfutAy97GTozXsCOvYpOtIz+Nr7JekYu4iTQSw

[root@6f42c58a52ce testapp]# /opt/splunkforwarder/bin/splunk cmd splunkd show-decrypted --value '$7$RLJg9Y2qsPjYkHZmXdiZ9GYHXBDPU2xfutAy97GTozXsCOvYpOtIz+Nr7JekYu4iTQSw'
myfirstpass4symmkey

And if we look on the DS, both are getting HTTP 200 responses in splunkd_access.log for their DS phonehomes:

10.89.0.5 - splunk-system-user [17/Nov/2022:03:08:56.344 +0000] "POST /services/broker/phonehome/connection_10.89.0.5_8089_uf8.dns.podman_6f42c58a52ce_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0 HTTP/1.1" 200 24 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms


10.89.0.6 - splunk-system-user [17/Nov/2022:03:09:24.132 +0000] "POST /services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5 HTTP/1.1" 200 471 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms

Wrap up

So we did a bit of a deep dive into how UFs (and other deployment clients) talk to DSes. We enabled DS authentication of the UFs and saw how code changes in Splunk 9.0.0 made it possible to enable pass4SymmKey authentication for app downloads from the DS. We even learned how to do pass4SymmKey secret rotation without shooting ourselves in the foot.

One important thing to (again) note. This is not mutual authentication. Having the DS authenticate clients using pass4SymmKey does not guarantee that clients are talking to a legitimate DS. Without correct TLS certificates and configurations, an attacker can still stand up a fake DS that does not require client pass4SymmKey authentication. In this scenario, clients will download and execute arbitrary apps from the attacker DS. Always perform a realistic assessment of your threat model and take the right steps to reduce risk in your environment.