Splunk pass4SymmKey for deployment client -> deployment server

Duane

Introduction

So you want to secure your Splunk deployment server?  There’s a couple of different angles to consider:

  1. Are all clients connecting to a given deployment server permitted to do so?
  2. Is the client certain that the deployment server they are talking to is the real one and not an impostor?

Let’s start at the docs, in the Securing Splunk Enterprise manual.  Here’s a screen cap from that page taken today (late July 2018):

I’ve highlighted the “Deployment server to deployment clients” part. That’s where we will focus our efforts today. The advice here is to use pass4SymmKey in order to secure Deployment Client to Deployment Server. As a reminder, pass4SymmKey is a symmetric secret shared between two Splunk nodes to authenticate system-to-system REST API usage.  The pass4SymmKey comes up frequently in conversations about License Masters, Cluster Masters, and Search Head Clustering.  But, in my experience, the use of pass4SymmKey related to Deployment Server is rare.  Let’s look into this a little deeper.

The Splunk server.conf file- has a pass4SymmKey option that can be set in a few different stanzas, so you can use a different value for different modes of communication.  At a minimum, there is one in the [general] stanza and one in the [clustering] stanza.  I’m going to steal a quote from the $SPLUNK_HOME/etc/system/README/server.conf.spec file.

pass4SymmKey = <password>
* Authenticates traffic between:
  * License master and its license slaves.
  * Members of a cluster; see Note 1 below.
  * Deployment server (DS) and its deployment clients (DCs); see Note 2
    below.
* Note 1: Clustering may override the passphrase specified here, in
  the [clustering] stanza.  A clustering searchhead connecting to multiple
  masters may further override in the [clustermaster:stanza1] stanza.
* Note 2: By default, DS-DCs passphrase auth is disabled.  To enable DS-DCs
  passphrase auth, you must *also* add the following line to the
  [broker:broker] stanza in restmap.conf:
     requireAuthentication = true
* In all scenarios, *every* node involved must set the same passphrase in
  the same stanza(s) (i.e. [general] and/or [clustering]); otherwise,
  respective communication (licensing and deployment in case of [general]
  stanza, clustering in case of [clustering] stanza) will not proceed.
* Unencrypted passwords must not begin with "$1$", as this is used by
  Splunk software to determine if the password is already encrypted.

There’s two quick caveats right out of the box.

First, the Deployment Client -> Deployment Server pass4SymmKey and the License Slave -> License Master pass4SymmKey are set using the same setting.  So, if you have a Deployment Server that is also a License Slave, then you’ll have to use the same pass4SymmKey all around.  This is probably not ideal.  Splunk Support will usually give you a “Deployment Server License” file.  It’s a 0 byte license that enables the Enterprise features.  This way,  your DS isn’t depending on your License Master and you can use different pass4SymmKeys for your DS/DC comms and your LM/LS comms.

Second, you have to explicitly enable the use of pass4SymmKey for Deployment Server.  That is what we will do today, and we will review the security properties achieved by this.

Enabling pass4SymmKey authentication at the DS

On our DS, we will create a $SPLUNK_HOME/etc/system/local/restmap.conf file with the stanza shown above, specifically:

[root@ds local]# cat <<EOF >> /opt/splunk/etc/system/local/restmap.conf
> [broker:broker]
> requireAuthentication = true
> EOF

Now, let’s use btool to make certain that my settings are as I expect them to be:

[root@6e6965b19f06 local]# /opt/splunk/bin/splunk btool --debug restmap list broker:broker
/opt/splunk/etc/system/local/restmap.conf [broker:broker]
/opt/splunk/etc/system/default/restmap.conf authKeyStanza = deployment
/opt/splunk/etc/system/default/restmap.conf match = /broker
/opt/splunk/etc/system/local/restmap.conf requireAuthentication = true

The requireAuthentication we added is there, but there’s something I didn’t expect.  See the authKeyStanza setting?  If we go digging a bit more in the spec file for server.conf.spec, we’ll find a mention of this:

[deployment]
pass4SymmKey = 
* Authenticates traffic between Deployment server (DS) and its deployment
clients (DCs).
* By default, DS-DCs passphrase auth is disabled. To enable DS-DCs
passphrase auth, you must *also* add the following line to the
[broker:broker] stanza in restmap.conf:
requireAuthentication = true
* If it is not set in the deployment stanza, the key will be looked in
the general stanza
* Unencrypted passwords must not begin with "$1$", as this is used by
Splunk software to determine if the password is already encrypted.

From the docs, this appears to have been added (to the spec files at least) around version 6.2.4, see SPL-99169.  But, it’s not been (to my knowledge) very well advertised.  Maybe this post and the docs feedback I’m submitting will help publicize it a little more.  But, given this new knowledge, we should be able to use a different pass4SymmKey for DC/DS as we do for license master connectivity — assuming we have a “reasonably modern” Splunk on both ends.  So let’s set up our DS to expect a pass4SymmKey, and our UF to use the same one to talk to it.

[root@ds local]# cat <<EOF >> /opt/splunk/etc/system/local/server.conf
> [deployment]
> pass4SymmKey = myReallyAwesomeSecret123
> EOF

Update 2020-06-30

Discussing this feature with Martin Müller, there are some limitations here.  The [broker:broker] authentication for pass4SymmKey only protects the DS “control channel” API endpoint.  There are other API endpoints, like the app download endpoint, that will not require pass4SymmKey.  What this means is that an attacker who knows the names of your serverclasses and apps will still be able to download those apps from your DS without authentication.  See Splunk Ideas https://ideas.splunk.com/ideas/EID-I-391 which discusses this.

Enabling pass4SymmKey at the UF

[root@uf local]# cat <<EOF >> /opt/splunkforwarder/etc/system/local/server.conf
> [deployment]
> pass4SymmKey = myReallyAwesomeSecret123
> EOF
[root@uf local]# cat <<EOF >> /opt/splunkforwarder/etc/system/local/deploymentclient.conf
> [target-broker:deploymentServer]
> targetUri = 172.17.0.2:8089 
> EOF

We then restart both, and can see the UF successfully calling home to the Deployment Server.

09-03-2018 23:34:20.509 +0000 INFO HttpPubSubConnection - SSL connection with id: connection_172.17.0.3_8089_172.17.0.3_ec9863789248_64A3C887-9546-45E5-B2DD-12A3395C07CE
09-03-2018 23:34:20.514 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_172.17.0.3_8089_172.17.0.3_ec9863789248_64A3C887-9546-45E5-B2DD-12A3395C07CE
09-03-2018 23:34:21.376 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_172.17.0.3_8089_172.17.0.3_ec9863789248_64A3C887-9546-45E5-B2DD-12A3395C07CE
09-03-2018 23:34:21.378 +0000 INFO DC:HandshakeReplyHandler - Handshake done.

Wrong Password

Now, let’s change the pass4SymmKey at the UF to see what happens to a UF that has the wrong pass4SymmKey.  So I open my server.conf in my editor on the UF, change the pass4SymmKey line under [deployment] to:

pass4SymmKey = anotherAwesomeSecret456

followed by another quick restart.  We see pretty quickly looking at the UF’s own logs that the client is continuously erroring out on its attempt to phone home, but the only evidence of a problem at the Deployment Server is HTTP status 401’s in splunkd_access.log, like so:

172.17.0.3 - - [04/Sep/2018:00:10:07.860 +0000] "POST /services/broker/connect/64A3C887-9546-45E5-B2DD-12A3395C07CE/ec9863789248/a0c72a66db66/linux-x86_64/8089/7.1.2/64A3C887-9546-45E5-B2DD-12A3395C07CE/universal_forwarder/ec9863789248 HTTP/1.1" 401 148 - - - 0ms
172.17.0.3 - - [04/Sep/2018:00:11:03.778 +0000] "POST /services/broker/connect/64A3C887-9546-45E5-B2DD-12A3395C07CE/ec9863789248/a0c72a66db66/linux-x86_64/8089/7.1.2/64A3C887-9546-45E5-B2DD-12A3395C07CE/universal_forwarder/ec9863789248 HTTP/1.1" 401 148 - - - 0ms
172.17.0.3 - - [04/Sep/2018:00:11:37.783 +0000] "POST /services/broker/connect/64A3C887-9546-45E5-B2DD-12A3395C07CE/ec9863789248/a0c72a66db66/linux-x86_64/8089/7.1.2/64A3C887-9546-45E5-B2DD-12A3395C07CE/universal_forwarder/ec9863789248 HTTP/1.1" 401 148 - - - 0ms
172.17.0.3 - - [04/Sep/2018:00:12:45.953 +0000] "POST /services/broker/connect/64A3C887-9546-45E5-B2DD-12A3395C07CE/ec9863789248/a0c72a66db66/linux-x86_64/8089/7.1.2/64A3C887-9546-45E5-B2DD-12A3395C07CE/universal_forwarder/ec9863789248 HTTP/1.1" 401 148 - - - 0ms
172.17.0.3 - - [04/Sep/2018:00:13:59.000 +0000] "POST /services/broker/connect/64A3C887-9546-45E5-B2DD-12A3395C07CE/ec9863789248/a0c72a66db66/linux-x86_64/8089/7.1.2/64A3C887-9546-45E5-B2DD-12A3395C07CE/universal_forwarder/ec9863789248 HTTP/1.1" 401 148 - - - 0ms

Can a fake DS still work?

So we’ve been relatively successful at satisfying our first requirement – we can ensure that only authorized UFs are allowed to connect to our DS and download apps from it.  The key management does not work great at scale, as everyone must use the same pass4SymmKey.  But, the basic requirement is met.  What about our second requirement?  Can we be certain, though, that the client is talking to an authorized DS?  Let’s make our DS no longer require client authentication by disabling restmap.conf:

[root@ds splunk]# cd /opt/splunk/etc/system/local/
[root@ds local]# mv restmap.conf restmap.conf.save
[root@ds local]# vi server.conf
   #  Remove the [deployment] stanza entirely from server.conf
[root@ds local]# /opt/splunk/bin/splunk restart

After disabling the DS-side authentication requirement, the UF is able to connect to the DS immediately — even though it is sending a pass4SymmKey that does not match the DS.  Because the DS is no longer demanding a pass4SymmKey, what the client sends does not matter.  So, this proves more-or-less that pass4SymmKey is helpful for one of our requirements, but not the other.

Conclusion

We’ve shown that pass4SymmKey does an adequate job of only allowing authenticated UFs to connect to a DS, but it does not provide strong enough authentication properties to prevent a client from connecting to a fake DS.  An attacker could, through several different techniques, trick your UFs into connecting to an attacker-controlled Deployment Server.  The Deployment Server can cause execution of arbitrary code on any Deployment Client connected to it, as the user running Splunk on the Deployment Client.  Your Deployment Server should be as robustly protected and monitored as your Puppet, Chef, SCCM, or other configuration management services.

If you want to protect your clients from connecting to a fake DS, then you will have to take the route of correctly configuring TLS on the Deployment Server to use a genuine CA-issued certificate (internal or external CA), and enable the deployment clients with the proper root chain, as well as enabling sslVerifyServerCert and commonNameToCheck.

Based on this testing, I’m going to go back to the excellent Splunk Docs team and suggest some edits to this section of the docs to make clear the difference in these two properties and how to best achieve each.  Look for updates here and there.

 

Leave a Reply

Your email address will not be published. Required fields are marked *