Splunk .conf 2014 slides and notes

This week I had the pleasure of speaking at Splunk .conf 2014.  George Starcher and I spoke on configuring Splunk’s various SSL options, with the goal of providing a cookbook with SSL configurations appropriate for moving from a POC/trial install into production.  Other that some audio problems (sorry!), I thought the session went very well.  As a rookie presenter, I owe George a great deal of thanks for both convincing me to submit this talk and for helping me to prepare and present.  If this talk was a success at all, it was entirely due to George.  Thanks, George!

Attached are the slides:

Splunk-SSL-Presentation (PDF)

There were several fantastic questions raised during the talk that I’d like to answer here before I forget them.

Why is SSL client authentication of forwarders worthwhile?

It really all depends on your environment.  Because we recommend that all forwarders share a common certificate (the throwaway certificate), client identification of a forwarder really comes down to the rough question of “is this a box I generally trust or not?”  A great example of where this might be valuable is a public cloud deployment.  We gave an example of a colleague who had to accept data from AWS instances where they struggled to predict what IP addresses those systems were coming from.  So he had to open his SSL data input to essentially anywhere, and used a client certificate as a simple way to keep entirely unassociated systems from forwarding data to his indexers.  In a purely private environment, you will probably have better tools to control who is allowed to submit data to your indexers.

Can I use tools other than OpenSSL to manage keys and certificates?

Splunk will use OpenSSL internally as its SSL library, not much can be done about that.  But if you have other tools you would prefer to use in order to generate and manage certificates,  everything will work fine as long as you export the keys and certificates into the PEM-encoded x.509 files that OpenSSL expects.  Some tools (GSKit was specifically mentioned in the question) can do exports to PKCS12 (PFX) files, which you can then bust open using the ‘openssl pkcs12‘ command (mentioned in the bonus slides).

What is the performance impact of doing SSL authentication?

It should be (nearly) negligible, but it depends a little on whether we are talking about something where Splunk has already enabled SSL by default (like the Splunkd REST port) versus something that is completely cleartext by default like data forwarding.  In the cases where SSL is enabled by default, the additional CPU cost of performing certificate verification and common-name checking are not incredibly high.  For data forwarding, there is certainly some additional CPU load introduced by turning on SSL.  The actual impact would be somewhat dependent on your environment.

What about wildcard certs?  Can I use those to simplify SSL configuration for my large and complex deployment?

There’s no reason you could not.  However, we have not tested how Splunk’s common name checking works with wildcard certificates.  Typically, wildcards are used where an application (like a browser) does common name checking based on the DNS name of the site.  In Splunk’s case, most of the SSL processing (browsers-to-Splunkweb being an obvious exception) uses common names statically set in configuration files.  I’m simply not sure how the common-name checking code in Splunk handles wildcards.  In a large, complex environment it may be acceptable to use a model of “certificate per role” where you have a single cert for “indexers”, another for “deployment server”, and another still for “forwarder”.  You lose a little individuality / ability to identify servers uniquely, but you simplify your configuration greatly.

Why use ECC crypto?

The principal advantage of Elliptic Curve Cryptography is that the same level of effective security can be attained using a much smaller key.  According to an NSA brief, an ECC key of 256 bits provides security comparable to an RSA key length of 3072 bits and needs less computing power to perform its cryptographic operations.  As we discussed in the talk, the main downside to ECC is the high cost in commercial certificates.  Eventually there will be more competition in the ECC certificate business and costs will come down.  Until then, commercial ECC certificates are cost prohibitive for many businesses.

Can you DS-deploy an app, with the password encrypted inside?

Splunk’s password encryption for SSL key files is based on the splunk.secret file.  Normally this file is randomly generated the first time a Splunk instance is started.  If you can distribute splunk.secret to a Splunk system prior to its first time starting, then all password encryption will be done using this distributed file.  Now you will be able to DS-deploy an app containing encrypted passwords and have them encrypted in storage in all places.  You can see more about this in the Splunk docs.

If there are any other questions, please bring those up in the comments.