Splunking bash history

The history tools built into the bash shell are rather powerful and a great source of information about what has been done to a system.  One thing we can do to make these even more useful is add them as a data source in Splunk.  While imperfect (see caveats below), this can be helpful in demonstrating that your systems are well-monitored for activity performed by users.

If you were going to do this, you might consider something as simple as:

[monitor:///home]
whitelist=\.bash_history$
disabled=false

This is a good starting point, but it suffers from some issues:

1. Recursively walking all of /home to find just a few small files can be expensive
2. Users outside of /home are not seen
3. The .bash_history file does not have timestamps, so you’ll get a _time equal to the index time

We can do better.  Bash gives us lots of options to change how its history works.  Let’s try this snippet in /usr/local/bin/bash-history.sh:

HISTBASEDIR=/var/log/bashhist

# are we an interactive shell?
if [ "$PS1" ] && [ -d $HISTBASEDIR ]; then

        REALNAME=`who am i | awk '{ print $1 }'`
        EFFNAME=`id -un`
        mkdir -m 700 $HISTBASEDIR/$EFFNAME >/dev/null 2>&1

        shopt -s histappend
        shopt -s lithist
        shopt -s cmdhist

        unset  HISTCONTROL && export HISTCONTROL
        unset  HISTIGNORE && export HISTIGNORE
        export HISTSIZE=10000
        export HISTTIMEFORMAT="%F %T "
        export HISTFILE=$HISTBASEDIR/$EFFNAME/history-$REALNAME

    case $TERM in
    xterm*)
            PROMPT_COMMAND='history -a && printf "\033]0;%s@%s:%s\007" "${USER}" "${HOSTNAME%%.*}" "${PWD/#$HOME/~}"'
                ;;
    screen)
            PROMPT_COMMAND='history -a && printf "\033]0;%s@%s:%s\033\\" "${USER}" "${HOSTNAME%%.*}" "${PWD/#$HOME/~}"'
        ;;
    *)
                        PROMPT_COMMAND='history -a'
        ;;
      esac
  # Turn on checkwinsize
  shopt -s checkwinsize
  PS1="[\u@\h \W]\\$ "
fi

This will configure bash to do several things:

1. Store history files in /var/log/bashhist/<effective-username>/history-<actual-username>
2. Add timestamps to history files
3. Persist the history file to disk more frequently

The difference here between “effective” and “actual” plays a part when using su or  sudo to run a shell as another user.  When you enable these settings for root, and then do a sudo su –, the history file for root while you are su’ed will be stored in /var/log/bashhist/root/history-duane — or whatever your username is.

To fully enable this, we need to do a few more things.  First, we need to make the /var/log/bashhist directory with (hopefully) appropriate permissions. The top-level directory needs to be 777 permissions in order to allow the shell snippet to make directories for new users as needed. Like with /tmp, the sticky-bit should prevent wild deletion by other users.

sudo mkdir -m 1777 /var/log/bashhist

We need to enable users-of-interest to source this snippet of shell into their .profile scripts. Add to the necessary .profile scripts:

source /usr/local/bin/bash-history.sh

Then we need to configure Splunk to read the history files with the right settings for timestamps. In props.conf:

[bash_history]
TIME_PREFIX = ^#
TIME_FORMAT = %s
EXTRACT-userids = ^/var/log/bashhist/(?<effective_user>[^/]+)/history-(?<real_user>.*)$ in source

And in inputs.conf:

[monitor:///var/log/bashhist]
disabled=false
sourcetype=bash_history

If you’ve done all of this right, when you run the history command in bash you should see dates and times next to them, and bash histories should be searchable in Splunk.

 

Caveats

I mentioned caveats.  There are a few – and I’m probably leaving some out.

Shell history is not an accurate source of audit-quality proof of things that were done / not done on a system.  Nothing keeps a user from editing their history file after the fact.  Splunk does pick up changes to the history file quickly and forwards them off-host.  This may be a mitigating factor toward users editing the file to try to hide history, but a clever person would be able to evade.  It is a good way to help keep honest people honest, but it’s not a strong control against an attacker or a dishonest person.

Also, shell history is written after a command finishes.  If a user run a long-running command, like an ssh to a remote host, it won’t show up in the history until the task exits.

So, while it’s imperfect, this technique might be useful to you in keeping track of things happening on your Unix systems.  Suggestions and improvements welcome.