Notes from November.2016 meeting

Thanks to everyone that came out to The Learning Academy for today’s meeting.  We had a great meeting with lots of discussion.

  • I brought up the idea of getting together at the beginning of December for a happy hour.  A couple people seemed interested.  I will send out an email to everyone in a few weeks to see who is interested.
  • Adam Tice with Splunk gave us an overview of what is new in Splunk 6.5
  • I showed building a rsyslog server and getting the logs in to Splunk
    • create sheet available below
  • The grouped talk about building your own dashboard by taking code from other application
    • Using other applications searches to build the dashboard of just want you want
    • Combining different dashboards in to a single dashboard
  • The group also talked to the students and each other about opportunities in security and specially in Splunk in the workforce


### Build your own rsyslog and send all the data to Splunk ###

In this test/demo environment I used three CentOS 7 machines on VMware Workstation.  This sheet is focusing on the rsyslog server machine and assumes that you have the client setup to forward syslog to the server and that the Splunk indexer is setup and set to receive Splunk data from a universal forwarder.


Splunk Indexer =
RSyslog Remote =
RSyslog Server =


## On the rsyslog server ##

# Make sure that we have the latest updates installed #
yum update -y

# Install the rsyslog and the documents for rsyslog
yum install rsyslog rsyslog-doc

# Edit the rsyslog config file
vi /etc/rsyslog.conf

# remove the comment (#) for the following lines:
$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

# Add the firewall rules to accept the data from other machines / devices
firewall-cmd –zone=public –permanent –add-port=514/tcp
firewall-cmd –zone=public –permanent –add-port=514/udp
firewall-cmd –reload

# Restart the rsyslog service
systemctl restart rsyslog

# Check to make sure that the system is responding to the ports
netstat -antup | grep 514

# You should see something like the following:



# Now watch the log file (/var/log/messages) to see if you are getting syslog data from the remote machine
tail -f /var/log/messages

## On the rsyslog remote ##

# Setup / Verify that it is set up to send syslog to the rsyslog server
vi /etc/rsyslog.conf

# Typically it will look like the following:
*.* @remote-host:514

# Then we can write a message to the message file to be sent
logger -t Splunk402 Splunk Rules!

## On the rsyslog server ##

# You should see a message pop up like:
Nov 16 12:17:33 remotemachine01 Splunk402: Splunk Rules!

# Now we want to install the Splunk Universal Forwarder.  Go to and select .rpm version that is the same as the OS for you.  If you are not logged in to the site, you will need to login or sign up for an account.  Click on the “Download via Command Line (wget)” and get the wget code.
wget -O splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm ‘’

# Install Splunk with YUM
yum –nogpgcheck localinstall splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm

# Start Splunk accepting the license
/opt/splunkforwarder/bin/splunk start –accept-license

# Enable Splunk to start at boot
/opt/splunkforwarder/bin/splunk enable boot-start

# If you are not using a Deployment Server to set the output.conf file, edit the file to send the data to Splunk
vi /opt/splunkforwarder/etc/system/local/outputs.conf



# Have Splunk grab the data from the /var/log folder
vi /opt/splunkforwarder/etc/system/local/inputs.conf


# Depending on your needs, you may need to add which index to send to or other factors.  Please consult the docs for inputs.conf.

# Now we need to restart Splunk to have all of our settings go in to effect.
/opt/splunkforwarder/bin/splunk restart

## You should now see the logs in Splunk.  Happy Splunking! ##

Encrypt Splunk Web with Let’s Encrypt

Everyone wants to encrypt everything, and we should.  But paying for certificates can get a bit expensive.  With is where Let’s Encrypt comes in to play.  Let’s Encrypt allows you to create a SSL certificate for FREE!  You read that correct.   Free as in beer.  They event have an automated script to help configure your server.  Link below for the Splunk Blog with a how-to set this up.

Using Splunk to Spot the Adversary

I recently read the Splunk blog Spotting the Adversary… with Splunk.  If you have a Windows based infrastructure, this is a great article to read as well as the referenced material.  In the article, it talks about having to go through millions of events to even try to find the adversary and the struggles that it causes.  The blog post is great because it even gives you a sample dashboard you can drop in to your environment, assuming you have the GPOs setup and that you have the Windows Infrastructure app installed.  Lucky for me as I did have those setup.

The blog post is based off of a document from the NSA’s IA team.  It is a great read.  At 54 pages, it isn’t the usual long drawn out government paper.  They even have screen shots of the options you need to make sure to change.

They also reference a talk from .conf2015.  In the blog post, they link the slide deck.  There is also recording of the slide deck with audio of the talk(

I found the dashboard was overall very good.  I did notice a few things not lining up correctly.   For example, EventCode 43 is listed under USB Activity and I am seeing the logs for Definition Update for Microsoft Endpoint Protection.  But the dashboard did break things down to where I was able to go through different menus and find things I might not have known without going through and digging in to the logs.

spotting adversary table

Splunk Incident Review Demo

Watch this demonstration of the Splunk Enterprise Security’s incident review framework to learn how you can detect, analyze and respond to security incidents and threats. The demo highlights benefits of using Asset Investigator, Risk and other key concepts in the context of investigating and managing overall Incident Response in SIEM workflows.

Splunk Threat Intelligence Demo

Watch this demonstration of the Splunk Enterprise Threat Intelligence capability and gain the ability to quickly detect and investigate threats using Threat Intelligence. Splunk Enterprise Security’s Threat Intelligence framework helps organizations to aggregate, prioritize and manage wide varieties of threat intelligence from threat lists such as STIX/TAXII, Open Source and many more.

Splunk Enterprise Security Demo

Watch this demonstration of Splunk Enterprise Security to learn the key capabilities of Splunk Enterprise Security and how to use it to solve key security challenges. Splunk Enterprise Security helps security practitioners detect, investigate and respond to internal and external attacks by simplifying threat management while minimizing risk to safeguard your business.

SPLing Bee – Update

Hello Everyone,

After posting about the SPLing Bee back in November, I have got around to trying it out myself.  After working out a few bugs, shout out to Charlie Huggard for the help, I have the SPLing Bee ready for download.

SPLing Bee Server App

In the app, there is the dashboards, inputs, indexes, props, and macros configuration files.


connection_host = ip
index = spl_bee
sourcetype = spl_bee_json
source = spl_bee_input


coldPath = $SPLUNK_DB/spl_bee/colddb
homePath = $SPLUNK_DB/spl_bee/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/spl_bee/thaweddb


KV_MODE = none
TIMESTAMP_FIELDS = LatestSubmissionTime
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
category = Structured
disabled = false
pulldown_type = true


definition = inputlookup round1.csv | append [| inputlookup round2.csv] | append [| inputlookup round3.csv] | append [| inputlookup round4.csv] | append [| inputlookup round5.csv] | append [| inputlookup round6.csv] | append [| inputlookup round7.csv] | append [| inputlookup round8.csv] | append [| inputlookup round9.csv]
iseval = 0

If the Splunk indexer you are using to play/judge is publicly available, you can have the contestants use Splunk Cloud trial to spin up machines to play.  If your indexer is on a private network, you will need to spin up a Splunk indexer/heavy forwarder to be able to play.  The contestant will need a full instance of Splunk and not a universal forwarder as they will need to index data, run searches against the data, and add a Splunk application to the indexer.

Things to do when setting up the game.

  1. Make sure that the indexer isn’t setup to use port 9997.  The SPLing Bee app will setup a TCP input to listen to port 9997.
  2. Update the text Round 1 of the dashboard with the correct instructions for the contest and where to send the data.
  3. Make sure there are machines available for the contestants or have the main indexer publicly facing so they can use a Splunk Cloud trial.

For running the contest, Splunk wrote the SPLing_Bee_Directions for .conf2015.

SPLing Bee

If you were not able to attend the Splunk’s conference, you missed out on a great game.  The SPLing Bee is a game that helped you to develop your searching skills.

Here are the SPLing_Bee_Directions that were wrote for .conf2015.

Here is the SPLing_Bee_Application.  You’ll also need the sendjobmeta custom search command, this is used for have your contestant instances send their search results / answers to a master instance of your choice –

It’s best to have everyone use cloud trial instances and install the app + index the sample dataset you want to use. The IP address will be different using the cloud instances whereas if people are using their local instances you may see the same IP come in when trying to judge.

October 2015 Meeting

Here is the PowerPoint from the October 2015 meeting:

Thank you to the Nonprofit Hub in Lincoln for allowing us to take over the area. Thank you to Tony Reinke from National Research Corporation, John Foss of Splunk, and Bret Brasfield of Continuum for your presentations. There was a lot of good discussion on Enterprise Security and we were very happy that Kanad Sharma from Splunk Professional Services was there and able to share his experiences.

Take the survey on this meeting and the over all group.

March 2015 Meeting Recap

Thank You!

Thank you to everyone that joined us for our developer focused meeting.  Thank you to Nimble Storage for providing the food.  Thank you to Continuum for providing the drinks.  And thank you to National Research for hosting the event.  It was great to learn more about writing code for Splunk.  Below you will find the PDF of both Tony Reinke of National Research and Brian Knopp of Nimble Storage’s presentations.

Splunk402 March 2015 Meeting


Please take a moment and answer a quick 5 question survey.  Let us know if you could or could not make the event, rate our speakers, and help us to decide the topics of upcoming meetings.

 Splunk .conf2015

Recently a call for papers has gone out for the 2015 Splunk Conference.  Got a great success story?  Made a killer app?  They will accept submissions via the Call for Papers until April 28th, and begin announcing accepted sessions on May 28, 2015.

Submit here:

Meeting Video

If you missed the meeting, you can watch the meeting below.  We noticed the audio cut out in a few places in the recordings.  We will try to catch this quicker in future meetings.