New User Group – Sioux Falls, SD

Are you in the Sioux Falls, South Dakota area? Are you a Splunk customer or want to learn more Splunk? We are starting up a Sioux Falls user group. No matter if you are just getting in to IT or been there for years, come and join us. This is a technical user group so you won’t have to sit and listen to sales people pitch at you. You will get to talk to other IT ninjas and gurus that love to tinker and play.

https://usergroups.splunk.com/group/sioux-falls-south-dakota.html

April 2017 Meeting Recap

User Group Meeting:

Thank you to everyone that came out to our meeting.  We had a great time learning about Splunk for AWS.  We also had a great discussion afterwards with many people sharing their experiences to help other members of the group.

The WebEx recording of Splunk for AWS is at:
https://splunk.webex.com/splunk/ldr.php?RCID=3f4e82ee9cbe110328218495db3e3d29

PowerPoint (in PDF form) from the WebEx:
Splunk AWS Presentation

The marketing video for Splunk for AWS – Gain End-to-End AWS Visibility:
https://www.splunk.com/en_us/resources/video.5jYXR4MjE6hL2mlcK7r-TlN3IFkgSQ5Z.html


Survey Data:

From the data collect in the web survey, we will continue to have our meeting over lunch or after work.  During these meetings we will focus on technical solutions.  I will work to find ways for the meeting to be more interactive with more troubleshooting type of meetings.

Comments:

  • Problems & Solutions; or a session to work through issues faced.
  • less sales, more tech details and app usage cases

April.2017 Meeting

The upcoming meeting is going to be light and fluffy. We will be talking about Clouds. To define that more, Amazon Web Services. Splunk and AWS have a great relationship. Many of us use AWS personally and professionally. Time to get that AWS data in to Splunk. We will go over setting up the App and then you will get a chance to have a Q&A with Laura.

We will be meeting at the AIM Exchange building (1905 Harney St, Omaha, NE 68102) over lunch. Hope to see everyone there.

Register for the event at:
https://goo.gl/jS1kYo

Notes from November.2016 meeting

Thanks to everyone that came out to The Learning Academy for today’s meeting.  We had a great meeting with lots of discussion.

  • I brought up the idea of getting together at the beginning of December for a happy hour.  A couple people seemed interested.  I will send out an email to everyone in a few weeks to see who is interested.
  • Adam Tice with Splunk gave us an overview of what is new in Splunk 6.5
  • I showed building a rsyslog server and getting the logs in to Splunk
    • create sheet available below
  • The grouped talk about building your own dashboard by taking code from other application
    • Using other applications searches to build the dashboard of just want you want
    • Combining different dashboards in to a single dashboard
  • The group also talked to the students and each other about opportunities in security and specially in Splunk in the workforce

 

### Build your own rsyslog and send all the data to Splunk ###

In this test/demo environment I used three CentOS 7 machines on VMware Workstation.  This sheet is focusing on the rsyslog server machine and assumes that you have the client setup to forward syslog to the server and that the Splunk indexer is setup and set to receive Splunk data from a universal forwarder.

—————

Splunk Indexer = 192.168.233.135
RSyslog Remote = 192.168.233.136
RSyslog Server = 192.168.233.137

—————

## On the rsyslog server ##

# Make sure that we have the latest updates installed #
yum update -y

# Install the rsyslog and the documents for rsyslog
yum install rsyslog rsyslog-doc

# Edit the rsyslog config file
vi /etc/rsyslog.conf

# remove the comment (#) for the following lines:
$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

# Add the firewall rules to accept the data from other machines / devices
firewall-cmd –zone=public –permanent –add-port=514/tcp
firewall-cmd –zone=public –permanent –add-port=514/udp
firewall-cmd –reload

# Restart the rsyslog service
systemctl restart rsyslog

# Check to make sure that the system is responding to the ports
netstat -antup | grep 514

# You should see something like the following:
capture

 

 

# Now watch the log file (/var/log/messages) to see if you are getting syslog data from the remote machine
tail -f /var/log/messages

## On the rsyslog remote ##

# Setup / Verify that it is set up to send syslog to the rsyslog server
vi /etc/rsyslog.conf

# Typically it will look like the following:
*.* @remote-host:514

# Then we can write a message to the message file to be sent
logger -t Splunk402 Splunk Rules!

## On the rsyslog server ##

# You should see a message pop up like:
Nov 16 12:17:33 remotemachine01 Splunk402: Splunk Rules!

# Now we want to install the Splunk Universal Forwarder.  Go to https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/linux and select .rpm version that is the same as the OS for you.  If you are not logged in to the site, you will need to login or sign up for an account.  Click on the “Download via Command Line (wget)” and get the wget code.
wget -O splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.5.0&product=universalforwarder&filename=splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm&wget=true’

# Install Splunk with YUM
yum –nogpgcheck localinstall splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm

# Start Splunk accepting the license
/opt/splunkforwarder/bin/splunk start –accept-license

# Enable Splunk to start at boot
/opt/splunkforwarder/bin/splunk enable boot-start

# If you are not using a Deployment Server to set the output.conf file, edit the file to send the data to Splunk
vi /opt/splunkforwarder/etc/system/local/outputs.conf

[tcpout]
defaultGroup=syslogdemo

[tcpout:syslogdemo]
server=192.168.233.135:9997

# Have Splunk grab the data from the /var/log folder
vi /opt/splunkforwarder/etc/system/local/inputs.conf

[monitor:///var/log]

# Depending on your needs, you may need to add which index to send to or other factors.  Please consult the docs for inputs.conf.

# Now we need to restart Splunk to have all of our settings go in to effect.
/opt/splunkforwarder/bin/splunk restart

## You should now see the logs in Splunk.  Happy Splunking! ##

Encrypt Splunk Web with Let’s Encrypt

Everyone wants to encrypt everything, and we should.  But paying for certificates can get a bit expensive.  With is where Let’s Encrypt comes in to play.  Let’s Encrypt allows you to create a SSL certificate for FREE!  You read that correct.   Free as in beer.  They event have an automated script to help configure your server.  Link below for the Splunk Blog with a how-to set this up.

http://blogs.splunk.com/2016/08/12/secure-splunk-web-in-five-minutes-using-lets-encrypt/?linkId=27654898

Using Splunk to Spot the Adversary

I recently read the Splunk blog Spotting the Adversary… with Splunk.  If you have a Windows based infrastructure, this is a great article to read as well as the referenced material.  In the article, it talks about having to go through millions of events to even try to find the adversary and the struggles that it causes.  The blog post is great because it even gives you a sample dashboard you can drop in to your environment, assuming you have the GPOs setup and that you have the Windows Infrastructure app installed.  Lucky for me as I did have those setup.

The blog post is based off of a document from the NSA’s IA team.  It is a great read.  At 54 pages, it isn’t the usual long drawn out government paper.  They even have screen shots of the options you need to make sure to change.

They also reference a talk from .conf2015.  In the blog post, they link the slide deck.  There is also recording of the slide deck with audio of the talk(https://conf.splunk.com/session/2015/recordings/2015-splunk-86v2.mp4).

I found the dashboard was overall very good.  I did notice a few things not lining up correctly.   For example, EventCode 43 is listed under USB Activity and I am seeing the logs for Definition Update for Microsoft Endpoint Protection.  But the dashboard did break things down to where I was able to go through different menus and find things I might not have known without going through and digging in to the logs.

spotting adversary table

Splunk Incident Review Demo

Watch this demonstration of the Splunk Enterprise Security’s incident review framework to learn how you can detect, analyze and respond to security incidents and threats. The demo highlights benefits of using Asset Investigator, Risk and other key concepts in the context of investigating and managing overall Incident Response in SIEM workflows.

Splunk Threat Intelligence Demo

Watch this demonstration of the Splunk Enterprise Threat Intelligence capability and gain the ability to quickly detect and investigate threats using Threat Intelligence. Splunk Enterprise Security’s Threat Intelligence framework helps organizations to aggregate, prioritize and manage wide varieties of threat intelligence from threat lists such as STIX/TAXII, Open Source and many more.

Splunk Enterprise Security Demo

Watch this demonstration of Splunk Enterprise Security to learn the key capabilities of Splunk Enterprise Security and how to use it to solve key security challenges. Splunk Enterprise Security helps security practitioners detect, investigate and respond to internal and external attacks by simplifying threat management while minimizing risk to safeguard your business.

SPLing Bee – Update

Hello Everyone,

After posting about the SPLing Bee back in November, I have got around to trying it out myself.  After working out a few bugs, shout out to Charlie Huggard for the help, I have the SPLing Bee ready for download.

SPLing Bee Server App

In the app, there is the dashboards, inputs, indexes, props, and macros configuration files.

inputs.conf

[tcp://9997]
connection_host = ip
index = spl_bee
sourcetype = spl_bee_json
source = spl_bee_input

indexes.conf

[spl_bee]
coldPath = $SPLUNK_DB/spl_bee/colddb
homePath = $SPLUNK_DB/spl_bee/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/spl_bee/thaweddb

props.conf

[spl_bee_results_csv]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = LatestSubmissionTime
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
DATETIME_CONFIG =
[spl_bee_json]
KV_MODE=json
NO_BINARY_CHECK = true
category = Structured
disabled = false
pulldown_type = true

macros.conf

[retrieve_results]
definition = inputlookup round1.csv | append [| inputlookup round2.csv] | append [| inputlookup round3.csv] | append [| inputlookup round4.csv] | append [| inputlookup round5.csv] | append [| inputlookup round6.csv] | append [| inputlookup round7.csv] | append [| inputlookup round8.csv] | append [| inputlookup round9.csv]
iseval = 0

If the Splunk indexer you are using to play/judge is publicly available, you can have the contestants use Splunk Cloud trial to spin up machines to play.  If your indexer is on a private network, you will need to spin up a Splunk indexer/heavy forwarder to be able to play.  The contestant will need a full instance of Splunk and not a universal forwarder as they will need to index data, run searches against the data, and add a Splunk application to the indexer.

Things to do when setting up the game.

  1. Make sure that the indexer isn’t setup to use port 9997.  The SPLing Bee app will setup a TCP input to listen to port 9997.
  2. Update the text Round 1 of the dashboard with the correct instructions for the contest and where to send the data.
  3. Make sure there are machines available for the contestants or have the main indexer publicly facing so they can use a Splunk Cloud trial.

For running the contest, Splunk wrote the SPLing_Bee_Directions for .conf2015.