Splunk Cloud Gateway – Issue with removing devices and enabling services

By: Jerry Swadley

I had Splunk Cloud Gateway in my dev lab working well.  Life was grand!  I could show off the dashboards and alerts I was getting on my phone as a Proof of Concept.  I added the other Splunk admin’s phone.  He was seeing the same.

Then the other admin left the company.

As I am working on doing more with metrics, I want to upgrade from 7.3.3 to  As I should, I upgraded my dev lab first.  After the upgrade I started fixing stuff related to orphaned searches due to my departed admin.  Then realized I should remove his phone from my dev lab as well.

NOTE:  My dev lab has 4 indexers, 1 server running LM/DMC, 1 for DS, 1 for DP, 1 for HF and 4 SH in a SHC.  All on 4 actual servers.  Topic for another day.

So I went to do that and that is when I found the issue.  When I clicked “Remove”, nothing happened.  I mean literally nothing.  I thought my browser was frozen.  I clicked cancel and the pop-up went away.  So it was not my browser, but something was not right with SCG.

A screenshot of a cell phone

Description automatically generated

I also have just bought an Apple TV (finally convinced my wife why she needed one.  Was that wrong?).  Anyway, I wanted to enable Splunk TV.

A screenshot of a cell phone

Description automatically generated

NOTE:  I cannot show you the “Oops, something went wrong.  Contact your Splunk admin” pop-up as my issue is resolved now.

But I did get it and it stressed me out!

I must admit that I put in a case with support.  After we finally got together, we were digging into $Splunk_Home/var/log/splunk/splunkd.log and we saw these error messages:

03-27-2020 10:10:38.141 -0500 ERROR UiAuth – Request from to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”
03-27-2020 10:10:38.364 -0500 ERROR UiAuth – Request from to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”

The actual issue is that I am very, very popular at my company.  I have about 50 tabs opened and had not cleared my cache or cookies since the upgrade in my dev lab.  The header being sent had an old key still in it.  Cleared all of that out and now it works!

Update: Dashboards and Splunk Cloud Gateway

In the first post about Splunk Cloud Gateway I talked about not being able to use Saved Searches with the Splunk Cloud Gateway. I worked with Splunk and got the official response back about the issue.

So you discovered a bug!  I will have a fix for this before the next release of the Cloud gateway app.  The issue is with the name of the saved search.  We are not url encoding the ref string when we should be which is causing the parser to fail.  If you want to work around this issue use a name for the saved search that doesn’t have any spaces.

After working with Splunk, I can confirm that this works. Using camel case (CamelCaseWorks) is a great way to accomplish this without spaces.

April 2017 Meeting Recap

User Group Meeting:

Thank you to everyone that came out to our meeting.  We had a great time learning about Splunk for AWS.  We also had a great discussion afterwards with many people sharing their experiences to help other members of the group.

The WebEx recording of Splunk for AWS is at:

PowerPoint (in PDF form) from the WebEx:
Splunk AWS Presentation

The marketing video for Splunk for AWS – Gain End-to-End AWS Visibility:

Survey Data:

From the data collect in the web survey, we will continue to have our meeting over lunch or after work.  During these meetings we will focus on technical solutions.  I will work to find ways for the meeting to be more interactive with more troubleshooting type of meetings.


  • Problems & Solutions; or a session to work through issues faced.
  • less sales, more tech details and app usage cases