Splunk Cloud Gateway – Issue with removing devices and enabling services

By: Jerry Swadley

I had Splunk Cloud Gateway in my dev lab working well.  Life was grand!  I could show off the dashboards and alerts I was getting on my phone as a Proof of Concept.  I added the other Splunk admin’s phone.  He was seeing the same.

Then the other admin left the company.

As I am working on doing more with metrics, I want to upgrade from 7.3.3 to  As I should, I upgraded my dev lab first.  After the upgrade I started fixing stuff related to orphaned searches due to my departed admin.  Then realized I should remove his phone from my dev lab as well.

NOTE:  My dev lab has 4 indexers, 1 server running LM/DMC, 1 for DS, 1 for DP, 1 for HF and 4 SH in a SHC.  All on 4 actual servers.  Topic for another day.

So I went to do that and that is when I found the issue.  When I clicked “Remove”, nothing happened.  I mean literally nothing.  I thought my browser was frozen.  I clicked cancel and the pop-up went away.  So it was not my browser, but something was not right with SCG.

A screenshot of a cell phone

Description automatically generated

I also have just bought an Apple TV (finally convinced my wife why she needed one.  Was that wrong?).  Anyway, I wanted to enable Splunk TV.

A screenshot of a cell phone

Description automatically generated

NOTE:  I cannot show you the “Oops, something went wrong.  Contact your Splunk admin” pop-up as my issue is resolved now.

But I did get it and it stressed me out!

I must admit that I put in a case with support.  After we finally got together, we were digging into $Splunk_Home/var/log/splunk/splunkd.log and we saw these error messages:

03-27-2020 10:10:38.141 -0500 ERROR UiAuth – Request from to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”
03-27-2020 10:10:38.364 -0500 ERROR UiAuth – Request from to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”

The actual issue is that I am very, very popular at my company.  I have about 50 tabs opened and had not cleared my cache or cookies since the upgrade in my dev lab.  The header being sent had an old key still in it.  Cleared all of that out and now it works!

Part 2 – Dashboards and Splunk Cloud Gateway

In the previous article (Dashboards and Splunk Cloud Gateway), I touched on the first opportunity to learn more of the new Splunk Cloud Gateway. Now I am going to talk about getting the Splunk TV app going.

One of the first things you need to do is enable Splunk TV in the Cloud Gateway app. In the Cloud Gateway app, go to configure and make sure to enable the Splunk TV (see below).

Next you will need to register the Apple TV to your Splunk Cloud Gateway. In the Cloud Gateway application, go to Register. When you first lauch the Splunk TV app on your Apple TV, it will give you an Activation Code. Enter that code in the Activation Code field of the Cloud Gateway register you device screen. Give the device a name. To continue setting up the device click Register.

You now will get the challenge code to verify. The code is at the bottom of the screen with your registration code. Then you will need to enter the credentials of your Splunk Enterprise account. Then click Continue to finish the setup.

You can verify that the system accepted you Splunk TV by going to Devices. Here you will be able to see all the devices connected via the Splunk Cloud Gateway. In this example I have a Splunk Mobile and a Splunk TV device connected.