Splunk Cloud Gateway – Issue with removing devices and enabling services

By: Jerry Swadley

I had Splunk Cloud Gateway in my dev lab working well.  Life was grand!  I could show off the dashboards and alerts I was getting on my phone as a Proof of Concept.  I added the other Splunk admin’s phone.  He was seeing the same.

Then the other admin left the company.

As I am working on doing more with metrics, I want to upgrade from 7.3.3 to 8.0.2.1.  As I should, I upgraded my dev lab first.  After the upgrade I started fixing stuff related to orphaned searches due to my departed admin.  Then realized I should remove his phone from my dev lab as well.

NOTE:  My dev lab has 4 indexers, 1 server running LM/DMC, 1 for DS, 1 for DP, 1 for HF and 4 SH in a SHC.  All on 4 actual servers.  Topic for another day.

So I went to do that and that is when I found the issue.  When I clicked “Remove”, nothing happened.  I mean literally nothing.  I thought my browser was frozen.  I clicked cancel and the pop-up went away.  So it was not my browser, but something was not right with SCG.

A screenshot of a cell phone

Description automatically generated

I also have just bought an Apple TV (finally convinced my wife why she needed one.  Was that wrong?).  Anyway, I wanted to enable Splunk TV.

A screenshot of a cell phone

Description automatically generated

NOTE:  I cannot show you the “Oops, something went wrong.  Contact your Splunk admin” pop-up as my issue is resolved now.

But I did get it and it stressed me out!

I must admit that I put in a case with support.  After we finally got together, we were digging into $Splunk_Home/var/log/splunk/splunkd.log and we saw these error messages:

03-27-2020 10:10:38.141 -0500 ERROR UiAuth – Request from 10.103.186.166 to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”
03-27-2020 10:10:38.364 -0500 ERROR UiAuth – Request from 10.103.186.166 to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”

The actual issue is that I am very, very popular at my company.  I have about 50 tabs opened and had not cleared my cache or cookies since the upgrade in my dev lab.  The header being sent had an old key still in it.  Cleared all of that out and now it works!