Splunk Cloud Gateway – Issue with removing devices and enabling services

By: Jerry Swadley

I had Splunk Cloud Gateway in my dev lab working well.  Life was grand!  I could show off the dashboards and alerts I was getting on my phone as a Proof of Concept.  I added the other Splunk admin’s phone.  He was seeing the same.

Then the other admin left the company.

As I am working on doing more with metrics, I want to upgrade from 7.3.3 to 8.0.2.1.  As I should, I upgraded my dev lab first.  After the upgrade I started fixing stuff related to orphaned searches due to my departed admin.  Then realized I should remove his phone from my dev lab as well.

NOTE:  My dev lab has 4 indexers, 1 server running LM/DMC, 1 for DS, 1 for DP, 1 for HF and 4 SH in a SHC.  All on 4 actual servers.  Topic for another day.

So I went to do that and that is when I found the issue.  When I clicked “Remove”, nothing happened.  I mean literally nothing.  I thought my browser was frozen.  I clicked cancel and the pop-up went away.  So it was not my browser, but something was not right with SCG.

A screenshot of a cell phone

Description automatically generated

I also have just bought an Apple TV (finally convinced my wife why she needed one.  Was that wrong?).  Anyway, I wanted to enable Splunk TV.

A screenshot of a cell phone

Description automatically generated

NOTE:  I cannot show you the “Oops, something went wrong.  Contact your Splunk admin” pop-up as my issue is resolved now.

But I did get it and it stressed me out!

I must admit that I put in a case with support.  After we finally got together, we were digging into $Splunk_Home/var/log/splunk/splunkd.log and we saw these error messages:

03-27-2020 10:10:38.141 -0500 ERROR UiAuth – Request from 10.103.186.166 to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”
03-27-2020 10:10:38.364 -0500 ERROR UiAuth – Request from 10.103.186.166 to “/en-US/splunkd/__raw/services/kvstore/delete_device?device_owner=admin&device_key=mdFZf2g1B72YyDwE1cbgxGnbQ4VxWofxi6juZJ6qOVY%3D” failed CSRF validation — expected key “[REDACTED]2307” and header had key “10977396672569430113”

The actual issue is that I am very, very popular at my company.  I have about 50 tabs opened and had not cleared my cache or cookies since the upgrade in my dev lab.  The header being sent had an old key still in it.  Cleared all of that out and now it works!

Update: Dashboards and Splunk Cloud Gateway

In the first post about Splunk Cloud Gateway I talked about not being able to use Saved Searches with the Splunk Cloud Gateway. I worked with Splunk and got the official response back about the issue.

So you discovered a bug!  I will have a fix for this before the next release of the Cloud gateway app.  The issue is with the name of the saved search.  We are not url encoding the ref string when we should be which is causing the parser to fail.  If you want to work around this issue use a name for the saved search that doesn’t have any spaces.

After working with Splunk, I can confirm that this works. Using camel case (CamelCaseWorks) is a great way to accomplish this without spaces.

Part 2 – Dashboards and Splunk Cloud Gateway

In the previous article (Dashboards and Splunk Cloud Gateway), I touched on the first opportunity to learn more of the new Splunk Cloud Gateway. Now I am going to talk about getting the Splunk TV app going.

One of the first things you need to do is enable Splunk TV in the Cloud Gateway app. In the Cloud Gateway app, go to configure and make sure to enable the Splunk TV (see below).

Next you will need to register the Apple TV to your Splunk Cloud Gateway. In the Cloud Gateway application, go to Register. When you first lauch the Splunk TV app on your Apple TV, it will give you an Activation Code. Enter that code in the Activation Code field of the Cloud Gateway register you device screen. Give the device a name. To continue setting up the device click Register.

You now will get the challenge code to verify. The code is at the bottom of the screen with your registration code. Then you will need to enter the credentials of your Splunk Enterprise account. Then click Continue to finish the setup.

You can verify that the system accepted you Splunk TV by going to Devices. Here you will be able to see all the devices connected via the Splunk Cloud Gateway. In this example I have a Splunk Mobile and a Splunk TV device connected.

Dashboards and Splunk Cloud Gateway

So you have downloaded Splunk Cloud Gateway (https://splunkbase.splunk.com/app/4250/) and you are ready to go on your Mobile Device or Apple TV, but when you check your app your dashboard(s) do not show up in the list. Oh no! Here is a couple things to check and I am sure this article will be updated as more information is learned.

Check the internals!

A great place to start is the internal logs of Splunk. In this case:

index=_internal source=”/opt/splunk/var/log/splunk/splunk_app_cloudgateway*”

This will let you see what is going on inside the Cloud Gateway app. Now we can start to search what what is going on in the dashboard by adding the name of your dashboard in to the search.

index=_internal source=”/opt/splunk/var/log/splunk/splunk_app_cloudgateway*” “awesome_dashboard”

There can be a lot of see so we might want to narrow it down to just the errors and warnings.

index=_internal source=”/opt/splunk/var/log/splunk/splunk_app_cloudgateway*” (log_level=WARNING OR log_level=ERROR) “awesome_dashboard”

Now we can start to get some where.

My error happened to be:

WARNING [dashboard_request_processor] [dashboard_request_processor] [fetch_dashboard_descriptions] [25941] Unable to parse dashboard description dashboard_id=https://127.0.0.1:8089/servicesNS/nobody/myapp/data/ui/views/mobile_today_01, request_id=ABC12345-6789-DEFG-HI01-JKLMNO234567 device_id=ABCDEFGHIJKLMNOPQRSTUVWXYZ12345678901234567= current_user=myuser is_alert=False

Ok, so Splunk can’t parse the dashboard. I checked the dashboard and made sure the role “cloudgateway” had access but still no luck. I even checked that the schedule search gave read permission to the role. No dice. Just by chance I created a new dashboard and happened to not use Scheduled Searches. I checked the mobile app and there it was, my awesome dashboard. Just to make sure I wasn’t seeing things I converted the inline to a saved search and then I got the errors again. Converted back to an inline search and there it was on my device. So just a helpful hint, if your dashboard isn’t showing up, check if there is scheduled searches.

Continue to Part 2, setting up Splunk TV.