Splunking with Chef Automate

Chef Automate has the ability to send out notifications of Chef Client failures and InSpec failures. These notifications can be to a Slack channel and / or to a Webhook.

This post shows how it is possible to send such notification to Splunk using a Webhook. This webhook is an Azure Function which acts as a relay to send the data to Splunk in the correct format. By sending the data into Splunk it allows such notifications to be searched and trends to be identified.

Introduction

Chef Automate can manage thousands of servers, so how can you keep up with failures that are happening across the server estate? This can be accomplished by using notifications.

Notificiations in Chef Automate can be configured for Chef Client Run (CCR) failures and InSpec failures. These notifications can be to a Slack channel or a Webhook. By using the webhook feature it is possible to send this data to another location, such as Splunk, which will allow analysis of these failures to be performed, thus helping to identify which nodes are most problematic and if there is a common trend.

The walkthough in the post will configure the following:

  • HTTP Event Collector (HEC) in Splunk
  • Azure Function to act as data relay
  • Notifications in Chef Automate

It is assumed that you have a Splunk instance or account, an Azure subscription and a Chef Automate environment.

Configuration

Splunk

The HEC is available in Splunk, but it has to be enabled. This activates the HEC in Splunk and defines the index to be used for storing data and what the format of the data is.

Enable HEC

  • Log into Splunk
    • Go to the instance you want data to end up in
  • From the navigation bar go to Settings > Data inputs
  • Click ‘HTTP Event Collector’
  • Click the Global Settings button
  • Set the ‘All Tokens’ to ‘Enabled’ and click on Save.

Enable HTTP Event Collector

Create HEC Input

  • Log into Splunk
    • Go to the instance you want data to end up in
  • From the navigation bar go to Settings > Data inputs
  • For the ‘HTTP Event Collector’ click on ‘Add new’
  • Set a ‘Name’ and then click Next
  • Set the ‘Source type’ to _json
  • Select the index (or create a new one) that the data should be saved to
  • Click Review
  • Click Submit

Create HTTP Event Collector Input

The Token that has been created, and can be seen in the list of Data Inputs, is required in the next step.

More information about setting up the HEC can be found here.

Azure Function

An Azure Resource Manager (ARM) template has been created which will allow you to deploy the necessary function to Azure that will allow the relaying of data from Chef Automate to Chef.

Repo: https://github.com/chef-partners/splunk-relay

The parameters file needs to be modified so that it has the information required to send data to Splunk.

Paremeter Description Example
splunkCustomerId The ID associated with the Splunk account into which notifications will be sent. This is the host in the url. prd-p-1234567abc
splunkToken This is the token that is associated with the HEC as created in the previous step FAD5E07B-B73B-4A5A-B2B9-2C03GT672C63
splunkChannel This can be any UUID. It is used to trace the messages that come in 17c23b1e-bf0a-4d6e-94fd-06114ef5c29f

To deploy the function run the following commamd. This can be achieved using Azure CLI 2.0 or PowerShell.

Azure CLI 2.0

The following commands will create a new Resource Group in Azure and then deploy the template to that resource group.

az group create -n automate-splunk-relay -l westeurope
az group deployment create -g automate-splunk-relay -n "Splunk Relay Function" --parameters @parameters.json --template-file azuredeploy.json

PowerShell

New-AzureRmResourceGroup -Name automate-splunk-relay -Location westeurope
New-AzureRmResourceGroupDeployment -Name "Splunk Relay Function" -ResourceGroupName automate-splunk-relay -TemplateFile azuredeploy.json -TemplateParameterFile parameters.json

Chef Automate Configuration

The deployment will contain in its output the two URLs that are required for both CCR failures and InSpec failures. This information will be in the deployment information for the Resource Group.

ARM Template Deployment Outputs

These two URLs are required to configure Chef Automate to send notifications.

  • Log into Chef Automate
  • Go to the “Nodes” tab
  • Click Notifications in the left hand navigation
  • Create 2 notifications, one for CCR failures and one for InSpec failures
    • Use the URLs from the ARM deployment for each of the notifications

Chef Automate WebHook Notifications

Logs in Splunk

If everything has been configured correctly then you will start to see notifications in your Splunk instance.

The following screenshot shows a CCR failure and an InSpec failure.

Chef Automate Notifications in Splunk

Share Comments