SmartStore Resource Guide

The below method is a temporary way of installing SmartStore configuration & indexes. Starting from the Splunk Operator release 1.0.2, an enhanced App installation framework is introduced which is the recommended method to install SmartStore indexes & configuration. The below method may still be used to specify the S3 access keys, which avoids storing them in the S3 buckets (via the App installation framework)

The Splunk Operator includes a method for configuring a SmartStore remote storage volume with index support using a Custom Resource. The SmartStore integration is not implemented as a StorageClass. This feature and its settings rely on support integrated into Splunk Enterprise. See SmartStore for information on the feature and implementation considerations.

  • SmartStore configuration is supported on these Custom Resources: Standalone and ClusterManager.
  • SmartStore support in the Splunk Operator is limited to Amazon S3 & S3-API-compliant object stores only if you are using the CRD configuration for S3 as described below.”
  • For Amazon S3, if you are using interface VPC endpoints with DNS enabled to access AWS S3, please update the corresponding volume endpoint URL with one of the DNS names from the endpoint. Please ensure that the endpoint has access to the S3 buckets using the credentials configured. Similarly other endpoint URLs with access to the S3 buckets can also be used.
  • Use of GCS with SmartStore is supported by using configuration via Splunk App.
  • Specification allows definition of SmartStore-enabled indexes only.
  • Already existing indexes data should be migrated from local storage to the remote store as a pre-requisite before configuring those indexes in the Custom Resource of the Splunk Operator. For more details, please see Migrate existing data on an indexer cluster to SmartStore.

SmartStore configuration involves indexes, volumes, and the volume credentials. Indexes and volume configurations are configured through the Custom Resource specification. The credentials for accessing the volume can either be done through IAM roles or static credential. For roles these can be configured via service accounts or annotations. For static keys the volume credentials are configured securely in a Kubernetes secret object, and that secret object is referred by the Custom Resource with SmartStore volume spec, through SecretRef

Storing Smartstore Secrets

Here is an example command to encode and load your static remote storage volume secret key and access key in the kubernetes secret object: kubectl create secret generic <secret_store_obj> --from-literal=s3_access_key=<access_key> --from-literal=s3_secret_key=<secret_key>

Example: kubectl create secret generic s3-secret --from-literal=s3_access_key=iRo9guRpeT2EWn18QvpdcqLBcZmW1SDg== --from-literal=s3_secret_key=ZXvNDSfRo64UelY7Y4JZTO1iGSZt5xaQ2

Creating a SmartStore-enabled Standalone instance

  1. Configure remote store credentials by either:
    • Configure IAM role based credentials via service account or annotations, or
    • Create a Secret object with Secret & Access credentials, as explained in Storing SmartStore Secrets
  2. Confirm your S3-based storage volume path and URL.
  3. Confirm the name of the Splunk indexes being used with the SmartStore volume.
  4. Create/Update the Standalone custom resource specification with volume and index configuration (see Example below)
  5. Apply the custom resource specification: kubectl -f apply Standalone.yaml

Example. Standalone.yaml:

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: s1
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  smartstore:
    defaults:
      volumeName: s2s3_vol
    indexes:
      - name: networkmonitor
        volumeName: s2s3_vol
        remotePath: $_index_name
      - name: salesdata
      - name: oslogs
    volumes:
      - name: s2s3_vol
        path: indexdata-s2-bucket/standaloneNodes/s1data/
        endpoint: https://s3-us-west-2.amazonaws.com
        secretRef: s3-secret
  1. In the above example, indexdata-s2-bucket is the bucket name on remote storage, standaloneNodes/s1data is the relative path on that bucket in which the index data is stored.
  2. There are 3 indexes defined in the above config example, i.e networkmonitor, salesdata and oslogs.
  3. defaults: section is configured with the s3 volumeName parameter. An index can override with a specific volumeName, as shown for networkmonitor index. (Note: Unless multiple S3 volumes are used, specifying the volumeName in defaults: section makes it simple, and no need to repeat it across all the indexes)
  4. If the remotePath is not explicitly specified, a default value of $_index_name is assumed. For the semantics of $_index_name, please see indexes.conf in Splunk docs.

The SmartStore parameters will be placed into the required .conf files in an app. The app is named as splunk-operator. In the case of a standalone deployment, the app is located at /opt/splunk/etc/apps/

Custom apps with higher precedence can potentially overwrite the index and volume configuration in the splunk-operator app. Hence, care should be taken to avoid conflicting SmartStore configuration in custom apps. See Configuration file precedence order

Creating a SmartStore-enabled Indexer Cluster

  1. Configure remote store credentials by either:
    • Configure IAM role based credentials via service account or annotations, or
    • Create a Secret object with Secret & Access credentials, as explained in Storing SmartStore Secrets
  2. Confirm your S3-based storage volume path and URL.
  3. Confirm the name of the Splunk indexes being used with the SmartStore volume.
  4. Create/Update the Cluster Manager custom resource specification with volume and index configuration (see Example below)
  5. Apply the custom resource specification: kubectl -f apply Clustermanager.yaml
  6. Follow the rest of the steps to Create an Indexer Cluster. See Example Deployments

Example. Clustermanager.yaml:

apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
  name: <name>
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  smartstore:
    volumes:
      - name: <remote_volume_name>
        path: <remote_volume_path>
        endpoint: https://s3-<region>.amazonaws.com
        secretRef: <secret_store_obj>
    indexes:
      - name: <index_name_1>
        remotePath: $_index_name
        volumeName: <remote_volume_name>
      - name: <index_name_2>
        remotePath: $_index_name
        volumeName: <remote_volume_name>
      - name: <index_name_3>
        remotePath: $_index_name
        volumeName: <remote_volume_name>

The SmartStore parameters will be placed into the required .conf files in an app. The app is named as splunk-operator. In case of a Indexer cluster deployment, the app is located on Cluster manager at /opt/splunk/etc/manager-apps/. Once the SmartStore configuration is populated to Cluster Manager’s splunk-operator app, Operator issues a bundle push command to Cluster Manager, so that the SmartStore configuration is distributed to all the peers in that indexer cluster

Custom apps with higher precedence can potentially overwrite the index and volume configuration in the splunk-operator app. Hence, care should be taken to avoid conflicting SmartStore configuration in custom apps. See Configuration file precedence order

SmartStore Resource Spec Parameters

There are additional SmartStore settings available for tuning and storage management. The settings are equivalent to the SmartStore settings defined in indexes.conf and server.conf for Splunk Enterprise. The SmartStore resource applies to the Standalone and ClusterManager Custom Resources. Refer to Custom Resources Reference for all settings.

Following is the table that maps the Custom Resource SmartStore spec to the Splunk docs.

See indexes.conf, and server.conf for more information about these configuration details.

Custom Resource Spec Splunk Config Splunk Stanza
volumeName + remotePath remotePath [<index name>], [default] in indexes.conf
maxGlobalDataSizeMB maxGlobalDataSizeMB [<index name>], [default] in indexes.conf
maxGlobalRawDataSizeMB maxGlobalRawDataSizeMB [<index name>], [default] in indexes.conf
hotlistRecencySecs hotlist_recency_secs [<index name>], [cachemanager]
hotlistBloomFilterRecencyHours hotlist_bloom_filter_recency_hours [<index name>], [cachemanager]
endpoint remote.s3.endpoint [volume:<name>]
path path [volume:<name>]
maxConcurrentUploads max_concurrent_uploads [cachemanager]
maxConcurrentDownloads max_concurrent_downloads [cachemanager]
maxCacheSize max_cache_size [cachemanager]
evictionPolicy eviction_policy [cachemanager]
evictionPadding eviction_padding [cachemanager]

Additional configuration

There are SmartStore/Index config settings that are not covered by the Custom Resource SmartStore spec. If there is a need to configure additional settings, this can be achieved by configuring the same via Apps:

  1. Create an App with the additional configuration For example, in order to set the remote S3 encryption scheme as sse-s3, create an app with the config in indexes.conf file under default/local sub-directory as follows: 
    [volume:\<remote_volume_name\>]
path = <remote_volume_path>
remote.s3.encryption = sse-s3
  2. Apply the CR with the necessary & supported Smartstore and Index related configs
  3. Install the App created using the currently supported methods (Note: This can be combined with the previous step)

RepFactor for Internal Indexes

For Indexer Cluster, all Smartstore enabled indexes must have repFactor set to auto. However, by default, the Cluster Manager’s _cluster app(by default) sets the repFactor to 0 for internal indexes like _metrics, _introspection, _telemetry, _metrics_rollup, and _configtracker. If you want to replicate these indexes, please follow the instructions in the Additional Configuration section to set the repFactor to auto.