Forwarding Splunk's internal logs to the indexers

On all Splunk nodes except the indexers, we want to have all of Splunk's internal logs forwarded to the indexers instead of indexing them locally; this reduces disk space usage and it makes the internal logs for all the Splunk nodes searchable without having to log into each node individually – you can determine which node a specific log entry came from by the host field.

In the /opt/splunk/etc/system/local directory, create an outputs.conf file and add the following entries (replacing the example <ipaddress>:9997 entries with the correct IP addresses and receiving port, if different) for your indexers. We'll let this node pick up this configuration upon the next restart of Splunk after you've set up the indexers. Remember to perform this step on all of your nodes (except indexers):

[indexAndForward]
index = false
 
[tcpout]
defaultGroup = dev_test_indexers 
forwardedindex.filter.disable = true 
indexAndForward = false 
 
[tcpout:dev_test_indexers]
server=172.31.28.223:9997,172.31.39.185:9997,172.31.13.169:9997

We're done with this node for now. We needed these components to be in place first so that we can point all the other servers to the license master for licensing, and point the indexers to this cluster master node during their setup.