Splunk UF Missing Events From Indexing To The Splunk Indexer
If you encounter an issue of missing events in Splunk SH. Firstly, verify it by checking the counts in raw log file and the Splunk. If it happens to be true then troubleshoot as explained below -
"The most probable reason of missing the events in the SH would be due to HIGH volume of logs on the server for UF to process. Due to the more events per second, UF might be unable to process all the logs and missing some of them in between."
The default data limit set on UF is 256 KBPS. You may need to lift this limit to allow UF process all the logs irrespective of size of the data per second.
- To verify in the forwarder: When the thruput limit is reached, monitoring pauses and the following events are recorded in splunkd.log:
grep 'retrying' splunkd.log
If it returns something like below then it's confirmed that UF is unale to send all the data to the indexer.
11-25-2021 09:40:24.990 +0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
- To confirm this, check the metrics logs and see the size of instantaneous_kbps
In Splunk -
index=_internal source="*metrics.log" group=thruput instantaneous_kbps>256
OR
In raw log file on UF -
grep "name=thruput" metrics.log
- Now, check the limit defined on the UF and change it accordingly if it's the default one i.e. 256 KBPS
/opt/splunkuniversalforwarder/bin/./splunk cmd btool limits list thruput --debug
Output will be something like this -
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf [thruput]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf maxKBps = 256
/opt/splunkforwarder/etc/system/default/limits.conf max_mem_usage_mb = 200
So, It's confirmed that thruput limit is set to 256 KBPS in this case and to uplift the limit, we'll have to create a limits.conf under system local and set it to 0.
[thruput]
maxKBps = 0
Once done, please restart the splunk deamon service on UF and verify the data.
Splunk official doc can be accessed at this location -
