Splunk UF Missing Events From Indexing To The Splunk Indexer

If you encounter an issue of missing events in Splunk SH. Firstly, verify it by checking the counts in raw log file and the Splunk. If it happens to be true then troubleshoot as explained below -

"The most probable reason of missing the events in the SH would be due to HIGH volume of logs on the server for UF to process. Due to the more events per second, UF might be unable to process all the logs and missing some of them in between."

The default data limit set on UF is 256 KBPS. You may need to lift this limit to allow UF process all the logs irrespective of size of the data per second.

• To verify in the forwarder: When the thruput limit is reached, monitoring pauses and the following events are recorded in splunkd.log:

          grep 'retrying' splunkd.log

If it returns something like below then it's confirmed that UF is unale to send all the data to the indexer.

11-25-2021 09:40:24.990 +0500 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...

• To confirm this, check the metrics logs and see the size of instantaneous_kbps

 In Splunk - 

index=_internal source="*metrics.log" group=thruput instantaneous_kbps>256

                                                                

                                                               OR

In raw log file on UF - 

grep "name=thruput" metrics.log

• Now, check the limit defined on the UF and change it accordingly if it's the default one i.e. 256 KBPS

/opt/splunkuniversalforwarder/bin/./splunk cmd btool limits list thruput --debug

Output will be something like this -

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf [thruput]

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf maxKBps = 256

/opt/splunkforwarder/etc/system/default/limits.conf                        max_mem_usage_mb = 200

So, It's confirmed that thruput limit is set to 256 KBPS in this case and to uplift the limit, we'll have to create a limits.conf under system local and set it to 0.

[thruput]

maxKBps = 0

Once done, please restart the splunk deamon service on UF and verify the data.

Splunk official doc can be accessed at this location -

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Troubleshootingeventsindexingdelay#Possible_thruput_limits

Emails Not Going Out From Splunk /Office Email Account

 Emails Not Going Out From Splunk /Office Email Account

If Reports/Alerts not going out from Splunk i.e. email is not going out. 

Check for error in splunkd logs using -

index=_internal sourcetype=splunkd sendemail. If below error -

STOREDRV.Submission.Exception:QuotaExceededException.MapiExceptionQuotaExceeded; Failed to process message due to a permanent exception with message Cannot submit message

Then, check account's mailbox by logging to MS admin panel -

Click mail -> Edit exchange properties under 'More actions'

It will take you to new site - 

https://outlook.office365.com/ecp/UsersGroups/EditMailbox.aspx?exsvurl=1&mkt=en-US&Realm=milvik.se&id=6c1fda8a-a235-4627-9024-3a77503db5fd

Click mailbox usage 

If it's full then login to user account and go to -

https://outlook.office.com/mail/options/general/storage

Empty deleted items

Empty sent items

That's it. You've clear-up some space and, you'll be able to send /receive emails now. 

 If Splunk KV Store Fails To Start/Restart

1. Check the KV store status using below command from bin directory -

./splunk show kvstore-status -auth <user_name>:<password>

2. Check for fully qualified domain name (FQDN) -

hostname --fqdn

3. Now, create a new SSL certificate using above FQDN -

./splunk createssl server-cert 3072 -d /opt/splunk/etc/auth -n server -c <FQDN>

4. Restart Splunk

5. Check the status of KV store now. It should be fine.