CHANGING SPLUNK DB LOCATION

If our Splunk installation lives at /opt/splunk then, by default, the #SPLUNK_DB path is going to be set to /opt/splunk/var/lib/splunk. 

Each index occupies a set of directories on the disk. By default, these directories live in #SPLUNK_DB

If you want to change the #SPLUNK_DB location while setting up the Splunk first then you just need to set the SPLUNK_DB stanza in /opt/splunk/etc/splunk-launch.conf

SPLUNK_DB=<new dir>/splunk

However, there can be another scenario wherein your Splunk DB path was already set to some other location however, you would want to change it again to a new directory. To get that done, follow the steps mentioned-

Step-1: Stop the splunkd

Step-2: Copy the complete dir data of #SPLUNK_DB to the new dir /disk. You may use the 'cp' or 'rsync' utility in Linux to do that. Detailed use of rsync is provided below.

Step-3: Modify the splunk-launch.conf under /opt/splunk/etc/ to set the $SPLUNK_DB path pointing to the new disk /dir

Step-4: Rename the 'mongo' folder to some other name inside the newly copied volume data. <new dir>/splunk/kvstore otherwise you may notice an error after starting the Splunk stating 'KV Store Process Terminated'

mv mongo mongo_bk

Step-4: Start the splunkd

(OPTIONAL) Step-5: If Splunk started but you see error messages on GUI then that might be due to permission issues on some folders and files. Update the ownership of such folders using the below command -

chown -R splunk:splunk

All set!

==================================

Here is how you can use rsync to copy data from the source location to the destination.

rysnc -avzh -P /src /dest

• With the -a option, the rsync will try to preserve both the exact permissions and group of the source
• If you want to see each file that gets copied as it happens, add the -v option
• Also, you can add -P to get a progress bar on each files which is helpful when you have very large files.
• -H Try to maintain hard links within transferred files
• The key rule here to remember is to use the -Z option when you are rsyncing files INTO your group storage areas
• If you want files on the target destination side to be deleted if they do not exist at the source. To do this you simply add the –delete option to rsync.
• The --update or -u option allows rsync to skip files that are still new in the destination directory
• To skip already existing files on the remote machine, we can include the --ignore-existing option

Subsequently, to sync only updated or modified files on the remote machine that have changed on the local machine, we can perform a dry run before copying files as below:

rsync -av --dry-run --update /src /dest

rsync -av --update /src /dest