Whenever i get an alert from a system-monitoring tool like Munin or Nagios, it’s almost always a disk usage issue. Unfortunately the issue is usually cause by developers who don’t clean-up after deployments or accumulate temp files without a means to remove them when no longer needed. But there is still plenty we can do.
1. Remove yum cache files. I suppose “apt” has something similar but i’m a Redhat man. You can do a yum cleanup as simply as the following. I’ve never seen this go wrong and i consider it very low risk:
yum clean all
2. Remove old backups and archives that you (or another sysadmin) made prior to doing maintenance. For example, i often backup the “/etc/httpd” directory with “tar -czf ~/etc-httpd-20150820.tgz /etc/httpd” before i start creating and changing files. I leave those backups in place until i know the process was a success.
3. Remove old log files. This one can take some investigation but start with something like this:
du -h --max-depth=1 /var/log
And find where the largest log files are being stored. Often i find debug levels were too high and associated log files get too large as a result. Also go into the “/var/log” directory and run “ls -lh” to see what’s there. Sometimes i might even go to the extent (if i’m out of other options) to “rm /var/log/*2015* -f” to remove all past log files that have rolled over. Security tip: Make sure you have those logs backed up or access to the via some other method.
4. Find those hogging space and send them an email. Consider the following command which is much like the “du” above.
du -h --max-depth=1 /home
You can find those hogging disk space and get onto them. In fact, you can simply run “du -h –max-depth=1 /” to get the same result system-side.
5. Change the frequency of local backups and/or change the content being included. There are times when the server simply doesn’t change often enough to warrant daily backups. You might consider running backups on Mon, Wed and Fri only. You might also consider not backing up the “/var/lib” directory if you’re already dumping MySQL (for example) and backing up the database dump. This is a business decision though and should be thought through first.
I hope i’ve given you something good to go with. Please add comments and ask questions as you like.