This is a list of things i do every time i log into a Linux server. It’s habit now and something you might consider doing too. Essentially we want to know who’s on the server, what the server state is and how different it is to normal.
Table of Contents
Check who’s on the server
Check who else is on the server is the “w” command. You can then broadcast a message to those other technicians to let them know they have company.
[agixuser2@www.example.com ~]$ w 12:21:18 up 402 days, 23:34, 4 users, load average: 0.00, 0.00, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT agixuser1 pts/6 192.168.1.2 03:38 8:19m 0.28s 0.45s sshd: agixuser1 [priv] agixuser2 pts/7 192.1678.1.6 12:21 0.00s 0.13s 0.09s w
You can see from the above that we have two users on that server including myself.
Say hi to the other logged-in technicians
I will let all other users (in this case just agixuser1) know i am here.
[agixuser2@www.example.com ~]$ wall "I'm glad to see you working on the weekend."
Check the servers uptime
Check how long the server has been up and what the load averages are:
[agixuser2@www.example.com ~]$ uptime 12:36:31 up 402 days, 23:49, 2 users, load average: 0.00, 0.00, 0.00
We can see the uptime is “402” days (not bad) and the load averages are all zeros. Load averages are based on server “load” 1 minute ago, 5 minutes ago and 15 minutes ago. The “load” is not a percentage but a calculated number based on all kinds of information. The lower, the better. If you see numbers above 10, your server is working very hard.
Check the disk space
Next we should check the disk space and memory utilisation.
[agixuser2@www.example.com ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-root
39G 18G 20G 48% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 243M 47M 184M 21% /boot
/dev/mapper/vg00-var 5.0G 3.6G 1.1G 77% /var
/dev/mapper/vg00-tmp 2.0G 109M 1.8G 6% /tmp
The disk space is fine. Anything near 90% “Use%” or where a filesystem is more used than i remember is cause for alarm.
Check the servers memory
Now check the memory using the “free” command. You can use “-m” to show the results in MB. Use “-G” as well for GB.
[agixuser2@www.example.com ~]$ free -m
total used free shared buffers cached
Mem: 8001 7538 463 25 638 5449
-/+ buffers/cache: 1450 6551
Swap: 4095 56 4039
The above server has “463” MB RAM free. Not cause for alarm given that it’s not swapping out (not using much swap space). If it was using more than a few hundred B of swap, i’d be concerned and up the memory on this server.
Check the servers CPU load
Next check the CPU utilisation and see if there is anything hogging resources.
top
The “top” command shows:
Nothing strange there. Tip, when you start “top”, press the number “1” key to seperate the CPU’s. This doesn’t work on a Mac/BSD server (as far as i know) but does on Linux.
Check the servers historic load
Finally we check the CPU and load history. We use the “sar” command.
[agixuser2@www.example.com ~]$ sar Linux 2.6.32-358.14.1.el6.x86_64 (www.example.com) 18/09/16 _x86_64_ (2 CPU) 00:00:01 CPU %user %nice %system %iowait %steal %idle 00:10:01 all 5.30 1.39 1.40 0.20 0.00 91.70 00:20:01 all 0.64 0.00 0.48 0.02 0.00 98.85 00:30:01 all 0.63 0.00 0.32 0.01 0.00 99.04 00:40:01 all 0.60 0.00 0.31 0.01 0.00 99.08 00:50:01 all 0.62 0.00 0.33 0.01 0.00 99.03 01:00:01 all 0.63 0.00 0.35 0.02 0.00 99.00 01:10:01 all 1.16 0.00 0.50 0.07 0.00 98.28 ...
The output goes over a few pages starting from midnight through to now with 10 minute intervals. The details are pretty self-explanatory but worth going over a little. The useful columns to us at a glance are “Time”, “%iowait” and “%idle”. We need to know the time to know when the server was under particular load. The i/o is important to know if the server is waiting on the underlying hardware to catch up which would indicate the disk hardware is a cause of performance degradation. Finally we can see how much of the time the server was idle and therefore how much it was under load.
Check for malicious scripts
One last thing i do is check what “perl” scripts are running. In my experience, a compromised server is likely to be used as a relay of some kind and “perl” is an excellent tool. So run the following to find out if any perl processes are running.
[agixuser2@www.example.com ~]$ ps aux | grep perl
At this point we have a great idea of how the server is looking. I hope this help. It does seem like a lot to do each time you log in and you don’t need to do this every time. But worth doing when you haven’t been on the given server for a while.