Tuesday, July 15, 2014

Linux: Check inode usage in folders from High to Low

Inodes are temporarily files stored on your server which are currently in use and not closed. They could be cache, temporary files or anything related and could be created in several different ways.
Your server most likely has an inode usage quota (especially on VPS and shared hosting accounts) and if you exceed this it has a very similar effect to exceeding your actual disk space or random access memory (ram). The server locks up and services start failing one by one in a random fashion.
Below is a command to see the directories with the highest to lowest inode usage on your server, in the current directory. Doing this in your server root might take very long so best is to CD into the folder/directory where you expect the high inode usage might be and then execute this command:
List of top 10 inodes
[root@root]# find -maxdepth 1 -mindepth 1 -type d |while read dir ; do echo -n "$dir - " ; find "$dir" |wc -l; done|sed 's|^./||'|column -t|sort -rnk3|head -n10
List of All inodes 
[root@root]# find -maxdepth 1 -mindepth 1 -type d |while read dir ; do echo -n "$dir - " ; find "$dir" |wc -l; done|sed 's|^./||'|column -t|sort -rnk3

Once you see which folders/directories have the highest inode usage, you can cd into them until you find the folder with the actual problem. It could be a cache folder or a folder storing temporary images or something related.

Monday, July 14, 2014

Freeing up Inodes - Cleaning Inodes

Today I run into a weird problem. When I tried to open my (Serendipity) blog I got this strange error:

Query failed: 

SELECT 
                    
 e.id,
 ...
 ORDER BY timestamp DESC
 LIMIT 15

/ Can't create/write to file '/tmp/#sql_6ee_0.MYI' (Errcode: 28)
I was kind of puzzled. All other sites were working fine (such us wiki and list of pim prototypes). I sshed to a server and checked the disk usage:
myusername@pim:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             3.8G  3.0G  685M  82% / 
Everything fine here. I checked the /tmp folder
myusername@pim:/tmp# ls -la 
total 32
drwxrwxrwt  4 root root 20480 Apr 18 12:42 .
drwxr-xr-x 22 root root  4096 Apr 18 11:20 ..
drwxrwxrwt  2 root root  4096 Mar 24 22:42 .ICE-unix
Nothing strange here either. I tried to login to mysql:
myusername@pim:/tmp$ mysql -u myusername -p
Enter password: 
Welcome to the MySQL monitor
Everything worked fine. So I checked the ERROR 28 the web page was giving me
myusername@pim:/tmp$ perror 28
OS error code  28:  No space left on device
tried to create a file with no success:
myusername@pim:/tmp$ touch bla.txt
touch: cannot touch `bla.txt': No space left on device
Then I googled a bit and found out that I might be running out of inodes.
myusername@pim:/tmp$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             250976  250957      19  100% /
Yep, that was it. Now I had to find which folder had a lot of files that were taking up the inodes (as root). Not all results are shown in the below output.
myusername@pim:/var$ cd /
myusername@pim:/$ sudo bash 
root@pim:/$ for i in /*; do echo $i; find $i -type f | wc -l; done
/bin 91
/boot 23
/dev 346
/etc 745
/home 5
/lib 1709
/media 0
/proc 11435
/root 5
/sbin 103
/sys 3863
/tmp 0
/usr 45188
/var    186774
went to /var and run the same command
root@pim:/var$ for i in ; do echo $i; find $i -type f | wc -l ;done
backups 1
cache 55
lib 173682
log 47
mail 2
run 16
spool 19
www 12950
So I needed to check /var/lib
mkljun@pim:/var/lib$ for i in ; do echo $i; sudo find $i -type f | wc -l ;done
apt 9
doc-base 37
dpkg 2342
misc 2
mlocate 1
mysql 665
php5 170453
screen-profiles 21
ucf 23
xml-core 2
The folder /var/lib/php5 contained 170k files! It was kind of strange why all these files were not deleted after a session timeout. I certainly did not have 170k users visiting my pages (I wished this to be true though).
root@pim:/var/lib# cd php5
root@pim:/var/lib/php5# ls -l | wc -l
170453
checked the maximum time set for a php5 sessions to live.
root@pim:/var/lib/php5# /usr/lib/php5/maxlifetime
24
24 minutes. I decided to delete older files.
root@pim:/var/lib/php5# find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 -exec rm {} \;
root@pim:/var/lib/php5# ls -la
total 11768
drwx-wx-wt  2 root     root     12017664 Apr 18 12:35 .
drwxr-xr-x 38 root     root         4096 Sep 22  2010 ..
-rw-------  1 www-data www-data      168 Apr 18 12:23 sess_389b8191e2563e8bb43c41e20e015c0d
-rw-------  1 www-data www-data      118 Apr 18 12:11 sess_52c9acc7779f5cfd7e7e92a900fe7f19
-rw-------  1 www-data www-data      160 Apr 18 12:27 sess_bb8cfa842ceb0027cc9e4082ad8b0419
checked the inodes usage.
root@pim:/tmp# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             250976   80524  170452   33% /
That was it. 

Why PHP is not deleting expired sessions is another question. I went to my php.ini file and found this:
; After this number of seconds, stored data will be seen as 'garbage' and
; cleaned up by the garbage collection process.
session.gc_maxlifetime = 1440

; NOTE: If you are using the subdirectory option for storing session files
;       (see session.save_path above), then garbage collection does not
;       happen automatically.  You will need to do your own garbage
;       collection through a shell script, cron entry, or some other method.
So all I needed to do was to create a cron job (I decided to rut it every hour as root)
root@pim:/tmp# crontab -e 

0      /usr/bin/find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 -exec /bin/rm {} \;

Hopefully this will help some other soul out there.

Example-2
Clean up inodes in linux

root@server1:/var/www/wespell.com# df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/sda       1966080 1957683   8397  100% /
udev            126290     405 125885    1% /dev
tmpfs           127451     320 127131    1% /run
none            127451       4 127447    1% /run/lock
none            127451       1 127450    1% /run/shm

Lets find which folder has the most inodes
root@server1:/var/www/wespell.com# cd /; for i in *; do echo -n $i,; find $i -print | wc -l; done
backup,5
bin,130
boot,11
dev,407
etc,2331
home,421
initrd.img,1
lib,2063
lib64,2
lost+found,1
media,1
mnt,1
opt,1
proc,47112
root,6227
run,322
sbin,146
selinux,1
srv,1
sys,17406
tmp,8025
usr,57375
var,1881350
vmlinuz,1

We see that that was VAR folder, run the same command under var.

In our case it was spool/postfix and deferred emails. Here what I did to clean up.
I don’t use postfix at all, so I will basically clear it and disable.

Lets Flush postfix mail queue
postsuper -d ALL
postsuper -d ALL deferred

And now we have
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda       1966080 201470 1764610   11% /
udev            126290    405  125885    1% /dev
tmpfs           127451    320  127131    1% /run
none            127451      4  127447    1% /run/lock
none            127451      1  127450    1% /run/shm
Yappp, now we have enough space.