Description of problem: Glusterfs quota directory showing incorrect useable size. I have enable quota on a directory and it is showing incorrect size of that directory. I have defined 1.1 TB for a directory and actual used size of that directory is 600GB when I check using du command. but when i am checking using gluster volume quota command then it is showing 800GB so there is difference of 200GB. Version-Release number of selected component (if applicable): [root@akash ~]# glusterfs --version glusterfs 3.2.7 built on Nov 27 2012 15:03:22 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
Hi Folks I am new to GFS, and would love to start working on bugs. Is this a valid bug & available to work on ? Regards, Mits
Hi, I am still waiting for response to proceed this bug. Please do the needful. Regards, Ramesh
Hi Ramesh && Mitesh, There is a similar bug reported, and it might be a duplicate of the same https://bugzilla.redhat.com/show_bug.cgi?id=812230. This happens in 2 cases, * If you've created hard links in different directories, quota treats it as 2 different contributions, where as du (disk usage) is not doubled for creating hard links. * Quota stores the limit in inode_ctx, and lookup (see quota_lookup_cbk) is responsible for updating the ctx. But in NFS mount the NFS doesn't send lookups, leading to wrong calculations of quota limits. If your problem relates to 1st case, its an expected behaviour, else its a bug as in later mentioned case. Wish you happy hacking into GFS.
I got into this bug with NFS. I am running gluster 3.5.2 on ubuntu 12.04 xfs bricks # gluster volume quota sas03 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /EgoTempSata03 322.0GB 80% 262.4GB 59.6GB # du -sh EgoTempSata03/ 81G EgoTempSata03/
Hi Peter, could you let us know if there are hard-links in the contents stored on your volume/directory? Thanks, Niels
There are no hard link on the volume/dir and i already have features.quota-deem-statfs on
I moved the volume from a replicated to a pure distributed volume and the mismatch still happening
Created attachment 982628 [details] Quota Verification Tool Hi Peter, Can you please run the attached script 'quota-verify' in the back-end for all the bricks where quota is enabled and provide us the output? The script basically crawls the directory given as argument. It collects quota "contri" and "size" extended attribute and also "block size" from stat call. script can create log whose size can be few MBs This script needs to be executed individually for all the bricks. Thanks and Regards, Vijay
Hi Vijaikumar Here are the outputs of all our 6 nodes on a particular brick path. It clearly shows mismatch Thanks! Peter root@glusterprod001:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02 [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.2 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.4 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.12 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.26 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.36 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.41 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.45 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.47 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.48 = 1999867904 (links: 1) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02/iqqa01 = 229984161280 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (229984161280 != 17998811136) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02 = 229984161280 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (229984161280 != 17998811136) root@glusterprod002:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02 [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.5 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.6 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.8 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.9 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.18 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.24 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.31 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.42 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.49 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.60 = 1999867904 (links: 1) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02/iqqa01 = 199697740288 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (199697740288 != 19998679040) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02 = 199697740288 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (199697740288 != 19998679040) root@glusterprod003:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02 [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.1 = 1999220224 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.13 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.16 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.20 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.30 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.50 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.58 = 1999867904 (links: 1) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02/iqqa01 = 115703288320 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (115703288320 != 13998427648) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02 = 115703288320 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (115703288320 != 13998427648) root@glusterprod004:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02 [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.14 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.21 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.27 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.32 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.35 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.39 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.40 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.55 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.61 = 1999867904 (links: 1) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02/iqqa01 = 143990489088 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (143990489088 != 17998811136) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02 = 143990489088 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (143990489088 != 17998811136) root@glusterprod005:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02 [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.3 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.22 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.25 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.28 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.33 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.34 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.43 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.44 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.51 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.53 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.54 = 1999867904 (links: 1) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02/iqqa01 = 139844609024 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (139844609024 != 21998546944) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02 = 139844609024 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (139844609024 != 21998546944) root@glusterprod006:~/scripts# ./quota-verify -b /brick04/gfs/AseIqBackupSata02 [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.7 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.10 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.11 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.15 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.17 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.19 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.23 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.29 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.37 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.38 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.46 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.52 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.56 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.57 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.59 = 1999867904 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa010115152000.62 = 1855666688 (links: 1) [DEBUG] file contri /brick04/gfs/AseIqBackupSata02/iqqa01/iqqa01_date.txt = 512 (links: 1) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02/iqqa01 = 159555537408 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02/iqqa01 (159555537408 != 31853685760) Hexadecimal number > 0xffffffff non-portable at ./quota-verify line 120. [DEBUG] dir size /brick04/gfs/AseIqBackupSata02 = 159555537408 [ERROR] quota size and actual size doesn't match for dir /brick04/gfs/AseIqBackupSata02 (159555537408 != 31853685760)
Hi Peter, Try below workaround and see if this helps solving the problem Set the below mentioned directory as dirty from back-end on all the nodes in cluster: # setfattr -n trusted.glusterfs.quota.dirty -v 0x3100 /brick04/gfs/AseIqBackupSata02/iqqa01 # setfattr -n trusted.glusterfs.quota.dirty -v 0x3100 /brick04/gfs/AseIqBackupSata02 From mount point initiate readdirp # ls -l /iqqa01 # ls -l /
Tested on DEV system and works! However when we did that on prod, it brought down the ALL the bricks on that volume. I have to start force for that volume and at the end it fixed the quota -Peter
is there any less intrusive way to do this? Seems like the number of files might affect the quota rescan process?
Hi Peter, Brick going down may be because of bug# 1176393 with readdirp, this is now fix in latest code. With the re-scan process, the only impact could be performance (depends on the depth of the sub-directory) tree till the process is complete. We are already working on improving quota accounting, this will solve the accounting going wrong.
Does lastest code means 3.6.2 or 3.5.3? I am on 3.5.2 and hope same version patch would fix this. Thanks Peter
Hi Peter, Bug# 1176393 is fixed in 3.7. We will back-port the same to 3.6
How about back port to 3.5?? Thanks! Peter
Sure, we will back-port the fix to 3.5 and 3.6
(In reply to Vijaikumar Mallikarjuna from comment #18) > Sure, we will back-port the fix to 3.5 and 3.6 Is it still planned to back-port this to 3.5? Thanks! Chris
Hi Christian, In 3.7, we have made multiple fixes and quota-marker code is re-written. So it is hard to back-port these changes to 3.5 Thanks, Vijay
Many issues related to quota is fixed in 3.7.11, please file a new bug if it happens again in 3.7.11 or greater