Description of problem: With /mnt/client1 and /mnt/client2 being glusterfs native clients, the following test fails. touch /mnt/client1/foo ln /mnt/client1/foo /mnt/client1/foo.1 rm /mnt/client2/foo mv /mnt/client1/foo.1 /mnt/client1/foo mv: `foo.1' and `foo' are the same file Version-Release number of selected component (if applicable): 3.4.0 How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: Observed in RHEL 6.4.
This actually appears to be a mount/fuse,md-cache issue. For starters, the problem only occurs when md-cache is in the translator stack. The test sequence succeeds on a simple fuse+storage/posix volume, for example. What looks like occurs here is that while the target file has been unlinked from the 2nd mount, the associated dentry/inode still exists for the 1st mount (i.e., the inode is still linked in the inode table against the removed name). This appears to be the case based on a lookup for the newly unlinked file pointing state->loc.inode to the original inode. md-cache uses this loc->inode to look up cached attributes in the inode context. If the cached context has timed out, the request proceeds to the server and ultimately returns with an -1/ENOENT as intended. If the inode context is valid, however, md-cache turns the request around (into a successful lookup). The mv command includes some validation on the source and target which leads to successive lstat() calls on the source and target path. The lstat() call on the source populates md-cache with metadata on the inode and the lstat() call on the target results in a hit in md-cache due to the stale dentry/inode mapping.
I can get around this by unlinking the inode on a revalidate lookup. I'll post a patch and a test case for review...
REVIEW: http://review.gluster.org/5337 (mount/fuse: unlink the inode on revalidate if entry not found) posted (#1) for review on master by Brian Foster (bfoster)
REVIEW: http://review.gluster.org/5337 (mount/fuse: unlink the inode on revalidate if entry not found) posted (#2) for review on master by Brian Foster (bfoster)
COMMIT: http://review.gluster.org/5337 committed in master by Anand Avati (avati) ------ commit 70e5090326f029e6bcebd398572b39d3705d0bf3 Author: Brian Foster <bfoster> Date: Wed Jul 17 11:17:04 2013 -0400 mount/fuse: unlink the inode on revalidate if entry not found If an inode/dentry is linked via a client and removed via a separate client, the inode/dentry mapping in the initial client remains. A lookup of the removed name on the initial client typically returns ENOENT once the associated caches expire. If the initial client has multiple dentries linked to the same inode, however, lookups on the non-removed dentry create windows of time where lookups on the stale/removed name return successfully. This occurs because the stale mapping resolves to the still valid inode and tricks md-cache into returning valid lookup data. To correct this situation, unlink the stale inode mapping on a failed (ENOENT) revalidation lookup (i.e., when fuse has resolved the inode but a lookup returns ENOENT). Note that with this change, the state still occurs until an md-cache window has expired, allowed a lookup to pass through to the server and given the fuse translator an opportunity to clean up. Change-Id: I47dde2f11e2ef5b8dd51e9ac8be0f36cdb5081a3 BUG: 985074 Signed-off-by: Brian Foster <bfoster> Reviewed-on: http://review.gluster.org/5337 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Anand Avati <avati>
Brian, Carrying the discussion forward from review.gluster.com, I found that following use case results in file being deleted even without your patch. Test case: [root@unused build]# touch /mnt/gfs/file [root@unused build]# ln /mnt/gfs/file /mnt/gfs/link [root@unused build]# ls /mnt/gfs file link [root@unused build]# rm -fv /mnt/glusterfs/link removed `/mnt/glusterfs/link' [root@unused build]# mv /mnt/gfs/file /mnt/gfs/link mv: overwrite `/mnt/gfs/link'? y [root@unused build]# ls /mnt/gfs [root@unused build]# ls /home/export/ptop-new/ [root@unused build]# mount localhost:ptop on /mnt/glusterfs type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) localhost:ptop on /mnt/gfs type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) [raghu@unused glusterfs]$ uname -a Linux unused 3.3.4-5.fc17.x86_64 #1 SMP Mon May 7 17:29:34 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux After some debugging I found that kernel is sending unlink instead of rename. Please note that this bug does not occur if I set md-cache-timeout to 0. Since its a cache coherency issue, I am suspecting this to be a different manifestation of same bug. Hence I am reopening the bug. Brian says he cannot reproduce this bug in his setup. Hence, occurence of this bug might depend on version of kernel. Brian's patch which got accepted has no effect on this bug. The bug is being observed even without his patch and persists even in the presence of his patch. regards, Raghavendra.
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report. glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user