There is an enormous sequential I/O performance regression. In a simple O_DIRECT write test simulating a kvm/gluster sequential write, single-thread O_DIRECT write performance was only about 25% of its performance with buffered writes. How reproducible: Every time. Steps to Reproduce: 1. strace -f -ttT -o strc.log -p glusterfsd-pid & 2. on Gluster client mountpoint: dd if=/dev/zero of=/mnt/glusterfs/ben/x.dd bs=1024k count=8 oflag=direct 3. control-C to stop strace Actual results: results for buffered & O_DIRECT write & read from bare-metal hypervisor host gprfc001: [root@gprfc001 ben]# dd if=/dev/zero of=f01 bs=1024k count=8k 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 14.1862 s, 606 MB/s (LINE SPEED!) [root@gprfc001 ben]# dd if=/dev/zero of=f01 bs=1024k count=8k oflag=direct 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 51.8699 s, 166 MB/s [root@gprfc001 ben]# dd of=/dev/null if=f01 bs=1024k count=8k 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 19.1809 s, 448 MB/s [root@gprfc001 ben]# dd of=/dev/null if=f01 bs=1024k count=8k iflag=direct 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 36.0143 s, 239 MB/s The reason that buffered reads are slow is probably absence of the readahead translator -- we should be able to reach 800 MB/s with readahead translator, 1100 MB/s with read-ahead-page-count=16. Expected results: We used to be able to run writes near 10-GbE line speed.
I think your fix works, what compose is it in?
We can close this bug. I marked bug 922578 as a duplicate of this bug.
REVIEW: http://review.gluster.org/4697 (performance/write-behind: Enable write-behind when strict_O_DIRECT is not set.) posted (#2) for review on master by Vijay Bellur (vbellur)
COMMIT: http://review.gluster.org/4697 committed in master by Anand Avati (avati) ------ commit ea982a764b7cb447eb866129fef2cfafaa48eb6a Author: Vijay Bellur <vbellur> Date: Wed Mar 20 11:06:19 2013 +0530 performance/write-behind: Enable write-behind when strict_O_DIRECT is not set. When open() with O_DIRECT happens, write-behind was being disabled for the fd irrespective of strict_O_DIRECT option. This commit disables write-behind only when strict_O_DIRECT is enabled. Change-Id: Ieef180e52910c3bf64d46b26b0e5dc3b8542f6d2 BUG: 923556 Signed-off-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/4697 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy>
REVIEW: http://review.gluster.org/5108 (performance/write-behind: Enable write-behind when strict_O_DIRECT is not set.) posted (#1) for review on release-3.4 by Vijay Bellur (vbellur)
COMMIT: http://review.gluster.org/5108 committed in release-3.4 by Anand Avati (avati) ------ commit 96c530295eaf8b8ba6e231e6c77fa1cb9302498d Author: Vijay Bellur <vbellur> Date: Wed Mar 20 11:06:19 2013 +0530 performance/write-behind: Enable write-behind when strict_O_DIRECT is not set. When open() with O_DIRECT happens, write-behind was being disabled for the fd irrespective of strict_O_DIRECT option. This commit disables write-behind only when strict_O_DIRECT is enabled. Change-Id: Ieef180e52910c3bf64d46b26b0e5dc3b8542f6d2 BUG: 923556 Signed-off-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/4697 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy> Reviewed-on: http://review.gluster.org/5108 Reviewed-by: Anand Avati <avati>