Description of problem: Gluster / Swift integration does not track when a given piece of metadata has been changed by the user. One request that is out of date could come in after another later request over-writing the new metadata with the old. Version-Release number of selected component (if applicable): RHS 2.0, GlusterFS 3.3.0 How reproducible: Tough to reproduce. Would have to fake up a delayed operation in order to see this happen easily. Steps to Reproduce: n/a Actual results: Old metadata potentially over-writes new metadata. Expected results: New metadata remains in tact. Additional info:
Today we ignore the Swift method of tracking which metadata updates to apply using timestamps. We need to properly track the time stamps and only apply new metadata changes. This requires that all servers properly keep their system times in sync.
The history of this is rooted in how Swift stores metadata on the volume. There are two paths for storing metadata in swift: 1. When the object is created only * Implies that a POST method is rewritten at the proxy server layer as a COPY operation with updated metadata (I think, one has to check the code to verify this is true) 2. Using a fast-POST method * There is a setting for the object server that tells the object server to not use a COPY and POST method, but to use the object servers POST method directly In the first case, all objects are written once, as new objects simple entirely rewrite the old, including the metadata. In the second case, all objects are still written once, but additional .meta files are created to store the metadata POST payloads. In regular Swift, objects are actually stored on disk using a directory (in the partition hierarchy, hashed by it's name), using files name <timestamp>.data, where multiple PUT operations use the timestamp of the operation to name the .data file. That way new writes don't strictly over-write old ones, they just supersede them in the sorted list of timestamp-named files. The .meta files from the fast-POST operations also use timestamps, so they'll end up sorting after the original data files ensuring that metadata is considered and not the metadata originally stored with the .data files. Now for gluster-swift integration, we just write metadata into the existing file, but there is no facility to globally order all operations on the file such that the most recent metadata writes win. Since there is no facility to atomically update the extended attributes used to store this metadata, there is really no way to ensure the concerns of this bug can be solved. One more piece of background, there is an X-PUT-TIMESTAMP header that is added by the client, or if it is missing, added by the proxy server to ensure that each request has an explicit timestamp for ordering purposes. This header is what is used by regular swift to provide timestamp for use on disk (I believe this is true, perhaps this needs to be double checked).