14/11/20 15:39:44 [Executor task launch worker-1 ] INFO Executor: Running task 5.0 in stage 0.0 (TID 5) 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO TorrentBroadcast: Started reading broadcast variable 1 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: ensureFreeSpace(1890) called with curMem=0, maxMem=7223692492 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1890.0 B, free 6.7 GB) 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO BlockManagerMaster: Updated info of block broadcast_1_piece0 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO TorrentBroadcast: Reading broadcast variable 1 took 333 ms 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: ensureFreeSpace(2544) called with curMem=1890, maxMem=7223692492 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.5 KB, free 6.7 GB) 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO HadoopRDD: Input split: s3n://mybucket/myfile:335544320+67108864 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO TorrentBroadcast: Started reading broadcast variable 0 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: ensureFreeSpace(32763) called with curMem=4434, maxMem=7223692492 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.0 KB, free 6.7 GB) 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO TorrentBroadcast: Reading broadcast variable 0 took 101 ms 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: ensureFreeSpace(294833) called with curMem=37197, maxMem=7223692492 14/11/20 15:39:45 [Executor task launch worker-1 ] INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 287.9 KB, free 6.7 GB) 14/11/20 15:39:46 [Executor task launch worker-1 ] INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 14/11/20 15:39:46 [Executor task launch worker-1 ] DEBUG NativeS3FileSystem: getFileStatus retrieving metadata for key 'myfile' 14/11/20 15:39:46 [Executor task launch worker-1 ] DEBUG Jets3tNativeFileSystemStore: Getting metadata for key: myfile from bucket:mybucket 14/11/20 15:39:46 [Executor task launch worker-1 ] DEBUG NativeS3FileSystem: getFileStatus returning 'file' for key 'myfile' ===> 14/11/20 15:39:46 [Executor task launch worker-1 ] INFO NativeS3FileSystem: Opening 's3n://mybucket/myfile' for reading 14/11/20 15:39:46 [Executor task launch worker-1 ] DEBUG Jets3tNativeFileSystemStore: Getting key: myfile from bucket:mybucket <=== AAHH WHY? 14/11/20 15:40:35 [Executor task launch worker-1 ] INFO NativeS3FileSystem: Opening key 'myfile' for reading at position '335544320' 14/11/20 15:40:35 [Executor task launch worker-1 ] DEBUG Jets3tNativeFileSystemStore: Getting key: myfile from bucket:mybucket with byteRangeStart: 335544320 14/11/20 15:41:07 [Executor task launch worker-1 ] INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 1930 bytes result sent to driver 14/11/20 15:41:07 [Executor task launch worker-1 ] INFO Executor: Running task 11.0 in stage 0.0 (TID 11) 14/11/20 15:41:07 [Executor task launch worker-1 ] INFO HadoopRDD: Input split: s3n://mybucket/myfile:738197504+67108864 14/11/20 15:41:07 [Executor task launch worker-1 ] DEBUG NativeS3FileSystem: getFileStatus retrieving metadata for key 'myfile' 14/11/20 15:41:07 [Executor task launch worker-1 ] DEBUG Jets3tNativeFileSystemStore: Getting metadata for key: myfile from bucket:mybucket 14/11/20 15:41:07 [Executor task launch worker-1 ] DEBUG NativeS3FileSystem: getFileStatus returning 'file' for key 'myfile' 14/11/20 15:41:07 [Executor task launch worker-1 ] INFO NativeS3FileSystem: Opening 's3n://mybucket/myfile' for reading 14/11/20 15:41:07 [Executor task launch worker-1 ] DEBUG Jets3tNativeFileSystemStore: Getting key: myfile from bucket:mybucket