andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/util
History
Peter (Stig) Edwards f8d8cf53fe Fix log_write_bench -bytes_per_sync option. (#1375) 
Hello and thanks for RocksDB,
 
When log_write_bench is run with the -bytes_per_sync option, the option does not influence any *sync* behaviour.
 
> strace -e trace=write,sync_file_range ./log_write_bench -record_interval 0 -record_size 1048576 -num_records 11 -bytes_per_sync 2097152 2>&1 | egrep '^(sync|write.*XXXX)'
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
 
I suspect that this is because the bytes_per_sync option now needs to be using a `WritableFileWriter` and not a `WritableFile`.
 
With the diff below applied, it changes to:
 
> strace -e trace=write,sync_file_range ./log_write_bench -record_interval 0 -record_size 1048576 -num_records 11 -bytes_per_sync 2097152 2>&1 | egrep '^(sync|write.*XXXX)'
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
sync_file_range(0x3, 0, 0x200000, 0x2)  = 0
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
sync_file_range(0x3, 0x200000, 0x200000, 0x2) = 0
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
sync_file_range(0x3, 0x400000, 0x200000, 0x2) = 0
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
sync_file_range(0x3, 0x600000, 0x200000, 0x2) = 0
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1048576) = 1048576
sync_file_range(0x3, 0x800000, 0x200000, 0x2) = 0
 
( Note that the first 1MB is not synced as mentioned in util/file_reader_writer.cc::WritableFileWriter::Flush() )
 
This diff also includes the fix from https://github.com/facebook/rocksdb/pull/1373
 
> diff -du util/log_write_bench.cc.orig util/log_write_bench.cc
--- util/log_write_bench.cc.orig        2016-10-04 12:06:29.115122580 -0400
+++ util/log_write_bench.cc     2016-10-05 07:24:09.677037576 -0400
@@ -14,6 +14,7 @@
 #include <gflags/gflags.h>

 #include "rocksdb/env.h"
+#include "util/file_reader_writer.h"
 #include "util/histogram.h"
 #include "util/testharness.h"
 #include "util/testutil.h"
@@ -38,19 +39,21 @@
   env_options.bytes_per_sync = FLAGS_bytes_per_sync;
   unique_ptr<WritableFile> file;
   env->NewWritableFile(file_name, &file, env_options);
+  unique_ptr<WritableFileWriter> writer;
+  writer.reset(new WritableFileWriter(std::move(file), env_options));

   std::string record;
-  record.assign('X', FLAGS_record_size);
+  record.assign(FLAGS_record_size, 'X');

   HistogramImpl hist;

   uint64_t start_time = env->NowMicros();
   for (int i = 0; i < FLAGS_num_records; i++) {
     uint64_t start_nanos = env->NowNanos();
-    file->Append(record);
-    file->Flush();
+    writer->Append(record);
+    writer->Flush();
     if (FLAGS_enable_sync) {
-      file->Sync();
+      writer->Sync(false);
     }
     hist.Add(env->NowNanos() - start_nanos);
2016-10-11 16:45:51 -07:00
..
aligned_buffer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
allocator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
arena_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
arena.cc
util/arena.cc: FreeBSD: More portable use of mmap(MAP_ANON) (#1254)
2016-08-10 13:52:23 -07:00
arena.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
autovector_test.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
autovector.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
bloom_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
bloom.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
build_version.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
cache_bench.cc
Introduce ClockCache
2016-08-19 12:28:19 -07:00
cache_test.cc
Fix cache_test valgrind_check failure
2016-08-29 10:40:00 -07:00
cf_options.cc
Support running consistency checks in release mode
2016-10-07 17:21:45 -07:00
cf_options.h
Support running consistency checks in release mode
2016-10-07 17:21:45 -07:00
channel.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
clock_cache.cc
Fix ClockCache memory leak
2016-08-31 08:56:34 -07:00
clock_cache.h
Introduce ClockCache
2016-08-19 12:28:19 -07:00
coding_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
coding.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
coding.h
TableBuilder / TableReader support for range deletion
2016-08-19 15:10:31 -07:00
compaction_job_stats_impl.cc
Single Delete Mismatch and Fallthrough statistics
2016-08-16 08:21:43 -07:00
comparator.cc
Improve BytewiseComparatorImpl::FindShortestSeparator
2016-04-25 23:02:14 -07:00
compression.h
Support ZSTD with finalized format
2016-09-06 12:22:16 -07:00
concurrent_arena.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
concurrent_arena.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
crc32c_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
crc32c.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
crc32c.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_options.cc
Split DBOptions into ImmutableDBOptions and MutableDBOptions
2016-09-23 16:34:04 -07:00
db_options.h
Split DBOptions into ImmutableDBOptions and MutableDBOptions
2016-09-23 16:34:04 -07:00
delete_scheduler_test.cc
Revert "Support SST files with Global sequence numbers"
2016-10-07 14:05:12 -07:00
delete_scheduler.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
delete_scheduler.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dynamic_bloom_test.cc
Fix clang analyzer errors
2016-07-08 17:50:51 -07:00
dynamic_bloom.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dynamic_bloom.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
env_basic_test.cc
TableBuilder / TableReader support for range deletion
2016-08-19 15:10:31 -07:00
env_chroot.cc
Support POSIX RandomRWFile
2016-09-13 12:08:22 -07:00
env_chroot.h
Introduce chroot Env
2016-05-06 17:42:50 -07:00
env_hdfs.cc
Add NoSpace subcode to IOError (#1320)
2016-09-07 12:37:45 -07:00
env_posix.cc
Support POSIX RandomRWFile
2016-09-13 12:08:22 -07:00
env_test.cc
Avoid hard-coded sleep in EnvPosixTestWithParam.TwoPools
2016-09-16 17:45:12 -07:00
env.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
event_logger_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
event_logger.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
event_logger.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
fault_injection_test_env.cc
[rocksdb] Recovery path sequence miscount fix
2016-05-10 14:06:07 -07:00
fault_injection_test_env.h
[rocksdb] Recovery path sequence miscount fix
2016-05-10 14:06:07 -07:00
file_reader_writer_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_reader_writer.cc
Cleanup unused variable pending_fsync_.
2016-08-05 10:31:41 -07:00
file_reader_writer.h
Cleanup unused variable pending_fsync_.
2016-08-05 10:31:41 -07:00
file_util.cc
Split DBOptions into ImmutableDBOptions and MutableDBOptions
2016-09-23 16:34:04 -07:00
file_util.h
Split DBOptions into ImmutableDBOptions and MutableDBOptions
2016-09-23 16:34:04 -07:00
filelock_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filter_policy.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
hash.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
hash.h
Refactor cache.cc
2016-07-15 10:41:36 -07:00
heap_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
heap.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
histogram_test.cc
Fix Build Error
2016-03-11 22:56:25 -08:00
histogram_windowing.cc
Fix in HistogramWindowingImpl
2016-03-17 14:28:41 -07:00
histogram_windowing.h
Histogram Concurrency Improvement and Time-Windowing Support
2016-03-11 16:54:25 -08:00
histogram.cc
Fix FB internal CI build failure
2016-03-15 11:38:15 -07:00
histogram.h
Histogram Concurrency Improvement and Time-Windowing Support
2016-03-11 16:54:25 -08:00
instrumented_mutex.cc
Ensure Correct Behavior of StatsLevel kExceptDetailedTimers and kExceptTimeForMutex (#1308)
2016-09-01 19:57:55 -07:00
instrumented_mutex.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
io_posix.cc
Support POSIX RandomRWFile
2016-09-13 12:08:22 -07:00
io_posix.h
Support POSIX RandomRWFile
2016-09-13 12:08:22 -07:00
iostats_context_imp.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
iostats_context_test.cc
IOStatsContext::ToString() add option to exclude zero counters
2016-02-23 10:26:24 -08:00
iostats_context.cc
IOStatsContext::ToString() add option to exclude zero counters
2016-02-23 10:26:24 -08:00
kv_map.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_buffer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_buffer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_write_bench.cc
Fix log_write_bench -bytes_per_sync option. (#1375)
2016-10-11 16:45:51 -07:00
logging.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
logging.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
lru_cache_test.cc
LRU cache mid-point insertion
2016-08-19 16:43:31 -07:00
lru_cache.cc
Not insert into block cache if cache is full and not holding handle
2016-08-23 13:53:49 -07:00
lru_cache.h
Option to cache index/filter blocks with priority
2016-08-23 13:44:13 -07:00
memenv.cc
In-memory environment read beyond EOF
2016-05-27 12:10:26 -07:00
mock_env_test.cc
Create env_basic_test [pluggable Env part 2]
2016-06-03 15:13:03 -07:00
mock_env.cc
In-memory environment read beyond EOF
2016-05-27 12:10:26 -07:00
mock_env.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
murmurhash.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
murmurhash.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
mutexlock.h
Persistent Read Cache (6) Persistent cache tier implentation - File layout
2016-07-19 12:01:46 -07:00
options_helper.cc
Split DBOptions into ImmutableDBOptions and MutableDBOptions
2016-09-23 16:34:04 -07:00
options_helper.h
Support running consistency checks in release mode
2016-10-07 17:21:45 -07:00
options_parser.cc
Refactor GetMutableOptionsFromStrings
2016-09-13 21:12:43 -07:00
options_parser.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
options_sanity_check.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
options_sanity_check.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
options_settable_test.cc
Support running consistency checks in release mode
2016-10-07 17:21:45 -07:00
options_test.cc
Refactor GetMutableOptionsFromStrings
2016-09-13 21:12:43 -07:00
options.cc
Support running consistency checks in release mode
2016-10-07 17:21:45 -07:00
perf_context_imp.h
fix ios build error
2016-02-17 20:22:40 +08:00
perf_context.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
perf_level_imp.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
perf_level.cc
Assert boundary checks for SetPerfLevel()
2016-06-01 09:07:09 -07:00
perf_step_timer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
posix_logger.h
Cleanup auto-roll logger flush-while-rolling test
2016-07-07 11:35:40 -07:00
random.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
random.h
Fixes warnings and ensure correct int behavior on 32-bit platforms.
2016-03-16 22:57:57 +01:00
rate_limiter_test.cc
loose the assertion condition of rate_limiter_test
2016-09-20 12:28:59 -07:00
rate_limiter.cc
fix rate limiter to avoid starvation
2016-07-01 00:16:29 -07:00
rate_limiter.h
Handle overflow case of rate limiter's paramters
2016-05-27 16:15:28 -07:00
sharded_cache.cc
add stats to Cache::LookUp()
2016-09-01 13:50:39 -07:00
sharded_cache.h
add stats to Cache::LookUp()
2016-09-01 13:50:39 -07:00
slice_transform_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
slice.cc
Merge db/slice.cc into util/slice.cc
2016-06-10 16:37:36 -07:00
sst_file_manager_impl.cc
Fix SstFileManager uninitialized data member
2016-02-18 11:25:19 -08:00
sst_file_manager_impl.h
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
statistics_test.cc
Fix Statistics TickersNameMap miss match with Tickers enum
2016-07-25 16:05:50 -07:00
statistics.cc
Add Statistics::getAndResetTickerCount().
2016-10-11 10:54:11 -07:00
statistics.h
Add Statistics::getAndResetTickerCount().
2016-10-11 10:54:11 -07:00
status_message.cc
Add NoSpace subcode to IOError (#1320)
2016-09-07 12:37:45 -07:00
status.cc
Add NoSpace subcode to IOError (#1320)
2016-09-07 12:37:45 -07:00
stderr_logger.h
Stderr info logger
2016-04-01 11:06:06 -07:00
stop_watch.h
Use StopWatch to do statistic job in db_impl_add_file.cc
2016-08-02 14:53:29 -07:00
string_util.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
string_util.h
FreeBSD does not have std::to_string (#1190)
2016-06-29 07:35:17 -07:00
sync_point.cc
Make SyncPoint return immediately when disabled
2016-08-16 06:19:46 -07:00
sync_point.h
Add a new feature to enforce a sync point only active on a thread
2016-07-07 11:29:14 -07:00
testharness.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
testharness.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
testutil.cc
Support running consistency checks in release mode
2016-10-07 17:21:45 -07:00
testutil.h
Revert "Support SST files with Global sequence numbers"
2016-10-07 14:05:12 -07:00
thread_list_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
thread_local_test.cc
Fix lambda expression for clang/windows
2016-08-23 13:34:56 -07:00
thread_local.cc
Fold function for thread-local data
2016-08-22 15:37:39 -07:00
thread_local.h
Fold function for thread-local data
2016-08-22 15:37:39 -07:00
thread_operation.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
thread_status_impl.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
thread_status_updater_debug.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
thread_status_updater.cc
Use pure if-then check instead of assert in EraseColumnFamilyInfo
2016-03-04 16:03:31 -08:00
thread_status_updater.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
thread_status_util_debug.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
thread_status_util.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
thread_status_util.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
threadpool_imp.cc
Fix Windows Build
2016-09-02 17:10:28 -07:00
threadpool_imp.h
Expose ThreadPool under include/rocksdb/threadpool.h
2016-08-26 10:41:35 -07:00
transaction_test_util.cc
Add multithreaded transaction test
2016-03-11 15:16:52 -08:00
transaction_test_util.h
Fix AppVeyor build error
2016-03-15 10:57:33 -07:00
xfunc.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xfunc.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xxhash.cc
Enable MS Warning C4804 : unsafe use of type 'bool' in operation
2015-11-18 16:23:19 -08:00
xxhash.h
Prevent xxhash symbols from polluting global namespace
2015-03-12 12:07:10 -07:00