Commit Graph

4138 Commits

Author SHA1 Message Date
krad
778cf4449b Adding email notification.
Summary: The email notifications needs to be at command level and the job level.
Adding command level notification.

Test Plan: Run command manually

Reviewers: igor sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-08 14:19:03 -07:00
Andres Noetzli
3a0df7f161 Fixed comparison in ForwardIterator when computing hint for GetNextLevelIndex()
Summary: When computing the hint for GetNextLevelIndex(), ForwardIterator was doing a redundant comparison. This patch fixes the comparison (using https://github.com/facebook/rocksdb/blob/master/db/version_set.cc#L158 as a reference) and moves it inside an assert because we expect `level_files[f_idx]` to contain the next key after Seek(), so user_key should always be smaller than the largest key.

Test Plan: make clean all check

Reviewers: rven, anthony, yhchiang, igor, sdong

Reviewed By: sdong

Subscribers: tnovak, sdong, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46227
2015-09-08 09:47:54 -07:00
Venkatesh Radhakrishnan
91f3c90792 Fix case when forward iterator misses a new update
Summary:
This diff fixes a case when the forward iterator misses a new
insert when the mutable iterator is not current. The test is also
improved and the check for deleted iterators is made more informative.

Test Plan: DBTailingIteratorTest.*Trim

Reviewers: tnovak, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D46167
2015-09-04 14:28:45 -07:00
Igor Canadi
ff1953c890 Merge pull request #707 from dkorolev/master
Fixed a typo in INSTALL.md
2015-09-03 19:52:27 -07:00
Dima
7b463e657b Fixed a typo in INSTALL.md 2015-09-03 19:46:09 -07:00
Mayank Pundir
d9f42aa60d Adding a verifyBackup method to BackupEngine
Summary: This diff adds a verifyBackup method to BackupEngine. The method verifies the name and size of each file in the backup.

Test Plan: Unit test cases created and passing.

Reviewers: igor, benj

Subscribers: zelaine.fong, yhchiang, sdong, lgalanis, dhruba, AaronFeldman

Differential Revision: https://reviews.facebook.net/D46029
2015-09-03 17:27:21 -07:00
Paul Marinescu
50dc5f0c5a Replace BackupRateLimiter with GenericRateLimiter
Summary: BackupRateLimiter removed and uses replaced with the existing GenericRateLimiter

Test Plan:
make all check

make clean
USE_CLANG=1 make all

make clean
OPT=-DROCKSDB_LITE make release

Reviewers: leveldb, igor

Reviewed By: igor

Subscribers: igor, dhruba

Differential Revision: https://reviews.facebook.net/D46095
2015-09-03 17:00:09 -07:00
krad
20ef64cae0 Moving jobs to use gcc-4.9
Summary: Fixed the glitch in Sandcastle and added gcc-4.9 support. Moving the
jobs to 4.9

Test Plan: Manually run the script

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-03 15:20:36 -07:00
agiardullo
0f1aab6c12 Add SetLockTimeout for Transactions
Summary: MyRocks wants to be able to change the lock timeout of a transaction that has already started.  Expose existing SetLockTimeout function to users.

Test Plan: unit test

Reviewers: spetrunia, rven, sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45987
2015-09-02 20:07:19 -07:00
Igor Canadi
14456aea52 Fix compile
Summary: There was a merge conflict with https://reviews.facebook.net/D45993

Test Plan: make check

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46065
2015-09-02 16:05:53 -07:00
Igor Canadi
76f286cc82 Optimize bloom filter cache misses
Summary:
This optimizes the case when (cache_index_and_filter_blocks=1) and bloom filter is not present in the cache.

Previously we did:
1. Read meta block from file
2. Read the filter position from the meta block
3. Read the filter

Now, we pre-load the filter position on Table::Open(), so we can skip steps (1) and (2) on bloom filter cache miss. Instead of 2 IOs, we do only 1.

Test Plan: make check

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46047
2015-09-02 15:36:47 -07:00
Yueh-Hsuan Chiang
0e6e5472e1 Fixed a compile warning in rocksjni/loggerjnicallback.cc
Summary:
This patch fixes the following compile warning.
    java/rocksjni/loggerjnicallback.cc: In constructor ‘rocksdb::LoggerJniCallback::LoggerJniCallback(JNIEnv*, jobject)’:
    java/rocksjni/loggerjnicallback.cc:19:14: warning: unused variable ‘rs’ [-Wunused-variable]
       const jint rs = env->GetJavaVM(&m_jvm);
                  ^

Test Plan: make rocksdbjavastaticrelease

Reviewers: sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45981
2015-09-02 15:07:44 -07:00
krad
b8a962d4f5 Adding commands for few more CI jobs.
Summary: Added commands to spawn crash, stress, asan, asan_crash, clang CI jobs

Test Plan: Manual test

Reviewers: sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-02 14:06:03 -07:00
Igor Canadi
8a2d59a356 Add Cloudera's blog post to USERS.md
Summary: As title

Test Plan: none

Reviewers: yhchiang, rven, anthony, IslamAbdelRahman, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46053
2015-09-02 14:04:51 -07:00
Andres Noetzli
3c9cef1eed Unified maps with Comparator for sorting, other cleanup
Summary:
This diff is a collection of cleanups that were initially part of D43179.
Additionally it adds a unified way of defining key-value maps that use a
Comparator for sorting (this was previously implemented in four different
places).

Test Plan: make clean check all

Reviewers: rven, anthony, yhchiang, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45993
2015-09-02 13:58:22 -07:00
sdong
3e0a672c50 Bug fix: table readers created by TableCache::Get() doesn't have latency histogram reported
Summary: TableCache::Get() puts parameters in the wrong places so that table readers created by Get() will not have the histogram updated.

Test Plan: Will write a unit test for that.

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D46035
2015-09-02 12:57:07 -07:00
Igor Canadi
b42cd6bed5 Remove the need for LATEST_BACKUP in BackupEngine
Summary:
In the first implementation of BackupEngine, LATEST_BACKUP was the commit point. The backup became committed after the write to LATEST_BACKUP completed.

However, we can avoid the need for LATEST_BACKUP. Instead of write to LATEST_BACKUP, the commit point can be the rename from `meta/<backup_id>.tmp` to `meta/<backup_id>`. Once we see that there exists a file `meta/<backup_id>` (without tmp), we can assume that backup is valid.

In this diff, we still write out the file LATEST_BACKUP. We need to do this so that we can maintain backward compatibility. However, the new version doesn't depend on this file anymore. We get the latest backup by `ls`-ing `meta` directory.

This diff depends on D41925

Test Plan: Adjusted backupable_db_test to this new behavior

Reviewers: benj, yhchiang, sdong, AaronFeldman

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42069
2015-09-02 11:49:49 -07:00
Siying Dong
0f763db20a Merge pull request #705 from yuslepukhin/rate_limiter_fix
Make WinEnv::NowMicros return system time
2015-09-02 11:35:17 -07:00
Amit Arya
20c44fefb7 t6913679: Use fallocate on LOG FILESS
Summary: Use fallocate on LOG FILES to

Test Plan:
make check
+
===check with strace===

[arya@devvm1441 ~/rocksdb] strace -e trace=fallocate ./ldb --db=/tmp/test_new scan
fallocate(3, 01, 0, 4194304)            = 0

Reviewers: sdong, anthony, IslamAbdelRahman, kradhakrishnan, lgalanis, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D45969
2015-09-02 11:17:02 -07:00
Dmitri Smirnov
f14c3363e1 Make WinEnv::NowMicros return system time
Previous change for the function
  555ca3e7b7 (diff-bdc04e0404c2db4fd3ac5118a63eaa4a)
  made use of the QueryPerformanceCounter to return microseconds values that do not repeat
  as std::chrono::system_clock returned values that made auto_roll_logger_test fail.

 The interface documentation does not state that we need to return
 system time describing the return value as a number of microsecs since some
 moment in time. However, because on Linux it is implemented using gettimeofday
 various pieces of code (such as GenericRateLimiter) took advantage of that
 and make use of NowMicros() as a system timestamp. Thus the previous change
 broke rate_limiter_test on Windows.

 In addition, the interface name NowMicros() suggests that it is actually
 a timestamp so people use it as such.

 This change makes use of the new system call on Windows that returns
 system time with required precision. This change preserves the fix
 for  auto_roll_logger_test and fixes rate_limiter_test.

 Note that DBTest.RateLimitingTest still fails due to a separately reported issue.
2015-09-02 11:12:07 -07:00
Yueh-Hsuan Chiang
aad0572f83 Fixed the build issue of rocksdbjavastaticrelease
Summary: This patch fixed couple build issues of rocksdbjavastaticrelease.

Test Plan: make rocksdbjavastaticrelease

Reviewers: sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45915
2015-09-01 11:07:43 -07:00
Tomislav Novak
5508122ed6 Fix a perf regression in ForwardIterator
Summary:
I noticed that memtable iterator usually crosses the `iterate_upper_bound`
threshold when tailing. Changes introduced in D43833 made `NeedToSeekImmutable`
always return true in such case, even when `Seek()` only needs to rewind the
memtable iterator. In a test I ran, this caused the "tailing efficiency"
(ratio of calls to `Seek()` that only affect the memtable versus all seeks)
to drop almost to zero.

This diff attempts to fix the regression by using a different flag to indicate
that `current_` is over the limit instead of resetting `valid_` in
`UpdateCurrent()`.

Test Plan: `DBTestTailingIterator.TailingIteratorUpperBound`

Reviewers: sdong, rven

Reviewed By: rven

Subscribers: dhruba, march

Differential Revision: https://reviews.facebook.net/D45909
2015-09-01 09:54:30 -07:00
Andres Notzli
b722007778 Fix listener_test when using ROCKSDB_MALLOC_USABLE_SIZE
Summary:
Flushes in listener_test happened to early when ROCKSDB_MALLOC_USABLE_SIZE was
active (e.g. when compiling with ROCKSDB_FBCODE_BUILD_WITH_481=1) due to
malloc_usable_size() reporting a better estimate (similar to
https://reviews.facebook.net/D43317 ). This patch grows the write buffer size
slightly to compensate for this.

Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make listener_test && ./listener_test

Reviewers: rven, anthony, yhchiang, igor, sdong

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45921
2015-08-31 23:11:12 -07:00
Yueh-Hsuan Chiang
40cd91b7fe Fixed compile warning in rocksdbjava
Summary:
Fixed the following compile warning in rocksdbjava:

    java/rocksjni/comparatorjnicallback.cc:20:14: warning: unused variable ‘rs’ [-Wunused-variable]
       const jint rs = env->GetJavaVM(&m_jvm);
                  ^
    java/rocksjni/comparatorjnicallback.cc: In member function ‘JNIEnv* rocksdb::BaseComparatorJniCallback::getJniEnv() const’:
    java/rocksjni/comparatorjnicallback.cc:45:8: warning: unused variable ‘rs’ [-Wunused-variable]
       jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL);
            ^
    java/rocksjni/loggerjnicallback.cc: In constructor ‘rocksdb::LoggerJniCallback::LoggerJniCallback(JNIEnv*, jobject)’:
    java/rocksjni/loggerjnicallback.cc:19:14: warning: unused variable ‘rs’ [-Wunused-variable]
       const jint rs = env->GetJavaVM(&m_jvm);
                  ^
    java/rocksjni/loggerjnicallback.cc: In member function ‘JNIEnv* rocksdb::LoggerJniCallback::getJniEnv() const’:
    java/rocksjni/loggerjnicallback.cc:33:8: warning: unused variable ‘rs’ [-Wunused-variable]
       jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL);
            ^

Test Plan: make rocksdbjava -j32

Reviewers: sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45891
2015-08-31 18:36:17 -07:00
Yueh-Hsuan Chiang
90415cfebe Fixed a compile warning in linux32 environment.
Summary:
Fixed the following compile warning in linux32 environment.

    ==> linux32: util/sst_dump_tool.cc: In member function ‘int
                 rocksdb::SstFileReader::ShowAllCompressionSizes(size_t)’:
    ==> linux32: util/sst_dump_tool.cc:167:50: warning: format ‘%lu’ expects
                 argument of type ‘long unsigned int’, but argument 3 has type
                 ‘size_t {aka unsigned int}’ [-Wformat=]
    ==> linux32:    fprintf(stdout, "Block Size: %lu\n", block_size);

Test Plan: make sst_dump

Reviewers: anthony, IslamAbdelRahman, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45885
2015-08-31 18:35:12 -07:00
sdong
9d6503f88d Fix arena_test test break using glibc-2.17
Summary: arena_test is failing with glibc-2.17. Make it more robust

Test Plan: Run arena_test using both of glibc-2.17 and 2.2 and make sure both passes.

Reviewers: yhchiang, rven, IslamAbdelRahman, kradhakrishnan, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D45879
2015-08-31 16:53:23 -07:00
agiardullo
77a28615ec Support static Status messages
Summary: Provide a way to specify a detailed static error message for a Status without incurring a memcpy.  Let me know what people think of this approach.

Test Plan: added simple test

Reviewers: igor, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D44259
2015-08-31 16:13:29 -07:00
agiardullo
18db1e4695 better db_bench options for transactions
Summary:
Pessimistic Transaction expiration time checking currently causes a performace regression,  Lets disable it in db_bench by default.

Also, in order to be able to better tune how much contention we're simulating, added new optinos to set lock timeout and snapshot.

Test Plan: run db_bench randomtranansaction

Reviewers: sdong, igor, yhchiang, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45831
2015-08-31 15:56:07 -07:00
Igor Canadi
0be260523c Merge pull request #702 from PraveenSinghRao/remove_spurious
remove spurious compression definitions
2015-08-31 13:48:26 -07:00
Ari Ekmekji
8b689546b6 Add Subcompactions to Universal Compaction Unit Tests
Summary:
Now that the approach to parallelizing L0-L1 level-based
compactions by breaking the compaction job into subcompactions is
being extended to apply to universal compactions as well, the unit
tests need to account for this and run the universal compaction
tests with subcompactions both enabled and disabled.

Test Plan: make all && make check

Reviewers: sdong, igor, noetzli, anthony, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D45657
2015-08-31 12:59:02 -07:00
krad
c6d870ffb0 Merge branch 'arcpatch-D45741' 2015-08-31 12:42:56 -07:00
krad
57b3a87731 Adding sandcastle determinator for RocksDB
Summary: This fuels commands to be executed for different sandcastle jobs. This is a nice way to separate RocksDB specific commands from Sandcastle specific job definition.

Test Plan: None. Will be tested when we add Sandcastle job.

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45741
2015-08-31 12:37:26 -07:00
sdong
3d78eb66bb Arena usage to be calculated using malloc_usable_size()
Summary: malloc_usable_size() gets a better estimation of memory usage. It is already used to calculate block cache memory usage. Use it in arena too.

Test Plan: Run all unit tests

Reviewers: anthony, kradhakrishnan, rven, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D43317
2015-08-31 09:39:27 -07:00
Andres Noetzli
effd9dd1e1 Fix deadlock in WAL sync
Summary:
MarkLogsSynced() was doing `logs_.erase(it++);`. The standard is saying:

```
all iterators and references are invalidated, unless the erased members are at an end (front or back) of the deque (in which case only iterators and references to the erased members are invalidated)
```

Because `it` is an iterator to the first element of the container, it is
invalidated, only one iteration is executed and `log.getting_synced = false;`
is not being done, so `while (logs_.front().getting_synced)` in `WriteImpl()`
is not terminating.

Test Plan: make db_bench && ./db_bench --benchmarks=fillsync

Reviewers: igor, rven, IslamAbdelRahman, anthony, kradhakrishnan, yhchiang, sdong, tnovak

Reviewed By: tnovak

Subscribers: kolmike, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45807
2015-08-28 18:06:32 -07:00
Praveen Rao
64f07deb88 remove spurious compression definitions 2015-08-28 11:17:02 -07:00
Andres Noetzli
72a9b73c9e Removed unnecessary checks in DBTest.ApproximateMemoryUsage
Summary:
Just realized that after D45675, part of the code in
DBTest.ApproximateMemoryUsage, does not really test anything anymore, so I
removed it.

Test Plan: make clean all check

Reviewers: rven, igor, sdong, anthony, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45783
2015-08-28 11:13:20 -07:00
Venkatesh Radhakrishnan
cb164bfc48 Do not delete iterators for immutable memtables.
Summary:
The immutable memtable iterators are allocated from an arena and there
is no benefit from deleting these. Also the immutable memtables
themselves will continue to be in memory until the version set
containing it is alive. We will not remove immutable memtable iterators
over the upper bound. We now add immutable iterators to the test.

Test Plan: db_tailing_iter_test.TailingIteratorTrimSeekToNext

Reviewers: tnovak, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45597
2015-08-28 11:07:07 -07:00
sdong
7a0dbdf3ac Add ZSTD (not final format) compression type
Summary: Add ZSTD compression type. The same way as adding LZ4.

Test Plan: run all tests. Generate files in db_bench. Make sure reads succeed. But the SST files cannot be opened in older versions. Also some other adhoc tests.

Reviewers: rven, anthony, IslamAbdelRahman, kradhakrishnan, igor

Reviewed By: igor

Subscribers: MarkCallaghan, maykov, yoshinorim, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D45747
2015-08-28 11:01:13 -07:00
Igor Canadi
e2db15efde Merge pull request #701 from PraveenSinghRao/usewinapi_notcruntime
Remove usage of C runtime API that has file handle limitation
2015-08-27 17:13:55 -07:00
Andres Noetzli
e853191c17 Fix DBTest.ApproximateMemoryUsage
Summary:
This patch fixes two issues in DBTest.ApproximateMemoryUsage:
- It was possible that a flush happened between getting the two properties in
  Phase 1, resulting in different numbers for the properties and failing the
  assertion. This is fixed by waiting for the flush to finish before getting
  the properties.
- There was a similar issue in Phase 2 and additionally there was an issue that
  rocksdb.size-all-mem-tables was not monotonically increasing because it was
  possible that a flush happened just after getting the properties and then
  another flush just before getting the properties in the next round. In this
  situation, the reported memory usage decreased. This is fixed by forcing a
  flush before getting the properties.

Note: during testing, I found that kFlushesPerRound does not seem very
accurate. I added a TODO for this and it would be great to get some input on
what to do there.

Test Plan:
The first issue can be made more likely to trigger by inserting a
`usleep(10000);` between the calls to GetIntProperty() in Phase 1.
The second issue can be made more likely to trigger by inserting a
`if (r != 0) usleep(10000);` before the calls to GetIntProperty() and a
`usleep(10000);` after the calls.
Then execute make db_test && ./db_test --gtest_filter=DBTest.ApproximateMemoryUsage

Reviewers: rven, yhchiang, igor, sdong, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45675
2015-08-27 16:17:08 -07:00
Siying Dong
7c916a5d38 Merge pull request #699 from OpenChannelSSD/to_fb_master
Helper functions to support direct IO
2015-08-27 10:06:48 -07:00
Javier González
0886f4f66a Helper functions to support direct IO
Summary:
This patch adds the helper functions and variables to allow a backend
implementing WritableFile to support direct IO when persisting a
memtable.

Test Plan:
Since there is no upstream implementation of WritableFile supporting
direct IO, the new behavior is disabled.

Tests should be provided by the backend implementing WritableFile.
2015-08-27 08:36:56 +02:00
Praveen Rao
7e327980a3 Remove usage of C runtime API that has file handle limitation 2015-08-26 18:51:18 -07:00
Yueh-Hsuan Chiang
8ef0144e2f Add argument --show_table_properties to db_bench
Summary:
Add argument --show_table_properties to db_bench

  -show_table_properties (If true, then per-level table properties will be
    printed on every stats-interval when stats_interval is set and
    stats_per_interval is on.) type: bool default: false

Test Plan:
./db_bench --show_table_properties=1 --stats_interval=100000 --stats_per_interval=1
./db_bench --show_table_properties=1 --stats_interval=100000 --stats_per_interval=1 --num_column_families=2

Sample Output:

    Compaction Stats [column_family_name_000001]
    Level    Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt)  KeyIn KeyDrop
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
      L0      3/0          5   0.8      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0     86.3         0        17    0.021          0       0      0
      L1      5/0          9   0.9      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000          0       0      0
      L2      9/0         16   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000          0       0      0
     Sum     17/0         31   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     86.3         0        17    0.021          0       0      0
     Int      0/0          0   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     83.9         0         2    0.022          0       0      0
    Flush(GB): cumulative 0.030, interval 0.004
    Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard

    Level[0]: # data blocks=2571; # entries=84813; raw key size=2035512; raw average key size=24.000000; raw value size=8481300; raw average value size=100.000000; data block size=5690119; index block size=82415; filter block size=0; (estimated) table size=5772534; filter policy name=N/A;
    Level[1]: # data blocks=4285; # entries=141355; raw key size=3392520; raw average key size=24.000000; raw value size=14135500; raw average value size=100.000000; data block size=9487353; index block size=137377; filter block size=0; (estimated) table size=9624730; filter policy name=N/A;
    Level[2]: # data blocks=7713; # entries=254439; raw key size=6106536; raw average key size=24.000000; raw value size=25443900; raw average value size=100.000000; data block size=17077893; index block size=247269; filter block size=0; (estimated) table size=17325162; filter policy name=N/A;
    Level[3]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A;
    Level[4]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A;
    Level[5]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A;
    Level[6]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A;

Reviewers: anthony, IslamAbdelRahman, MarkCallaghan, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45651
2015-08-26 18:27:23 -07:00
Yueh-Hsuan Chiang
1fb2abae2d ColumnFamilyOptions serialization / deserialization.
Summary:
This patch adds GetStringFromColumnFamilyOptions(), the inverse function
of the existing GetColumnFamilyOptionsFromString(), and improves
the implementation of GetColumnFamilyOptionsFromString().

Test Plan: Add a test in options_test.cc

Reviewers: igor, sdong, anthony, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: noetzli, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45009
2015-08-26 16:13:56 -07:00
Igor Canadi
5f4166c90e ReadaheadRandomAccessFile -- userspace readahead
Summary:
ReadaheadRandomAccessFile acts as a transparent layer on top of RandomAccessFile. When a Read() request is issued, it issues a much bigger request to the OS and caches the result. When a new request comes in and we already have the data cached, it doesn't have to issue any requests to the OS.

We add ReadaheadRandomAccessFile layer only when file is read during compactions.

D45105 was incorrectly closed by Phabricator because I committed it to a separate branch (not master), so I'm resubmitting the diff.

Test Plan: make check

Reviewers: MarkCallaghan, sdong

Reviewed By: sdong

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D45123
2015-08-26 15:25:59 -07:00
Igor Canadi
16ebe3a2a9 Mmap reads should not return error if reading past file
Summary:
Currently, mmap returns IOError when user tries to read data past the end of the file. This diff changes the behavior. Now, we return just the bytes that we can, and report the size we returned via a Slice result. This is consistent with non-mmap behavior and also pread() system call.

This diff is taken out of D45123.

Test Plan: make check

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45645
2015-08-26 14:51:38 -07:00
sdong
d286b5df90 DBIter to out extra keys with higher sequence numbers when changing direction from forward to backward
Summary:
When DBIter changes iterating direction from forward to backward, it might see some much larger keys with higher sequence ID. With this commit, these rows will be actively filtered out. It should fix existing disabled tests in db_iter_test.

This may not be a perfect fix, but it introduces least impact on existing codes, in order to be safe.

Test Plan:
Enable existing tests and make sure they pass. Add a new test DBIterWithMergeIterTest.InnerMergeIteratorDataRace8.
Also run all existing tests.

Reviewers: yhchiang, rven, anthony, IslamAbdelRahman, kradhakrishnan, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D45567
2015-08-26 13:01:39 -07:00
Andres Noetzli
3795449c9d Fix DBTest.GetProperty
Summary:
DBTest.GetProperty was failing occasionally (see task #8131266). The reason was
that the test closed the database before the compaction was done. When the test
reopened the database, RocksDB would schedule a compaction which in turn
created table readers and lead the test to fail the assertion that
rocksdb.estimate-table-readers-mem is 0. In most cases, GetIntProperty() of
rocksdb.estimate-table-readers-mem happened before the compaction created the
table readers, hiding the problem. This patch changes the
WaitForFlushMemTable() to WaitForCompact(). WaitForFlushMemTable() is not
necessary because it is already being called a couple of lines before without
any insertions in-between.

Test Plan:
Insert `usleep(10000);` just after `Reopen(options);` on line 2333 to make the issue more likely, then run:
make db_test && while ./db_test --gtest_filter=DBTest.GetProperty; do true; done

Reviewers: rven, yhchiang, anthony, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45603
2015-08-26 10:10:26 -07:00
Igor Canadi
a7834a1292 Merge pull request #698 from yuslepukhin/address_noexcept_windows
Address noexcept and const integer lambda capture on win
2015-08-25 17:15:23 -07:00