Commit Graph

1836 Commits

Author SHA1 Message Date
Haobo Xu
0f0076ed5a [RocksDB] Reduce memory footprint of the blockbased table hash index.
Summary:
Currently, the in-memory hash index of blockbased table uses a precise hash map to track the prefix to block range mapping. In some use cases, especially when prefix itself is big, the memory overhead becomes a problem. This diff introduces a fixed hash bucket array that does not store the prefix and allows prefix collision, which is similar to the plaintable hash index, in order to reduce the memory consumption.
Just a quick draft, still testing and refining.

Test Plan: unit test and shadow testing

Reviewers: dhruba, kailiu, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D19047
2014-06-18 18:16:07 -07:00
Igor Canadi
3525aac9e5 Change order of parameters in adaptive table factory
Summary:
This is minor, but if we put the writing talbe factory as the third parameter, when we add a new table format, we'll have a situation:
1) block based factory
2) plain table factory
3) output factory
4) new format factory

I think it makes more sense to have output as the first parameter.

Also, fixed a NewAdaptiveTableFactory() call in unit test

Test Plan: unit test

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D19119
2014-06-18 07:04:37 +02:00
sdong
8c265c08f1 HashLinkList to log distribution of number of entries aross buckets
Summary: Add two parameters of hash linked list to log distribution of number of entries across all buckets, and a sample row when there are too many entries in one single bucket.

Test Plan: Turn it on in plain_table_db_test and see the logs.

Reviewers: haobo, ljin

Reviewed By: ljin

Subscribers: leveldb, nkg-, dhruba, yhchiang

Differential Revision: https://reviews.facebook.net/D19095
2014-06-17 17:55:36 -07:00
Yueh-Hsuan Chiang
4bff7a8a87 Merge pull request #177 from nanwu/master
specify the command to install build_tools/mac-install-gflags.sh file in...
2014-06-17 15:52:37 -07:00
nawu
b982e65f8b specify the command to install build_tools/mac-install-gflags.sh file in doc 2014-06-17 17:03:21 -05:00
sdong
200e4b4a72 Add a table factory that can read DB with both of PlainTable and BlockBasedTable in it
Summary: The new table factory is used if users want to convert a DB from one table format to the other. A user can use this table to open a DB written using one table format and write new files to another table format.

Test Plan: add a unit test

Reviewers: haobo, igor

Reviewed By: igor

Subscribers: dhruba, ljin, yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D19017
2014-06-17 11:49:22 -07:00
Igor Canadi
4f18bfe376 Merge pull request #176 from bgrainger/mutexrw-unlock
Add separate Read/WriteUnlock methods in MutexRW.
2014-06-17 20:38:06 +02:00
Yueh-Hsuan Chiang
e6e259b8ab Include max_write_buffer_number >= 2 to SanitizeOptions.
Summary: Include max_write_buffer_number >= 2 to SanitizeOptions.

Test Plan: make all check

Reviewers: haobo, sdong, igor, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D19077
2014-06-16 16:26:46 -07:00
sdong
cadc1adffa Refactor: group metadata needed to open an SST file to a separate copyable struct
Summary:
We added multiple fields to FileMetaData recently and are planning to add more.
This refactoring separate the minimum information for accessing the file. This object is copyable (FileMetaData is not copyable since the ref counter). I hope this refactoring can enable further improvements:

(1) use it to design a more efficient data structure to speed up read queries.
(2) in the future, when we add information of storage level, we can easily do the encoding, instead of enlarge this structure, which might expand memory work set for file meta data.

The definition is same as current EncodedFileMetaData used in two level iterator, so now the logic in two level iterator is easier to understand.

Test Plan: make all check

Reviewers: haobo, igor, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba, yhchiang

Differential Revision: https://reviews.facebook.net/D18933
2014-06-16 16:10:52 -07:00
Bradley Grainger
2d02ec6533 Add separate Read/WriteUnlock methods in MutexRW.
Some platforms, particularly Windows, do not have a single method that can
release both a held reader lock and a held writer lock; instead, a
separate method (ReleaseSRWLockShared or ReleaseSRWLockExclusive) must be
called in each case.

This may also be necessary to back MutexRW with a shared_mutex in C++14;
the current language proposal includes both an unlock() and a
shared_unlock() method.
2014-06-16 15:41:46 -07:00
Yueh-Hsuan Chiang
4d913cfbc3 Fix a bug causing LOG is not created when max_log_file_size is set.
Summary:
Fix a bug causing LOG is not created when max_log_file_size is set.
This bug is reported in issue #174.

Test Plan:
Add TEST(AutoRollLoggerTest, LogFileExistence).
make auto_roll_logger_test
./auto_roll_logger_test

Reviewers: haobo, sdong, ljin, igor, igor2

Reviewed By: igor2

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D19053
2014-06-16 10:27:42 -07:00
sdong
983c93d731 VersionSet::Get(): Bring back the logic of skipping key range check when there are <=3 level 0 files
Summary:
https://reviews.facebook.net/D17205 removed the logic of skipping file key range check when there are less than 3 level 0 files. This patch brings it back.

Other than that, add another small optimization to avoid to check all the levels if most higher levels don't have any file.

Test Plan: make all check

Reviewers: ljin

Reviewed By: ljin

Subscribers: yhchiang, igor, haobo, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D19035
2014-06-13 15:51:44 -07:00
Barnaby
a52a4e0952 Update README.md
Moved doc/index.html to wiki
2014-06-13 14:11:10 -07:00
sdong
9202d9b625 Fix sst_dump for PlainTable
Summary: sst_dump now doesn't work well for PlainTable. Not sure when it started, but this should fix it.

Test Plan: Run sst_dump against a file that used to fail.

Reviewers: yhchiang, haobo, igor

Reviewed By: igor

Subscribers: dhruba, ljin, leveldb

Differential Revision: https://reviews.facebook.net/D19023
2014-06-12 11:03:03 -07:00
Lei Jin
c83b085770 prefetch bloom filter data block for L0 files
Summary: as title

Test Plan:
db_bench
the initial result is very promising. I will post results of complete
runs

Reviewers: dhruba, haobo, sdong, igor

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18867
2014-06-12 10:06:18 -07:00
Lei Jin
578cf84ddf give correct metric name for grep in regression script
Summary: as title

Test Plan: jenkin

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D19005
2014-06-10 16:32:09 -07:00
Lei Jin
c4b3817d7c fix regression test
Summary: as title

Test Plan: push

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18999
2014-06-10 11:24:24 -07:00
sdong
88a1691a1e BlockBasedTable::PrefixMayMatch() to bloom setting to the beginning of the function
Summary: In BlockBasedTable::PrefixMayMatch() we calculate prefix even if bloom is not config. Move the check before

Test Plan: make all check

Reviewers: igor, ljin

Reviewed By: ljin

Subscribers: wuj, leveldb, haobo, yhchiang, dhruba

Differential Revision: https://reviews.facebook.net/D18993
2014-06-10 11:14:22 -07:00
Lei Jin
e2d3101cf1 collect metrics for in memory workload get/seek
Summary:
collect in-memory workload get/seek metrics so that we can alert on
regression

Test Plan: ran locally

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18969
2014-06-10 09:59:16 -07:00
Lei Jin
77db08f27b fix forward iterator bug
Summary: obvious

Test Plan: db_test

Reviewers: sdong, haobo, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18987
2014-06-10 09:57:26 -07:00
sdong
80f409ea37 Clean PlainTableReader's variables for better data locality
Summary:
Clean PlainTableReader's data structures:
(1) inline bloom_ (in order to do this, change DynamicBloom to allow lazy initialization)
(2) remove some variables only used when initialization from the class
(3) put variables not used in normal read code paths to the end of the class and reference prefix_extractor directly
(4) make Options a reference.

Test Plan: make all check

Reviewers: haobo, ljin

Reviewed By: ljin

Subscribers: igor, yhchiang, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D18891
2014-06-09 13:53:39 -07:00
Igor Canadi
f43c8262c2 Don't compress block bigger than 2GB
Summary: This is a temporary solution to a issue that we have with compression libraries. See task #4453446.

Test Plan: make check doesn't complain :)

Reviewers: haobo, ljin, yhchiang, dhruba, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18975
2014-06-09 12:26:09 -07:00
sdong
ee5a51e6ce sst_dump: still try to print out table properties even if failing to read the file
Summary: Even if the file is corrupted, table properties are usually available to print out. Now sst_dump would just fail without printing table properties. With this patch, table properties are still try to be printed out.

Test Plan: run sst_dump against multiple scenarios

Reviewers: igor, yhchiang, ljin, haobo

Reviewed By: haobo

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D18981
2014-06-09 11:21:44 -07:00
Yueh-Hsuan Chiang
3e701a654f [Java] Improve documentation for RocksEnv and its C++ resource.
Summary:
Improve documentation for RocksEnv and its C++ resource.  Specifically,
the result of RocksEnv::Default() is a singleton, and the ownership
of its c++ resource belongs to rocksdb c++.  As a result, calling
dispose() of the return value of RocksDB::Default() will be no-op.

Test Plan: no code change.

Reviewers: haobo, ankgup87

Reviewed By: ankgup87

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18945
2014-06-07 17:27:03 -07:00
Igor Canadi
0365eaf12e remove unnecessary printf 2014-06-06 18:27:44 -07:00
Igor Canadi
a0191c9dfe Create Missing Column Families
Summary: Provide an convenience option to create column families if they are missing from the DB. Task #4460490

Test Plan: added unit test. also, stress test for some time

Reviewers: sdong, haobo, dhruba, ljin, yhchiang

Reviewed By: yhchiang

Subscribers: yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D18951
2014-06-06 18:04:56 -07:00
Igor Canadi
99d3eed2fd Write Fast-path for single column family
Summary: We have a perf regression of Write() even with one column family. Make fast path for single column family to avoid the perf regression. See task #4455480

Test Plan: make check

Reviewers: sdong, ljin

Reviewed By: sdong, ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18963
2014-06-06 17:26:23 -07:00
Yueh-Hsuan Chiang
e72b02e3c2 [Java] Add basic Java binding for rocksdb::Env.
Summary: Add basic Java binding for rocksdb::Env.

Test Plan:
make rocksdbjava
make jtest
cd java
./jdb_bench.sh --max_background_compactions=1
./jdb_bench.sh --max_background_compactions=10

Reviewers: sdong, ankgup87, haobo

Reviewed By: haobo

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18903
2014-06-05 17:09:25 -07:00
sdong
b92a19a431 sst_dump: Set dummy prefix extractor for binary search index in block based table
Summary: Now sst_dump fails in block based tables if binary search index is used, as it requires a prefix extractor. Add it.

Test Plan: Run it against such a file to make sure it fixes the problem.

Reviewers: yhchiang, kailiu

Reviewed By: kailiu

Subscribers: ljin, igor, dhruba, haobo, leveldb

Differential Revision: https://reviews.facebook.net/D18927
2014-06-05 15:37:23 -07:00
Igor Canadi
5d870717ae Correctly preallocate files in universal compaction
Summary: In universal compaction, MaxFileSizeForLevel is ULLONG_MAX. We've been preallocation files to UULONG_MAX size all these time :)

Test Plan: make check

Reviewers: dhruba, haobo, ljin, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18915
2014-06-05 13:19:35 -07:00
Yueh-Hsuan Chiang
166cc5b456 Merge pull request #157 from ankgup87/master
[Java] Add RestoreBackupableDB
2014-06-05 09:24:55 -07:00
Ankit Gupta
1a6c1a5ddd Fix build 2014-06-05 13:34:38 +01:00
Ankit Gupta
2fa0a993b8 Fix build 2014-06-05 13:25:08 +01:00
Ankit Gupta
08314321fa Merge branch 'master' of https://github.com/facebook/rocksdb 2014-06-05 13:17:53 +01:00
Igor Canadi
457bae6911 Fix regression test
Summary:
388d2054c7
added extra line to db_bench output, breaking regression tests. This diff makes it more robust and fixes the issue

Test Plan: ran it

Reviewers: ljin, sdong

Reviewed By: sdong

Subscribers: sdong, leveldb

Differential Revision: https://reviews.facebook.net/D18897
2014-06-04 09:59:44 -07:00
Igor Canadi
552c49f0f4 Remove upper bound for rate limiting unit test 2014-06-03 13:58:44 -07:00
Igor Canadi
fd27001072 Fix compile errors on Mac
Summary: https://phabricator.fb.com/P11372644

Test Plan: compiles

Reviewers: sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18873
2014-06-03 12:28:58 -07:00
sdong
df9069d23f In DB::NewIterator(), try to allocate the whole iterator tree in an arena
Summary:
In this patch, try to allocate the whole iterator tree starting from DBIter from an arena
1. ArenaWrappedDBIter is created when serves as the entry point of an iterator tree, with an arena in it.
2. Add an option to create iterator from arena for following iterators: DBIter, MergingIterator, MemtableIterator, all mem table's iterators, all table reader's iterators and two level iterator.
3. MergeIteratorBuilder is created to incrementally build the tree of internal iterators. It is passed to mem table list and version set and add iterators to it.

Limitations:
(1) Only DB::NewIterator() without tailing uses the arena. Other cases, including readonly DB and compactions are still from malloc
(2) Two level iterator itself is allocated in arena, but not iterators inside it.

Test Plan: make all check

Reviewers: ljin, haobo

Reviewed By: haobo

Subscribers: leveldb, dhruba, yhchiang, igor

Differential Revision: https://reviews.facebook.net/D18513
2014-06-02 17:44:57 -07:00
sdong
462796697c dynamic_bloom: replace some divide (remainder) operations with shifts in locality mode, and other improvements
Summary:
This patch changes meaning of options.bloom_locality: 0 means disable cache line optimization and any positive number means use CACHE_LINE_SIZE as block size (the previous behavior is the block size will be CACHE_LINE_SIZE*options.bloom_locality). By doing it, the divide operations inside a block can be replaced by a shift.

Performance is improved:
https://reviews.facebook.net/P471

Also, improve the basic algorithm in two ways:
(1) make sure num of blocks is an odd number
(2) rotate bytes after every probe in locality mode. Since the divider is 2^n, unless doing it, we are never able to use all the bits.
Improvements of false positive: https://reviews.facebook.net/P459

Test Plan: make all check

Reviewers: ljin, haobo

Reviewed By: haobo

Subscribers: dhruba, yhchiang, igor, leveldb

Differential Revision: https://reviews.facebook.net/D18843
2014-06-02 17:36:38 -07:00
Igor Canadi
91ddd587cc Only signal cond variable if need to
Summary:
At the end of BackgroundCallCompaction(), we call SignalAll(), even though we don't need to. If compaction hasn't done anything and there's another compaction running, there is no need to signal on the condition variable. Doing so creates a tight feedback loop which results in log files like:

   wait for memtable flush
   compaction nothing to do
   wait for memtable flush
   compaction nothing to do

This change eliminates that

Test Plan:
make check
Also:

    icanadi@dev1440 ~ $ grep "nothing to do" /fast-rocksdb-tmp/rocksdb_test/column_family_test/LOG | wc -l
    7435
    icanadi@dev1440 ~ $ grep "nothing to do" /fast-rocksdb-tmp/rocksdb_test/column_family_test/LOG | wc -l
    372

First version is before the change, second version is after the change.

Reviewers: dhruba, ljin, haobo, yhchiang, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18855
2014-06-02 17:23:55 -07:00
Igor Canadi
8cb7ad83c3 Flush stale column families less aggressively
Summary:
We've seen some production issues where column family is detected as stale, although there is only one column family in the system. This is a quick fix that:
1) doesn't flush stale column families if there's only one of them
2) Use 4 as a coefficient instead of 2 for determening when a column family is stale. This will make flushing less aggressive, while still keep a nice dynamic flushing of very stale CFs.

Test Plan: make check

Reviewers: dhruba, haobo, ljin, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18861
2014-06-02 15:33:54 -07:00
sdong
593bb2c40b db_stress to add an option to periodically change background thread pool size.
Summary: Add an option to indicates the variation of background threads. If the flag is not 0, for every 100 milliseconds, adjust thread pool size to a value within the range.

Test Plan: run db_stress

Reviewers: haobo, dhruba, igor, ljin

Reviewed By: ljin

Subscribers: leveldb, nkg-

Differential Revision: https://reviews.facebook.net/D18759
2014-06-02 10:32:09 -07:00
Lei Jin
388d2054c7 forward iterator
Summary:
Forward iterator puts everything together in a flat structure instead of
a hierarchy of nested iterators. this should simplify the code and
provide better performance. It also enables more optimization since all
information are accessiable in one place.
Init evaluation shows about 6% improvement

Test Plan: db_test and db_bench

Reviewers: dhruba, igor, tnovak, sdong, haobo

Reviewed By: haobo

Subscribers: sdong, leveldb

Differential Revision: https://reviews.facebook.net/D18795
2014-05-30 14:31:55 -07:00
Lei Jin
f29c62fc6f add an iterator refresh option for SeekRandom
Summary: One more option to allow iterator refreshing when using normal iterator

Test Plan: ran db_bench

Reviewers: haobo, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18849
2014-05-30 14:09:22 -07:00
Ankit Gupta
fa1b62cabd Merge remote-tracking branch 'upstream/master' 2014-05-29 18:33:11 -07:00
sdong
9899b12780 ThreadID printed when Thread terminating in the same format as posix_logger
Summary: 220132b65e correctly fixed the issue of thread ID printing when terminating a thread. Nothing wrong with it. This diff prints the ID in the same way as in PosixLogger::logv() so that users can be more easily to correlates them.

Test Plan: run env_test and make sure it prints correctly.

Reviewers: igor, haobo, ljin, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D18819
2014-05-29 11:11:08 -07:00
Yueh-Hsuan Chiang
ab3e566699 [Java] Generalize dis-own native handle and refine dispose framework.
Summary:
1. Move disOwnNativeHandle() function from RocksDB to RocksObject
to allow other RocksObject to use disOwnNativeHandle() when its
ownership of native handle has been transferred.

2. RocksObject now has an abstract implementation of dispose(),
which does the following two things.  First, it checks whether
both isOwningNativeHandle() and isInitialized() return true.
If so, it will call the protected abstract function dispose0(),
which all the subclasses of RocksObject should implement.  Second,
it sets nativeHandle_ = 0.  This redesign ensure all subclasses
of RocksObject have the same dispose behavior.

3. All subclasses of RocksObject now should implement dispose0()
instead of dispose(), and dispose0() will be called only when
isInitialized() returns true.

Test Plan:
make rocksdbjava
make jtest

Reviewers: dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett, haobo

Reviewed By: haobo

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18801
2014-05-28 18:16:29 -07:00
Yueh-Hsuan Chiang
663627abf3 Merge pull request #153 from bpot/fix_jni_segfault
Segfault when using BackupableDB with Java bindings
2014-05-27 14:07:22 -07:00
Yueh-Hsuan Chiang
8e41e54e1b [Java] Makes DbBenchmark takes 0 and 1 as boolean values.
Summary:
Originally, the boolean arguments in Java DB benchmark only takes
true and false as valid input.  This diff allows boolean arguments
to be set using 0 or 1.

Test Plan:
make rocksdbjava
make jdb_bench
java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=1
java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=0
java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=1

Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett

Reviewed By: haobo

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18687
2014-05-27 13:52:44 -07:00
Ankit Gupta
84cbafa5ce Merge branch 'master' of https://github.com/facebook/rocksdb 2014-05-27 13:04:16 -07:00