Commit Graph

188 Commits

Author SHA1 Message Date
Jonah Cohen
c1a924b9f0 Move convenience.h to /include
Summary: Move header file so it can be referenced externally.

Test Plan: Rebuild.

Reviewers: ljin

Reviewed By: ljin

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28095
2014-10-31 12:08:43 -07:00
Igor Canadi
9f7fc3ac45 Turn on -Wshadow
Summary:
...and fix all the errors :)

Jim suggested turning on -Wshadow because it helped him fix number of critical bugs in fbcode. I think it's a good idea to be -Wshadow clean.

Test Plan: compiles

Reviewers: yhchiang, rven, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27711
2014-10-31 11:59:54 -07:00
sdong
4d2ba38b65 Make VersionBuilder unit testable
Summary:
Rename Version::Builder to VersionBuilder and expose its definition to a header.
Make VerisonBuilder not reference Version or ColumnFamilyData, only working with VersionStorageInfo.
Add version_builder_test which has a simple test.

Test Plan: make all check

Reviewers: rven, yhchiang, igor, ljin

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27969
2014-10-31 10:44:06 -07:00
Yueh-Hsuan Chiang
37e9b63701 Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h
Summary: Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27885
2014-10-29 17:57:00 -07:00
Yueh-Hsuan Chiang
6afafa3694 Apply InfoLogLevel to the logs in utilities/merge_operators/uint64add.cc
Summary:
Apply InfoLogLevel to the logs and add missing copy-right information
to  utilities/merge_operators/uint64add.cc.

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27897
2014-10-29 17:10:21 -07:00
sdong
76d1c28e82 Make CompactionPicker more easily tested
Summary:
Make compaction picker easier to test.
The basic idea is to separate a minimum subcomponent of Version to VersionStorageInfo, which just responsible to LSM tree. A stub VersionStorageInfo can then be easily created and passed into compaction picker so that we can check the outputs.

It now passes most tests. Still two things need to be done:
(1) deal with the FIFO compaction's file size.
(2) write an example test to make sure the interface can do the job.

Add a compaction_picker_test to make sure compaction picker codes can be easily unit tested.

Test Plan:
Pass all unit tests and compaction_picker_test

Reviewers: yhchiang, rven, igor, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27639
2014-10-29 15:16:53 -07:00
Lei Jin
834c67d77f rename FileLevel to LevelFilesBrief / unfriend CompactedDBImpl
Summary:
We have several different types of data structures for file information.
FileLevel is kinda of confusing since it only contains file range and
fd. Rename it to LevelFilesBrief to make it clear.
Unfriend CompactedDBImpl as a by product

Test Plan:
make release / make all
will run full test with all stacked diffs

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27585
2014-10-28 10:03:13 -07:00
sdong
5cc9adf5ba WriteBatchWithIndex's Iterator bug of SeekToFirst() and SeekToLast()
Summary: WriteBatchWithIndex's iterator's SeekToFirst() and SeekToLast() use offset=0 to indicate it is smaller than all the keys, which is wrong. offset=0 will decode a key "" (the header decodes like that). It could be larger than other keys in non-default comparators. Fix it by using a special flag of offset to indicate searching to the beginning of the CF.

Test Plan: Add a unit test that used to fail. Also, add some more tests to related cases, though they don't fail for now.

Reviewers: igor

Reviewed By: igor

Subscribers: rven, yhchiang, ljin, leveldb

Differential Revision: https://reviews.facebook.net/D24873
2014-10-15 07:33:32 -07:00
Igor Canadi
833357402c WriteBatchWithIndex supports an iterator that merge its change with a base iterator.
Summary: Add an iterator that combines base_iterator of type Iterator* with delta iterator of type WBWIIterator*.

Test Plan: nothing yet. work in progress

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: rven, yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D24741
2014-10-10 19:02:58 -07:00
sdong
4f65fbd197 WriteBatchWithIndex's iterator to support SeekToFirst(), SeekToLast() and Prev()
Summary: Support SeekToFirst(), SeekToLast() and Prev() in WBWIIterator, returned by WriteBatchWithIndex::NewIterator().

Test Plan: Write unit test cases to cover the case.

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: rven, yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D24765
2014-10-10 16:19:34 -07:00
sdong
f441b273ae WriteBatchWithIndex to support an option to overwrite rows when operating the same key
Summary: With a new option, when accepting a new key, WriteBatchWithIndex will find an existing index of the same key, and replace the content of it.

Test Plan: Add a unit test case.

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24753
2014-10-10 15:19:21 -07:00
Lei Jin
cd0d581ff5 convert Options from string
Summary: Allow accepting Options as a string of key/value pairs

Test Plan: unit test

Reviewers: yhchiang, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 10:00:12 -07:00
Igor Canadi
63eade401a Fix error introduced by merge 2014-10-07 09:50:29 -07:00
Nik Bougalis
b87db07152 Avoid dereferencing a null field 2014-10-07 01:40:29 -07:00
Igor Canadi
1a1b953471 Merge pull request #270 from tdfischer/check-with-unity
Build unity build on make check
2014-10-06 23:07:07 -07:00
fyrz
1c7c764761 Replaced obsolete comparator with builtin variant. 2014-10-05 21:51:09 +02:00
Danny Al-Gaaf
9d6f380866 backupable_db_test.cc: pass const string param by reference
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-10-01 11:57:01 +02:00
Danny Al-Gaaf
8ff0b40955 document_db_test.cc: pass const string param by reference
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-10-01 11:55:43 +02:00
Danny Al-Gaaf
177caca428 ttl/ttl_test.cc: pass const string param by reference
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-10-01 11:53:00 +02:00
Danny Al-Gaaf
86e29f0334 document_db.cc: remove unused variable
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-10-01 10:49:09 +02:00
Danny Al-Gaaf
e55aea5512 document_db.cc: fix assert
Check for lhs and not twice for rhs.

Fix for:

[utilities/document/document_db.cc:36] ->
 [utilities/document/document_db.cc:36]: (style) Same expression on both
 sides of '&&'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-09-30 23:30:32 +02:00
Danny Al-Gaaf
b140375565 ttl/ttl_test.cc: prefer prefix ++operator for non-primitive types
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-09-30 23:30:32 +02:00
Danny Al-Gaaf
43c789c8f2 spatialdb/spatial_db.cc: use !empty() instead of 'size() > 0'
Use empty() since it should be prefered as it has, following
the standard, a constant time complexity regardless of the
containter type. The same is not guaranteed for size().

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-09-30 23:30:32 +02:00
Danny Al-Gaaf
0de452ee9f document_db.cc: pass const parameter by reference
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-09-30 23:30:32 +02:00
Danny Al-Gaaf
873f1356a1 db_ttl_impl.h: pass func parameter by reference
Fix for:

[utilities/ttl/db_ttl_impl.h:209]: (performance) Function parameter
 'merge_op' should be passed by reference.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2014-09-30 23:30:32 +02:00
Lei Jin
fd5d80d55e CompactedDB: log using the correct info_log
Summary:
info_log from supplied Options can be nullptr. Using the one from
db_impl. Also call flush after that since no more loggging will happen
and LOG can contain partial output

Test Plan: verified with db_bench

Reviewers: igor, yhchiang, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24183
2014-09-29 12:45:04 -07:00
Lei Jin
2faf49d5f1 use GetContext to replace callback function pointer
Summary:
Intead of passing callback function pointer and its arg on Table::Get()
interface, passing GetContext. This makes the interface cleaner and
possible better perf. Also adding a fast pass for SaveValue()

Test Plan: make all check

Reviewers: igor, yhchiang, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24057
2014-09-29 11:09:09 -07:00
Lei Jin
fbd2dafc9f CompactedDBImpl::MultiGet() for better CuckooTable performance
Summary:
Add the MultiGet API to allow prefetching.
With file size of 1.5G, I configured it to have 0.9 hash ratio that can
fill With 115M keys and result in 2 hash functions, the lookup QPS is
~4.9M/s  vs. 3M/s for Get().
It is tricky to set the parameters right. Since files size is determined
by power-of-two factor, that means # of keys is fixed in each file. With
big file size (thus smaller # of files), we will have more chance to
waste lot of space in the last file - lower space utilization as a
result. Using smaller file size can improve the situation, but that
harms lookup speed.

Test Plan: db_bench

Reviewers: yhchiang, sdong, igor

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23673
2014-09-25 13:34:51 -07:00
Lei Jin
3c68006109 CompactedDBImpl
Summary:
Add a CompactedDBImpl that will enabled when calling OpenForReadOnly()
and the DB only has one level (>0) of files. As a performan comparison,
CuckooTable performs 2.1M/s with CompactedDBImpl vs. 1.78M/s with
ReadOnlyDBImpl.

Test Plan: db_bench

Reviewers: yhchiang, igor, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23553
2014-09-25 11:14:01 -07:00
sdong
d0de413f4d WriteBatchWithIndex to allow different Comparators for different column families
Summary:
Previously, one single column family is given to WriteBatchWithIndex to index keys for all column families. An extra map from column family ID to comparator is maintained which can override the default comparator given in the constructor. A WriteBatchWithIndex::SetComparatorForCF() is added for user to add comparators per column family.

Also move more codes into anonymous namespace.

Test Plan: Add a unit test

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb, yhchiang

Differential Revision: https://reviews.facebook.net/D23355
2014-09-22 13:47:39 -07:00
Torrie Fischer
ed9a2df8cc fix unity build 2014-09-19 16:10:33 -07:00
Yueh-Hsuan Chiang
035043559d Fixed a signed-unsigned comparison in spatial_db.cc -- issue #293
Summary:
Fixed a signed-unsigned comparison in spatial_db.cc

utilities/spatialdb/spatial_db.cc:542:38: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
cc1plus: all warnings being treated as errors
make: *** [utilities/spatialdb/spatial_db.o] Error 1

Test Plan:
make spatial_db_test
./spatial_db_test

Reviewers: ljin, sdong, reddragon, igor

Reviewed By: reddragon

Subscribers: reddragon, leveldb

Differential Revision: https://reviews.facebook.net/D23565
2014-09-18 10:57:20 -07:00
Igor Canadi
a52cecb56c Fix Mac compile 2014-09-09 18:42:35 -07:00
Xiaozheng Tie
6cc12860f0 Added a few statistics for BackupableDB
Summary:
Added the following statistics to BackupableDB:

1. Number of successful and failed backups in class BackupStatistics
2. Time taken to do a backup
3. Number of files in a backup

1 is implemented in the BackupStatistics class
2 and 3 are added in the BackupMeta and BackupInfo class

Test Plan:
1 can be tested using BackupStatistics::ToString(),
2 and 3 can be tested in the BackupInfo class

Reviewers: sdong, igor2, ljin, igor

Reviewed By: igor

Differential Revision: https://reviews.facebook.net/D22785
2014-09-09 13:44:42 -07:00
Igor Canadi
55114e7f40 Some updates for SpatialDB
Summary:
1. add db statistics
2. write out batch every millionth write

Test Plan: unit tests

Reviewers: ljin, sdong, yinwang

Reviewed By: yinwang

Differential Revision: https://reviews.facebook.net/D22755
2014-09-08 17:00:43 -07:00
Nik Bougalis
d40c1f742f Add missing break statement 2014-09-05 20:50:29 -07:00
liuhuahang
bb6ae0f80c fix more compile warnings
N/A

Change-Id: I5b6f9c70aea7d3f3489328834fed323d41106d9f
Signed-off-by: liuhuahang <liuhuahang@zerus.co>
2014-09-05 14:14:37 +08:00
Nik Bougalis
0cd0ec4fe0 Plug memory leak during index creation 2014-09-04 20:52:00 -07:00
Igor Canadi
a4816269f1 Relax backupable rate limiting test 2014-09-04 10:22:58 -07:00
liuhuahang
ef5b384729 fix a few compile warnings
1, const qualifiers on return types make no sense and will trigger a compile warning: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]

2, class HistogramImpl has virtual functions and thus should have a virtual destructor

3, with some toolchain, the macro __STDC_FORMAT_MACROS is predefined and thus should be checked before define

Change-Id: I69747a03bfae88671bfbb2637c80d17600159c99
Signed-off-by: liuhuahang <liuhuahang@zerus.co>
2014-09-04 23:06:23 +08:00
Igor Canadi
2a8faf797c Compact SpatialDB as we go, not at the end 2014-08-28 11:32:49 -07:00
Igor Canadi
1913ce27b9 more concurrent flushes in SpatialDB 2014-08-27 08:48:31 -07:00
Igor Canadi
808e809366 Adjust SpatialDB column family options 2014-08-27 07:56:10 -07:00
Igor Canadi
0c39f54dfb Use Vector memtable when bulk loading SpatialDB 2014-08-26 19:23:09 -07:00
Igor Canadi
ff6ec0eb17 Optimize SpatialDB
Summary:
Two things:
1. Use hash-based index for data column family
2. Use Get() instead of Iterator Seek() when DB is opened read-only

Test Plan: added read-only test in unit test

Reviewers: yinwang

Reviewed By: yinwang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22323
2014-08-25 17:50:18 -07:00
Lei Jin
384400128f move block based table related options BlockBasedTableOptions
Summary:
I will move compression related options in a separate diff since this
diff is already pretty lengthy.
I guess I will also need to change JNI accordingly :(

Test Plan: make all check

Reviewers: yhchiang, igor, sdong

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D21915
2014-08-25 14:22:05 -07:00
Igor Canadi
01cbdd2aae Optimize storage parameters for spatialDB
Summary: We need to start compression at level 1, while OptimizeForLevelComapaction() only sets up rocksdb to start compressing at level 2. I also adjusted some other things.

Test Plan: compiles

Reviewers: yinwang

Reviewed By: yinwang

Differential Revision: https://reviews.facebook.net/D22203
2014-08-20 11:14:01 -07:00
sdong
28b5c76004 WriteBatchWithIndex: a wrapper of WriteBatch, with a searchable index
Summary:
Add WriteBatchWithIndex so that a user can query data out of a WriteBatch, to support MongoDB's read-its-own-write.

WriteBatchWithIndex uses a skiplist to store the binary index. The index stores the offset of the entry in the write batch. When searching for a key, the key for the entry is read by read the entry from the write batch from the offset.

Define a new iterator class for querying data out of WriteBatchWithIndex. A user can create an iterator of the write batch for one column family, seek to a key and keep calling Next() to see next entries.

I will add more unit tests if people are OK about this API.

Test Plan:
make all check
Add unit tests.

Reviewers: yhchiang, igor, MarkCallaghan, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb, xjin

Differential Revision: https://reviews.facebook.net/D21381
2014-08-18 16:37:38 -07:00
miguelportilla
93e6b5e9d9 Changes to support unity build:
* Script for building the unity.cc file via Makefile
* Unity executable Makefile target for testing builds
* Source code changes to fix compilation of unity build
2014-08-11 13:22:47 -04:00
Igor Canadi
5e3d5c5f6e Simplify SpatialIndexCursor
Summary:
Since we have enough memory to hold all primary keys loaded from spatial index, it is better if we first load all of them (store them in unordered_set for deduplication) and then query on primary key column family one by one.

We need to dedup all IDs, so we'll end up storing all of them in memory even with the current approach.

Test Plan: ./spatial_db_test is happy

Reviewers: yinwang

Reviewed By: yinwang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D20949
2014-08-01 16:50:40 -04:00