Commit Graph

2307 Commits

Author SHA1 Message Date
Lei Jin
feadb9df53 fix cuckoo table builder test
Summary:
as title

Test Plan:
./cuckoo_table_builder_test

Reviewers:igor

CC:leveldb

Task ID: #

Blame Rev:
2014-09-17 15:48:33 -07:00
Igor Canadi
3c232e1647 Fix mac compile 2014-09-17 15:45:43 -07:00
Igor Canadi
54cada92b1 Run make format on PR #249 2014-09-17 15:08:50 -07:00
Igor Canadi
27b22f13a3 Merge pull request #249 from tdfischer/decompression-refactoring
Decompression refactoring
2014-09-17 15:01:40 -07:00
Torrie Fischer
fb6456b00d Replace naked calls to operator new and delete (Fixes #222)
This replaces a mishmash of pointers in the Block and BlockContents classes with
std::unique_ptr. It also changes the semantics of BlockContents to be limited to
use as a constructor parameter for Block objects, as it owns any block buffers
handed to it.
2014-09-17 13:50:07 -07:00
Lei Jin
5600c8f6e5 cuckoo table: return estimated size - 1
Summary:
This is to avoid cutting file prematurely and resulting file size to be
half of specified.

Test Plan: db_bench

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23541
2014-09-17 13:25:29 -07:00
Lei Jin
a062e1f2c4 SetOptions() for memtable related options
Summary: as title

Test Plan:
make all check
I will think a way to set up stress test for this

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23055
2014-09-17 12:49:13 -07:00
Lei Jin
e4eca6a1e5 Options conversion function for convenience
Summary: as title

Test Plan: options_test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23283
2014-09-17 12:46:32 -07:00
Igor Canadi
a7c2094526 Merge pull request #292 from saghmrossi/master
util/cache_bench.cc needs to include inttypes.h
2014-09-17 12:43:39 -07:00
Saghm Rossi
4d0523452a Merge branch 'master' of github.com:saghmrossi/rocksdb 2014-09-17 15:38:43 -04:00
Igor Canadi
60a4aa175e Test use_mmap_reads
Summary: We currently don't test mmap reads as part of db_test. Piggyback it on kWalDir test config.

Test Plan: make check

Reviewers: ljin, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23337
2014-09-17 12:31:53 -07:00
Yueh-Hsuan Chiang
94e43a1dfe [Java] Fixed 32-bit overflowing issue when converting jlong to size_t
Summary:
Fixed 32-bit overflowing issue when converting jlong to size_t by
capping jlong to std::numeric_limits<size_t>::max().

Test Plan:
make rocksdbjava
make jtest

Reviewers: ankgup87, ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23511
2014-09-17 12:30:06 -07:00
Saghm Rossi
f9eaaa66e6 added include for inttypes.h to fix nonworking printf statements 2014-09-17 15:15:53 -04:00
Yueh-Hsuan Chiang
f090575e43 Replaced "built on on earlier work" by "built on earlier work" in README.md 2014-09-17 01:16:17 -07:00
Igor Canadi
faad439ac4 Fix #284
Summary: This work on my compiler, but it turns out some compilers don't implicitly add constness, see: https://github.com/facebook/rocksdb/issues/284. This diff adds constness explicitly.

Test Plan: still compiles

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23409
2014-09-16 10:30:32 -07:00
Igor Canadi
49aacd8d2b Fix make install
Summary: See https://github.com/facebook/rocksdb/issues/283

Test Plan: make install/uninstall

Reviewers: ljin, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23373
2014-09-15 15:30:17 -07:00
Yueh-Hsuan Chiang
acb9348ff3 [Java] Include WriteBatch into RocksDBSample.java, fix how DbBenchmark.java handles WriteBatch.
Summary:
Include WriteBatch into RocksDBSample.java, fix how DbBenchmark.java handles WriteBatch.
Previously DbBenchmark.java does not use WriteBatch when benchmarks is set to fillbatch.

Test Plan:
make rocksdbjava -j32
make jtest
make jdb_bench
cd java
./jdb_bench.sh --benchmarks=fillbatch

Reviewers: naveenatceg, ljin, sdong, ankgup87

Reviewed By: ankgup87

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22983
2014-09-15 12:34:12 -07:00
Igor Canadi
4a27a2f193 Don't sync manifest when disableDataSync = true
Summary: As we discussed offline

Test Plan: compiles

Reviewers: yhchiang, sdong, ljin, dhruba

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22989
2014-09-15 11:32:01 -07:00
Igor Canadi
9b8480d93b Merge pull request #287 from yinqiwen/rate-limiter-crash-fix
fix rate limiter crash #286
2014-09-15 10:00:07 -07:00
yinqiwen
28be16b1dc fix rate limiter crash #286 2014-09-15 20:43:38 +08:00
Igor Canadi
04ce1b25f3 Fix #284 2014-09-13 14:14:10 -07:00
Lei Jin
add22e3515 standardize scripts to run RocksDB benchmarks
Summary:
Hope these scripts will allow people to run/repro benchmark easily
I think it is time to re-run flash benchmarks and report results
Please comment if any other benchmark runs are needed

Test Plan: ran it

Reviewers: yhchiang, igor, sdong

Reviewed By: igor

Subscribers: dhruba, MarkCallaghan, leveldb

Differential Revision: https://reviews.facebook.net/D23139
2014-09-12 16:25:35 -07:00
Igor Canadi
dee91c259d WriteThread
Summary: This diff just moves the write thread control out of the DBImpl. I will need this as I will control column family data concurrency by only accessing some data in the write thread. That way, we won't have to lock our accesses to column family hash table (mappings from IDs to CFDs).

Test Plan: make check

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23301
2014-09-12 16:23:58 -07:00
Igor Canadi
540a257f2c Fix WAL synced
Summary: Uhm...

Test Plan: nope

Reviewers: sdong, yhchiang, tnovak, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23343
2014-09-12 16:15:29 -07:00
Igor Canadi
24f034bf49 Merge pull request #282 from Chilledheart/develop
Fix build issue under macosx
2014-09-12 14:37:45 -07:00
Chilledheart
49fe329e5e Fix build issue under macosx 2014-09-13 05:05:22 +08:00
Yueh-Hsuan Chiang
ebb5c65e60 Add make install
Summary:
Add make install.  If INSTALL_PATH is not set, then rocksdb will be
installed under "/usr/local" directory (/usr/local/include for headers
and /usr/local/lib for library file(s).)

Test Plan:
Develop a simple rocksdb app, called test.cc, and do the followings.

make clean
make static_lib -j32
sudo make install
g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test
./test

sudo make uninstall
make clean
make shared_lib -j32
sudo make install
g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test
./test

make INSTALL_PATH=/tmp/path install
make INSTALL_PATH=/tmp/path uninstall
and make sure things are installed / uninstalled in the specified path.

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23211
2014-09-11 20:41:02 -07:00
Feng Zhu
0352a9fa91 add_wrapped_bloom_test
Summary:
1. wrap a filter policy like what fbcode/multifeed/rocksdb/MultifeedRocksDbKey.h
   to ensure that rocksdb works fine after filterpolicy interface change

Test Plan: 1. valgrind ./bloom_test

Reviewers: ljin, igor, yhchiang, dhruba, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23229
2014-09-11 16:33:46 -07:00
Igor Canadi
9c0e66ce98 Don't run background jobs (flush, compactions) when bg_error_ is set
Summary:
If bg_error_ is set, that means that we mark DB read only. However, current behavior still continues the flushes and compactions, even though bg_error_ is set.

On the other hand, if bg_error_ is set, we will return Status::OK() from CompactRange(), although the compaction didn't actually succeed.

This is clearly not desired behavior. I found this when I was debugging t5132159, although I'm pretty sure these aren't related.

Also, when we're shutting down, it's dangerous to exit RunManualCompaction(), since that will destruct ManualCompaction object. Background compaction job might still hold a reference to manual_compaction_ and this will lead to undefined behavior. I changed the behavior so that we only exit RunManualCompaction when manual compaction job is marked done.

Test Plan: make check

Reviewers: sdong, ljin, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23223
2014-09-11 16:24:16 -07:00
Igor Canadi
a9639bda84 Fix valgrind test
Summary: Get valgrind to stop complaining about uninitialized value

Test Plan: valgrind not complaining anymore

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23289
2014-09-11 15:36:30 -07:00
Igor Canadi
d1f24dc7ee Relax FlushSchedule test
Summary: The test makes sure that we don't call flush too often. For that, it's ok to check if we have less than 10 table files. Otherwise, the test is flaky because it's hard to estimate number of entries in the memtable before it gets flushed (any ideas?)

Test Plan: Still works, but hopefully less flaky.

Reviewers: ljin, sdong, yhchiang

Reviewed by: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23241
2014-09-11 11:00:45 -07:00
Igor Canadi
3d9e6f7759 Push model for flushing memtables
Summary:
When memtable is full it calls the registered callback. That callback then registers column family as needing the flush. Every write checks if there are some column families that need to be flushed. This completely eliminates the need for MakeRoomForWrite() function and simplifies our Write code-path.

There is some complexity with the concurrency when the column family is dropped. I made it a bit less complex by dropping the column family from the write thread in https://reviews.facebook.net/D22965. Let me know if you want to discuss this.

Test Plan: make check works. I'll also run db_stress with creating and dropping column families for a while.

Reviewers: yhchiang, sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23067
2014-09-10 18:46:09 -07:00
Igor Canadi
059e584dd3 [unit test] CompactRange should fail if we don't have space
Summary:
See t5106397.

Also, few more changes:
1. in unit tests, the assumption is that writes will be dropped when there is no space left on device. I changed the wording around it.
2. InvalidArgument() errors are only when user-provided arguments are invalid. When the file is corrupted, we need to return Status::Corruption

Test Plan: make check

Reviewers: sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23145
2014-09-10 17:00:00 -07:00
Lei Jin
dd641b2117 fix RocksDB java build
Summary: as title

Test Plan: make rocksdbjava

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23193
2014-09-10 12:00:32 -07:00
Feng Zhu
53404d9fb7 add_qps_info_in cache bench
Summary: print qps in summary

Test Plan: ./cache_bench

Reviewers: yhchiang, ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23079
2014-09-10 09:46:56 -07:00
Igor Canadi
a52cecb56c Fix Mac compile 2014-09-09 18:42:35 -07:00
Jonah Cohen
092f97e219 Fix comments and typos
Summary: Correct some comments and typos in RocksDB.

Test Plan: Inspection

Reviewers: sdong, igor

Reviewed By: igor

Differential Revision: https://reviews.facebook.net/D23133
2014-09-09 15:20:49 -07:00
Xiaozheng Tie
6cc12860f0 Added a few statistics for BackupableDB
Summary:
Added the following statistics to BackupableDB:

1. Number of successful and failed backups in class BackupStatistics
2. Time taken to do a backup
3. Number of files in a backup

1 is implemented in the BackupStatistics class
2 and 3 are added in the BackupMeta and BackupInfo class

Test Plan:
1 can be tested using BackupStatistics::ToString(),
2 and 3 can be tested in the BackupInfo class

Reviewers: sdong, igor2, ljin, igor

Reviewed By: igor

Differential Revision: https://reviews.facebook.net/D22785
2014-09-09 13:44:42 -07:00
Igor Canadi
0a42295a24 Fix SimpleWriteTimeoutTest
Summary:
In column family's SanitizeOptions() [1], we make sure that min_write_buffer_number_to_merge is normal value. However, this test depended on the fact that setting min_write_buffer_number_to_merge to be bigger than max_write_buffer_number will cause a deadlock. I'm not sure how it worked before.

This diff fixes it by scheduling sleeping background task, which will actually block any attempts of flushing.

[1] https://github.com/facebook/rocksdb/blob/master/db/column_family.cc#L104

Test Plan: the test works now

Reviewers: yhchiang, sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23103
2014-09-09 11:50:05 -07:00
sdong
06d986252a Always pass MergeContext as pointer, not reference
Summary: To follow the coding convention and make sure when passing reference as a parameter it is also const, pass MergeContext as a pointer to mem tables.

Test Plan: make all check

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: leveldb, dhruba, yhchiang

Differential Revision: https://reviews.facebook.net/D23085
2014-09-09 11:37:32 -07:00
Stanislau Hlebik
d343c3fe46 Improve db recovery
Summary: Avoid creating unnecessary sst files while db opening

Test Plan: make all check

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: zagfox, yhchiang, ljin, leveldb

Differential Revision: https://reviews.facebook.net/D20661
2014-09-09 11:18:50 -07:00
Igor Canadi
6bb7e3ef25 Merger test
Summary: I abandoned https://reviews.facebook.net/D18789, but I wrote a good unit test there, so let's check it in. :)

Test Plan: this is test

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22827
2014-09-08 22:24:40 -07:00
Igor Canadi
88841bd007 Explicitly cast char to signed char in Hash()
Summary:
The compilers we use treat char as signed. However, this is not guarantee of C standard and some compilers (for ARM platform for example), treat char as unsigned. Code that assumes that char is either signed or unsigned is wrong.

This change explicitly casts the char to signed version. This will not break any of our use cases on x86, which, I believe are all of them. In case somebody out there is using RocksDB on ARM AND using bloom filters, they're going to have a bad time. However, it is very unlikely that this is the case.

Test Plan: sanity test with previous commit (with new sanity test)

Reviewers: yhchiang, ljin, sdong

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D22767
2014-09-08 18:57:40 -07:00
Lei Jin
52311463e9 MemTableOptions
Summary: removed reference to options in WriteBatch and DBImpl::Get()

Test Plan: make all check

Reviewers: yhchiang, igor, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23049
2014-09-08 18:46:52 -07:00
Naveen
1d284db213 Addressing review comments 2014-09-08 17:44:52 -07:00
Igor Canadi
55114e7f40 Some updates for SpatialDB
Summary:
1. add db statistics
2. write out batch every millionth write

Test Plan: unit tests

Reviewers: ljin, sdong, yinwang

Reviewed By: yinwang

Differential Revision: https://reviews.facebook.net/D22755
2014-09-08 17:00:43 -07:00
Lei Jin
171d4ff4a2 remove TailingIterator reference in db_impl.h
Summary: as title

Test Plan: make release

Reviewers: igor

Differential Revision: https://reviews.facebook.net/D23073
2014-09-08 15:39:53 -07:00
Lei Jin
9b0f7ffa1c rename version_set options_ to db_options_ to avoid confusion
Summary: as title

Test Plan: make release

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23007
2014-09-08 15:25:01 -07:00
Igor Canadi
2d57828d0e Check stop level trigger-0 before slowdown level-0 trigger
Summary: ...

Test Plan: Can't repro the test failure, but let's see what jenkins says

Reviewers: zagfox, sdong, ljin

Reviewed By: sdong, ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23061
2014-09-08 15:23:58 -07:00
Lei Jin
659d2d50c3 move compaction_filter to immutable_options
Summary:
all shared_ptrs are in immutable_options now. This will also make
options assignment a little cheaper

Test Plan: make release

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23001
2014-09-08 15:09:25 -07:00