andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
A library that provides an embeddable, persistent key-value store for fast storage.
3,391 Commits 152 Branches 236 Tags 271 MiB
C++ 82.1%
Java 10.3%
C 2.5%
Python 1.7%
Perl 1.1%
Other 2.1%
47b8743984
Go to file
Igor Canadi 47b8743984 Make Compaction class easier to use 
Summary:
The goal of this diff is to make Compaction class easier to use. This should also make new compaction algorithms easier to write (like CompactFiles from @yhchiang and dynamic leveled and multi-leveled universal from @sdong).

Here are couple of things demonstrating that Compaction class is hard to use:
1. we have two constructors of Compaction class
2. there's this thing called grandparents_, but it appears to only be setup for leveled compaction and not compactfiles
3. it's easy to introduce a subtle and dangerous bug like this: D36225
4. SetupBottomMostLevel() is hard to understand and it shouldn't be. See this comment: afbafeaeae/db/compaction.cc (L236-L241). It also made it harder for @yhchiang to write CompactFiles, as evidenced by this: afbafeaeae/db/compaction_picker.cc (L204-L210)

The problem is that we create Compaction object, which holds a lot of state, and then pass it around to some functions. After those functions are done mutating, then we call couple of functions on Compaction object, like SetupBottommostLevel() and MarkFilesBeingCompacted(). It is very hard to see what's happening with all that Compaction's state while it's travelling across different functions. If you're writing a new PickCompaction() function you need to try really hard to understand what are all the functions you need to run on Compaction object and what state you need to setup.

My proposed solution is to make important parts of Compaction immutable after construction. PickCompaction() should calculate compaction inputs and then pass them onto Compaction object once they are finalized. That makes it easy to create a new compaction -- just provide all the parameters to the constructor and you're done. No need to call confusing functions after you created your object.

This diff doesn't fully achieve that goal, but it comes pretty close. Here are some of the changes:
* have one Compaction constructor instead of two.
* inputs_ is constant after construction
* MarkFilesBeingCompacted() is now private to Compaction class and automatically called on construction/destruction.
* SetupBottommostLevel() is gone. Compaction figures it out on its own based on the input.
* CompactionPicker's functions are not passing around Compaction object anymore. They are only passing around the state that they need.

Test Plan:
make check
make asan_check
make valgrind_check

Reviewers: rven, anthony, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: sdong, yhchiang, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36687
2015-04-10 15:01:54 -07:00
arcanist_util Integrate Jenkins with Phabricator 2015-04-07 11:56:29 -07:00
build_tools Remove use of whole-archive to include jemalloc 2015-04-09 15:10:53 -07:00
coverage Fix coverage script 2014-11-03 14:53:00 -08:00
db Make Compaction class easier to use 2015-04-10 15:01:54 -07:00
doc Remove seek compaction 2014-06-20 10:23:02 +02:00
examples Fix formatting 2015-02-09 09:53:30 -08:00
hdfs Remove unused parameter in CancelAllBackgroundWork 2015-03-16 21:07:54 -07:00
include Add thread-safety documentation to MemTable and related classes 2015-04-08 21:10:35 -07:00
java Fixed a typo in RocksDBSample.java 2015-03-25 11:09:30 -07:00
port stack_trace.cc: fix #elif check for OS_MACOSX 2015-03-17 12:00:55 +01:00
table Clean up compression logging 2015-04-06 12:50:44 -07:00
third-party Update COMMIT.md 2015-03-30 17:48:16 -07:00
tools Script to check whether RocksDB can read DB generated by previous releases and vice versa 2015-04-08 16:04:59 -07:00
util Fixed xfunc related compile errors in ROCKSDB_LITE 2015-04-09 21:05:18 -07:00
utilities Fix the compilation error in flashcache.cc on Mac 2015-04-07 15:27:23 -07:00
.arcconfig Integrate Jenkins with Phabricator 2015-04-07 11:56:29 -07:00
.clang-format A script that automatically reformat affected lines 2014-01-14 12:21:24 -08:00
.gitignore run 'make check's rules (and even subtests) in parallel 2015-04-06 12:35:25 -07:00
.travis.yml Only run db_test in Travis 2015-03-17 15:24:16 -07:00
AUTHORS Add AUTHORS file. Fix #203 2014-09-29 10:52:18 -07:00
CONTRIBUTING.md facebook accounts are not required for CLA signers 2014-07-08 05:57:54 -04:00
HISTORY.md A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge 2015-04-06 10:27:21 -07:00
INSTALL.md Optimize default compile to compilation platform by default 2014-12-15 11:29:41 +01:00
LICENSE Fix copyright year 2014-03-12 12:06:58 -07:00
Makefile MemTableList tests 2015-04-09 18:01:11 -07:00
PATENTS Fix the patent format 2013-10-16 15:37:32 -07:00
README.md Replaced "built on on earlier work" by "built on earlier work" in README.md 2014-09-17 01:16:17 -07:00
ROCKSDB_LITE.md RocksDBLite 2014-04-15 13:39:26 -07:00
src.mk build: don't use a glob for java/rocksjni/* 2015-04-07 15:19:25 -07:00
USERS.md Add LinkedIn back to USERS.md 2015-04-10 09:50:19 -07:00
Vagrantfile RocksDB on FreeBSD support 2015-02-26 15:19:17 -08:00

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

Build Status

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/