A library that provides an embeddable, persistent key-value store for fast storage.
Go to file
Dhruba Borthakur 7632fdb5cb Support taking a configurable number of files from the same level to compact in a single compaction run.
Summary:
The compaction process takes some files from LevelK and
merges it into LevelK+1. The number of files it picks from
LevelK was capped such a way that the total amount of
data picked does not exceed the maxfilesize of that level.
This essentially meant that only one file from LevelK
is picked for a single compaction.

For bulkloads, we would like to take many many file from
LevelK and compact them using a single compaction run.

This patch introduces a option called the 'source_compaction_factor'
(similar to expanded_compaction_factor). It is a multiplier
that is multiplied by the maxfilesize of that level to arrive
at the limit that is used to throttle the number of source
files from LevelK.  For bulk loads, set source_compaction_factor
to a very high number so that multiple files from the same
level are picked for compaction in a single compaction.

The default value of source_compaction_factor is 1, so that
we can keep backward compatibilty with existing compaction semantics.

Test Plan: make clean check

Reviewers: emayanke, sheki

Reviewed By: emayanke

CC: leveldb

Differential Revision: https://reviews.facebook.net/D6867
2012-11-21 08:37:03 -08:00
db Support taking a configurable number of files from the same level to compact in a single compaction run. 2012-11-21 08:37:03 -08:00
doc merge 1.5 2012-08-28 11:43:33 -07:00
hdfs This is the mega-patch multi-threaded compaction 2012-10-19 14:00:53 -07:00
helpers/memenv Fix all warnings generated by -Wall option to the compiler. 2012-11-06 14:07:31 -08:00
include/leveldb Support taking a configurable number of files from the same level to compact in a single compaction run. 2012-11-21 08:37:03 -08:00
java Add LevelDb's JNI wrapper 2012-10-05 13:13:49 -07:00
port Make compression options configurable. These include window-bits, level and strategy for ZlibCompression 2012-11-02 11:26:39 -07:00
scribe fix db_test error with scribe logger turned on 2012-08-28 11:22:58 -07:00
snappy Build with gcc-4.7.1-glibc-2.14.1. 2012-09-17 10:56:26 -07:00
table Metrics: record compaction drop's and bloom filter effectiveness 2012-11-09 11:38:45 -08:00
thrift Implement RowLocks for assoc schema 2012-10-03 23:19:01 -07:00
tools Enhance manifest_dump to print each individual edit. 2012-11-19 12:04:35 -08:00
util Support taking a configurable number of files from the same level to compact in a single compaction run. 2012-11-21 08:37:03 -08:00
.arcconfig Support arcdiff. 2012-05-09 23:35:05 -07:00
.gitignore Added bloom filter support. 2012-04-17 08:36:46 -07:00
AUTHORS reverting disastrous MOE commit, returning to r21 2011-04-19 23:11:15 +00:00
build_detect_platform Make sse compilation optional. 2012-11-13 16:25:57 -08:00
build_detect_version Record the version of the source repository that was used to build the leveldb library. 2012-08-24 15:18:43 -07:00
fbcode.gcc471.sh Make sse compilation optional. 2012-11-13 16:25:57 -08:00
LICENSE reverting disastrous MOE commit, returning to r21 2011-04-19 23:11:15 +00:00
Makefile Fix all warnings generated by -Wall option to the compiler. 2012-11-06 14:07:31 -08:00
NEWS sync with upstream @ 21409451 2011-05-21 02:17:43 +00:00
README @20776309 2011-04-20 22:48:11 +00:00
README.fb Push release 1.5.5.fb. 2012-11-13 16:28:11 -08:00
TODO A number of smaller fixes and performance improvements: 2011-06-22 02:36:45 +00:00

leveldb: A key-value store
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

The code under this directory implements a system for maintaining a
persistent key/value store.

See doc/index.html for more explanation.
See doc/impl.html for a brief overview of the implementation.

The public interface is in include/*.h.  Callers should not include or
rely on the details of any other header files in this package.  Those
internal APIs may be changed without warning.

Guide to header files:

include/db.h
    Main interface to the DB: Start here

include/options.h
    Control over the behavior of an entire database, and also
    control over the behavior of individual reads and writes.

include/comparator.h
    Abstraction for user-specified comparison function.  If you want
    just bytewise comparison of keys, you can use the default comparator,
    but clients can write their own comparator implementations if they
    want custom ordering (e.g. to handle different character
    encodings, etc.)

include/iterator.h
    Interface for iterating over data. You can get an iterator
    from a DB object.

include/write_batch.h
    Interface for atomically applying multiple updates to a database.

include/slice.h
    A simple module for maintaining a pointer and a length into some
    other byte array.

include/status.h
    Status is returned from many of the public interfaces and is used
    to report success and various kinds of errors.

include/env.h
    Abstraction of the OS environment.  A posix implementation of
    this interface is in util/env_posix.cc

include/table.h
include/table_builder.h
    Lower-level modules that most clients probably won't use directly