Commit Graph

502 Commits

Author SHA1 Message Date
Chip Turner
0b83a83191 Fix poor error on num_levels mismatch and few other minor improvements
Summary:
Previously, if you opened a db with num_levels set lower than
the database, you received the unhelpful message "Corruption:
VersionEdit: new-file entry."  Now you get a more verbose message
describing the issue.

Also, fix handling of compression_levels (both the run-over-the-end
issue and the memory management of it).

Lastly, unique_ptr'ify a couple of minor calls.

Test Plan: make check

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D8151
2013-01-25 15:37:26 -08:00
Dhruba Borthakur
16e96b1242 Cleanup TODO/NEWS/AUTHORS files
Summary:
These files are not relevant anymore.

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2013-01-25 09:11:26 -08:00
Chip Turner
772f75b3fb Stop continually re-creating build_version.c
Summary:
We continually rebuilt build_version.c because we put the
current date into it, but that's what __DATE__ already is.  This makes
builds faster.

This also fixes an issue with 'make clean FOO' not working properly.

Also tweak the build rules to be more consistent, always have warnings,
and add a 'make release' rule to handle flags for release builds.

Test Plan: make, make clean

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D8139
2013-01-24 17:51:39 -08:00
Chip Turner
3dafdfb2c4 Use fallocate to prevent excessive allocation of sst files and logs
Summary:
On some filesystems, pre-allocation can be a considerable
amount of space.  xfs in our production environment pre-allocates by
1GB, for instance.  By using fallocate to inform the kernel of our
expected file sizes, we eliminate this wasteage (that isn't recovered
until the file is closed which, in the case of LOG files, can be a
considerable amount of time).

Test Plan:
created an xfs loopback filesystem, mounted with
allocsize=4M, and ran db_stress.  LOG file without this change was 4M,
and with it it was 128k then grew to normal size.

Reviewers: dhruba

Reviewed By: dhruba

CC: adsharma, leveldb

Differential Revision: https://reviews.facebook.net/D7953
2013-01-24 12:25:13 -08:00
Chip Turner
2fdf91a4f8 Fix a number of object lifetime/ownership issues
Summary:
Replace manual memory management with std::unique_ptr in a
number of places; not exhaustive, but this fixes a few leaks with file
handles as well as clarifies semantics of the ownership of file handles
with log classes.

Test Plan: db_stress, make check

Reviewers: dhruba

Reviewed By: dhruba

CC: zshao, leveldb, heyongqiang

Differential Revision: https://reviews.facebook.net/D8043
2013-01-23 16:54:11 -08:00
Kosie van der Merwe
88b79b24f3 Fixed didIO not being set with no block_cache
Summary:
In `Table::BlockReader()` when there was no block cache `didIO` was not set.

This didn't seem to matter as `didIO` is only used to trigger seek compactions. However, I would like it if someone else could check that is the case.

Test Plan: `make check OPT="-g -O3"`

Reviewers: dhruba, vamsi

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D8133
2013-01-23 12:49:10 -08:00
Abhishek Kona
16903c35b0 Add counters to count gets and writes
Summary: Add Tickers to count Write's and Get's

Test Plan: make check

Reviewers: dhruba, chip

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7977
2013-01-17 12:27:56 -08:00
Kosie van der Merwe
3c3df7402f Fixed issues Valgrind found.
Summary:
Found issues with `db_test` and `db_stress` when running valgrind.

`DBImpl` had an issue where if an compaction failed then it will use the uninitialised file size of an output file is used. This manifested as the final call to output to the log in `DoCompactionWork()` branching on uninitialized memory (all the way down in printf's innards).

Test Plan:
Ran `valgrind --track_origins=yes ./db_test` and `valgrind ./db_stress` to see if issues disappeared.

Ran `make check` to see if there were no regressions.

Reviewers: vamsi, dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D8001
2013-01-17 10:04:45 -08:00
Aaron Kushner
dfcf6136cc Minor improvements to the regression testing
Summary:
Several fixes:
1) Use mktemp to create the files and directories
2) Take the stat file as an argument so that the buildservers
   can specify a file in the WORKSPACE and not in /tmp
3) Use nproc to set make -j value.
4) Check for valid values before sending to ODS
5) Cleanup the grep/cut pipeline to just use awk

Test Plan: Verify tests run and complete

Reviewers: sheki, dhruba

Reviewed By: sheki

Differential Revision: https://reviews.facebook.net/D7995
2013-01-16 14:47:20 -08:00
amayank
4b1e9f0a35 Added an API in rocksdb for checking for "invalid argument" and "not supported" for leveldb::Status
Summary: a function added to status.h to check whether Status::code is InvalidArgument and similarly for NotSupported state

Test Plan: visual inspection

Reviewers: heyongqiang, dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7989
2013-01-16 14:30:45 -08:00
Abhishek Kona
7d5a4383bb rollover manifest file.
Summary:
Check in LogAndApply if the file size is more than the limit set in
Options.
Things to consider : will this be expensive?

Test Plan: make all check. Inputs on a new unit test?

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7701
2013-01-16 12:09:44 -08:00
Abhishek Kona
c54884faa9 Fix Regression script.
Run it for a shorter time.
2013-01-16 11:23:35 -08:00
Chip Turner
a2dcd79c1e Add optional clang compile mode
Summary:
clang is an alternate compiler based on llvm.  It produces
nicer error messages and finds some bugs that gcc doesn't, such as the
size_t change in this file (which caused some write return values to be
misinterpreted!)

Clang isn't the default; to try it, do "USE_CLANG=1 make" or "export
USE_CLANG=1" then make as normal

Test Plan: "make check" and "USE_CLANG=1 make check"

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7899
2013-01-15 18:48:37 -08:00
Chip Turner
9bbcab57a9 Fix broken build
Summary: Mis-merged from HEAD, had a duplicate declaration.

Test Plan: make -j32 OPT=-g

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7911
2013-01-15 14:05:49 -08:00
Kosie van der Merwe
28fe86c48a Fixed bug with seek compactions on Level 0
Summary: Due to how the code handled compactions in Level 0 in `PickCompaction()` it could be the case that two compactions on level 0 ran that produced tables in level 1 that overlap. However, this case seems like it would only occur on a seek compaction which is unlikely on level 0. Furthermore, level 0 and level 1 had to have a certain arrangement of files.

Test Plan:
make check

Reviewers: dhruba, vamsi

Reviewed By: dhruba

CC: leveldb, sheki

Differential Revision: https://reviews.facebook.net/D7923
2013-01-15 12:43:09 -08:00
Abhishek Kona
8ce418cd79 Change default regression test location to /tmp from
/data/users/abhishekk
2013-01-15 12:23:11 -08:00
Abhishek Kona
917377c1fc Bash script to run db_bench with options and send data to ods.
Summary:
Basic Regression test.
Plan to run this every-night and record qps in ods.

Test Plan: ran locally and checked

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7929
2013-01-15 12:18:01 -08:00
Chip Turner
c0cb289d57 Various build cleanups/improvements
Summary:
Specific changes:

1) Turn on -Werror so all warnings are errors
2) Fix some warnings the above now complains about
3) Add proper dependency support so changing a .h file forces a .c file
to rebuild
4) Automatically use fbcode gcc on any internal machine rather than
whatever system compiler is laying around
5) Fix jemalloc to once again be used in the builds (seemed like it
wasn't being?)
6) Fix issue where 'git' would fail in build_detect_version because of
LD_LIBRARY_PATH being set in the third-party build system

Test Plan:
make, make check, make clean, touch a header file, make sure
rebuild is expected

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7887
2013-01-14 18:40:22 -08:00
Mark Callaghan
2ba125faf6 fix warning for unused variable
Test Plan: compile

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7857
2013-01-11 15:00:47 -08:00
Abhishek Kona
85ad13be1a Port fix for Leveldb manifest writing bug from Open-Source
Summary:
Pretty much a blind copy of the patch in open source.
Hope to get this in before we make a release

Test Plan: make clean check

Reviewers: dhruba, heyongqiang

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7809
2013-01-10 12:06:03 -08:00
Abhishek Kona
41d7809212 Pom changes to make relase 1.5.7 for java.
Summary: Ran ./build_java.sh bump_version 1.5.7

Test Plan: automated change

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7833
2013-01-10 10:43:43 -08:00
Kosie van der Merwe
4e9d9d989f Fixed wrong assumption in Table::Open()
Summary:
`Table::Open()` assumes that `size` correctly describes the size of `file`, added a check that the footer is actually the right size and for good measure added assertions to `Footer::DecodeFrom()`.

This was discovered by running `valgrind ./db_test` and seeing that `Footer::DecodeFrom()` was accessing uninitialized memory.

Test Plan:
make clean check

ran `valgrind ./db_test` and saw DBTest.NoSpace no longer complains about a conditional jump being dependent on uninitialized memory.

Reviewers: dhruba, vamsi, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7815
2013-01-09 10:44:30 -08:00
Dhruba Borthakur
f881d6f226 Release 1.5.7.fb
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2013-01-09 10:26:03 -08:00
Abhishek Kona
2e1ad2c48b Remove unnecessary asserts in table/merger.cc
Summary:
The asserts introduced in https://reviews.facebook.net/D7629 are
wrong.

The direction of iteration is changed after the function call so they
assert's fail.

Test Plan: make clean check

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7827
2013-01-09 09:52:45 -08:00
Kosie van der Merwe
d8371ef1f6 Fixing some issues Valgrind found
Summary: Found some issues running Valgrind on `db_test` (there are still some outstanding ones) and fixed them.

Test Plan:
make check

ran `valgrind ./db_test` and saw that errors no longer occur

Reviewers: dhruba, vamsi, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7803
2013-01-08 12:16:40 -08:00
Kosie van der Merwe
4d339d7462 Fixed memory leak in ShardedLRUCache
Summary: `~ShardedLRUCache()` was empty despite `init()` allocating memory on the heap. Fixed the leak by freeing memory allocated by `init()`.

Test Plan:
make check

Ran valgrind on db_test before and after patch and saw leaked memory went down

Reviewers: vamsi, dhruba, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7791
2013-01-08 11:24:15 -08:00
Dhruba Borthakur
628dc2aad9 db_bench should use the default value for max_grandparent_overlap_factor.
Summary:
This was a peformance regression caused by https://reviews.facebook.net/D6729.
The default value of max_grandparent_overlap_factor was erroneously
set to 0 in db_bench.

This was causing compactions to create really really small files because the max_grandparent_overlap_factor was erroneously set to zero in the benchmark.

Test Plan: Run --benchmarks=overwrite

Reviewers: heyongqiang, emayanke, sheki, MarkCallaghan

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7797
2013-01-08 11:21:11 -08:00
Kosie van der Merwe
d6e873f22f Added clearer error message for failure to create db directory in DBImpl::Recover()
Summary:
Changed CreateDir() to CreateDirIfMissing() so a directory that already exists now causes and error.

Fixed CreateDirIfMissing() and added Env.DirExists()

Test Plan:
make check to test for regessions

Ran the following to test if the error message is not about lock files not existing
./db_bench --db=dir/testdb

After creating a file "testdb", ran the following to see if it failed with sane error message:
./db_bench --db=testdb

Reviewers: dhruba, emayanke, vamsi, sheki

Reviewed By: emayanke

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7707
2013-01-07 10:11:18 -08:00
Mark Callaghan
4069f66cc7 Add --seed, --read_range to db_bench
Summary:
Adds the option --seed to db_bench to specify the base for the per-thread RNG.
When not set each thread uses the same value across runs of db_bench which defeats
IO stress testing.

Adds the option --read_range. When set to a value > 1 an iterator is created and
each query done for the randomread benchmark will do a range scan for that many
rows. When not set or set to 1 the existing behavior (a point lookup) is done.

Fixes a bug where a printf format string was missing.

Test Plan: run db_bench

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7749
2013-01-07 09:56:10 -08:00
Kosie van der Merwe
8cd86a7be5 Fixing and adding some comments
Summary:
`MemTableList::Add()` neglected to mention that it took ownership of the reference held by its caller.

The comment in `MemTable::Get()` was wrong in describing the format of the key.

Test Plan: None

Reviewers: dhruba, sheki, emayanke, vamsi

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7755
2013-01-03 17:13:56 -08:00
Abhishek Kona
3f7af03a2d Use a priority queue to merge files.
Summary:
Use a std::priority_queue in merger.cc instead of doing a o(n) search
every time.
Currently only the ForwardIteration uses a Priority Queue.

Test Plan: make all check

Reviewers: dhruba

Reviewed By: dhruba

CC: emayanke, zshao

Differential Revision: https://reviews.facebook.net/D7629
2013-01-02 13:52:25 -08:00
Dhruba Borthakur
d7d43ae21a ExtendOverlappingInputs too slow for large databases.
Summary:
There was a bug in the ExtendOverlappingInputs method so that
the terminating condition for the backward search was incorrect.

Test Plan: make clean check

Reviewers: sheki, emayanke, MarkCallaghan

Reviewed By: MarkCallaghan

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7725
2013-01-02 13:19:06 -08:00
Dhruba Borthakur
2fc394a327 Do not compile thrift for fbcode build.
Summary:
1. The thrift libraries do not need to be built anyore.
2. SSE is dynamically detected via
   1aae609b92

Test Plan: compile and build

Reviewers: sheki, emayanke

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7665
2012-12-27 16:11:11 -08:00
Dhruba Borthakur
5b05417df3 Complication error when using gcc 4.7.1.
Summary:
There is a compilation error while using gcc 4.7.1.
util/ldb_cmd.cc:381:3: error: ‘leveldb::ReadOptions::ReadOptions’ names the constructor, not the type
util/ldb_cmd.cc:381:37: error: expected ‘;’ before ‘read_options’
util/ldb_cmd.cc:381:49: error: statement cannot resolve address of overloaded function

Test Plan: make clean check

Reviewers: sheki, emayanke, zshao

Reviewed By: emayanke

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7659
2012-12-27 12:38:20 -08:00
Zheng Shao
0f762ac573 ldb: Add command "ldb query" to support random read from the database
Summary: The queries will come from stdin.  One key per line.  The output will be in stdout, in the format of "<key> ==> <value>" if found, or "<key>" if not found.  "--hex" uses HEX-encoded keys and values in both input and output.

Test Plan: ldb query --db=leveldb_db --hex

Reviewers: dhruba, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7617
2012-12-26 20:37:42 -08:00
Zheng Shao
dcece4707e ldb: Fix incorrect arg parsing
Summary: We were ignoring additional chars at the end of an arg.  This can create confusion, e.g. --disable_wal=0 will act the same as --disable_wal without any warnings.

Test Plan:
Tried this:
[zshao@dev485 ~/git/rocksdb] ./ldb dump --statsAAA
Failed: Unknown argument:--statsAAA

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7635
2012-12-26 20:09:10 -08:00
Zheng Shao
04832dbc02 sst_dump: Fix incorrect cmd args parsing
Summary: The current parsing logic ignores any additional chars after the arg.

Test Plan: "./sst_dump --verify_checksumAAA" now outputs error.

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7611
2012-12-26 10:57:04 -08:00
Abhishek Kona
396c3aa08e Create a long running test to check GetUpdatesSince.
Summary:
Create an Executable with
* A thread to do Put's
* A thread to use GetUpdatesSince. Check if we miss any sequence
  Numbers.

Test Plan: It runs.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7383
2012-12-21 14:10:06 -08:00
Zheng Shao
127ee2e677 manifest_dump: Fix the help message and make it consistent
Summary: ldb uses --output_hex so make manifest_dump do the same thing.

Test Plan:
[zshao@dev485 ~/git/rocksdb] ./manifest_dump --output_hex --file=/data/users/zshao/test_leveldb/MANIFEST-000034
manifest_file_number 42 next_file_number 43 last_sequence 2311567 log_number 36  prev_log_number 0
--- level 0 --- version# 0 ---
--- level 1 --- version# 0 ---
--- level 2 --- version# 0 ---
--- level 3 --- version# 0 ---
 5:27788699['0000027F4FBE0000' @ 1 : 1 .. '11CE749602C90000' @ 160642 : 1]
 7:27785313['11CE773DA7E00000' @ 160643 : 1 .. '23A4C63EC55D0000' @ 321094 : 1]
 9:27784288['23A4D581FCD30000' @ 321095 : 1 .. '3576291D12D00000' @ 481428 : 1]
 38:64378271['35762BF0E0CE0000' @ 481429 : 1 .. '5E987E0604700000' @ 852910 : 1]
 39:64379046['5E987EB0BDD50000' @ 852911 : 1 .. '87C954330E840000' @ 1224603 : 1]
 40:10169201['87C95507E49C0000' @ 1224604 : 1 .. '8E48DC0933B70000' @ 1283317 : 1]
 21:27798825['8E48DFB0D7CE0000' @ 1283318 : 1 .. 'A00675F8AD7E0000' @ 1443826 : 1]
 23:27793751['A006777536E30000' @ 1443827 : 1 .. 'B1D1787FE8670000' @ 1604553 : 1]
 25:27801659['B1D179289BB30000' @ 1604554 : 1 .. 'C396D3A69DCE0000' @ 1765012 : 1]
 27:27792661['C396DA1E03B10000' @ 1765013 : 1 .. 'D55C9974FCC10000' @ 1925513 : 1]
 29:27789095['D55C9B47CBC00000' @ 1925514 : 1 .. 'E71F67D11CCC0000' @ 2085789 : 1]
 31:27793145['E71F7A667E740000' @ 2085790 : 1 .. 'F8D4712EF3D90000' @ 2246454 : 1]
 41:11246031['F8D4715916A70000' @ 2246455 : 1 .. 'FFFFFCAE97DF0000' @ 2311567 : 1]
--- level 4 --- version# 0 ---
--- level 5 --- version# 0 ---
--- level 6 --- version# 0 ---

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7575
2012-12-21 10:45:24 -08:00
Zheng Shao
01d57a33fe sst_dump: Add --output_hex option and output the same format as ldb dump
Summary: Now sst_dump has the same option --output_hex as "ldb dump" and also share the same output format.  So we can do "sst_dump ... | ldb load ..." for an experiment.

Test Plan:
[zshao@dev485 ~/git/rocksdb] ./sst_dump --file=/data/users/zshao/test_leveldb/000005.sst  --output_hex | head -n 2
0000027F4FBE00000101000000000000 ==> D901000000000000000057596F7520726563656976656420746F6461792773207370656369616C20676966742120436C69636B2041636365707420746F207669657720796F75722047696674206265666F72652069742064697361707065617273210000000000000000
000007F9C2D400000102000000000000 ==> D1010000000000000000544974277320676F6F6420746F206265204B696E67E280A6206F7220517565656E212048657265277320796F75722054756573646179204D79737465727920476966742066726F6D20436173746C6556696C6C65219B7B227A636F6465223A22633566306531633039663764222C227470223A22613275222C227A6B223A22663638663061343262666264303966383435666239626235366365396536643024246234506C345139382E734C4D33522169482D4F31315A64794C4B7A4F4653766D7863746534625F2A3968684E3433786521776C427636504A414355795F70222C227473223A313334343936313737357D0000000000000000

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7587
2012-12-21 10:44:19 -08:00
Zheng Shao
70f0f50731 ldb: file_size and write_buffer_size options.
Summary:
This allows ldb to control the write_buffer_size (which reflects to L0 file size) and file_size (which reflects to L1 file size).  Since the target_file_size_ratio is 1 by default, all other levels will also have the same file size as L1.

As part of the diff, I also cleaned up some unused code and help messages.

Test Plan: ./ldb load --db=/data/users/zshao/test_leveldb --file_size=64000000 --write_buffer_size=32000000 --create_if_missing --input_hex --disable_wal

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7569
2012-12-21 10:40:38 -08:00
Abhishek Kona
1aae609b92 Use CRC32 ss42 instruction. Load it dynamically.
Test Plan: make all check

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb, zshao

Differential Revision: https://reviews.facebook.net/D7503
2012-12-21 10:20:32 -08:00
Zheng Shao
7521a225d1 sst_dump: Error message should include the case that compression algorithms are not supported.
Summary: It took me almost a day to debug this. :(  Although I got to learn the file format as a by-product, this time could be saved if we have better error messages.

Test Plan: gmake clean all; sst_dump --hex --file=000005.sst

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7551
2012-12-20 15:29:51 -08:00
Dhruba Borthakur
551f01fb23 Unit test to test block format.
Summary: This is a standalone unit test to test the format of a block.

Test Plan: ./block_test

Reviewers: sheki

Reviewed By: sheki

Differential Revision: https://reviews.facebook.net/D7533
2012-12-20 14:55:07 -08:00
Zheng Shao
58d144444e Add libevent include and lib directories
Summary:
Without this fix, I see failures like this:
[zshao@dev1049 /data/users/zshao/rocksdb] . fbcode.gcc471.sh; gmake clean libleveldb.a
. . .
./thrift/lib/cpp/async/TEventUtil.h:22:32: fatal error: event.h: No such file or directory

Test Plan: . fbcode.gcc471.sh; make clean libleveldb.a

Reviewers: dhruba, emayanke, sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7497
2012-12-20 14:31:54 -08:00
Dhruba Borthakur
f4c2b7cf97 Enhance ReadOnly mode to process the all committed transactions.
Summary:
Leveldb has an api OpenForReadOnly() that opens the database
in readonly mode. This call had an option to not process the
transaction log.  This patch removes this option and always
processes all transactions that had been committed. It has
been done in such a way that it does not create/write to
any new files in the process. The invariant of "no-writes"
to the leveldb data directory is still true.

This enhancement allows multiple threads to open the same database
in readonly mode and access all trancations that were committed right
upto the OpenForReadOnly call.

I changed the public API to match the new semantics because
there are no users who are currently using this api.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7479
2012-12-19 16:30:46 -08:00
Zheng Shao
be9b862d47 ldb: add "ldb load" command
Summary: This command accepts key-value pairs from stdin with the same format of "ldb dump" command.  This allows us to try out different compression algorithms/block sizes easily.

Test Plan: dump, load, dump, verify the data is the same.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7443
2012-12-19 01:45:59 -08:00
Abhishek Kona
2585979fd7 Release 1.5.6 for Java code + Script to automate it.
Test Plan: it compiles and deploys.

Reviewers: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7341
2012-12-17 12:11:11 -08:00
Dhruba Borthakur
3d1e92b05a Enhancements to rocksdb for better support for replication.
Summary:
1. The OpenForReadOnly() call should not lock the db. This is useful
so that multiple processes can open the same database concurrently
for reading.
2. GetUpdatesSince should not error out if the archive directory
does not exist.
3. A new constructor for WriteBatch that can takes a serialized
string as a parameter of the constructor.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7449
2012-12-17 11:40:19 -08:00
Kosie van der Merwe
62d48571de Added meta-database support.
Summary:
Added kMetaDatabase for meta-databases in db/filename.h along with supporting
fuctions.
Fixed switch in DBImpl so that it also handles kMetaDatabase.
Fixed DestroyDB() that it can handle destroying meta-databases.

Test Plan: make check

Reviewers: sheki, emayanke, vamsi, dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7245
2012-12-17 11:26:59 -08:00