Commit Graph

5033 Commits

Author SHA1 Message Date
Abhishek Kona
85ad13be1a Port fix for Leveldb manifest writing bug from Open-Source
Summary:
Pretty much a blind copy of the patch in open source.
Hope to get this in before we make a release

Test Plan: make clean check

Reviewers: dhruba, heyongqiang

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7809
2013-01-10 12:06:03 -08:00
Abhishek Kona
41d7809212 Pom changes to make relase 1.5.7 for java.
Summary: Ran ./build_java.sh bump_version 1.5.7

Test Plan: automated change

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7833
2013-01-10 10:43:43 -08:00
Kosie van der Merwe
4e9d9d989f Fixed wrong assumption in Table::Open()
Summary:
`Table::Open()` assumes that `size` correctly describes the size of `file`, added a check that the footer is actually the right size and for good measure added assertions to `Footer::DecodeFrom()`.

This was discovered by running `valgrind ./db_test` and seeing that `Footer::DecodeFrom()` was accessing uninitialized memory.

Test Plan:
make clean check

ran `valgrind ./db_test` and saw DBTest.NoSpace no longer complains about a conditional jump being dependent on uninitialized memory.

Reviewers: dhruba, vamsi, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7815
2013-01-09 10:44:30 -08:00
Dhruba Borthakur
f881d6f226 Release 1.5.7.fb
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2013-01-09 10:26:03 -08:00
Abhishek Kona
2e1ad2c48b Remove unnecessary asserts in table/merger.cc
Summary:
The asserts introduced in https://reviews.facebook.net/D7629 are
wrong.

The direction of iteration is changed after the function call so they
assert's fail.

Test Plan: make clean check

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7827
2013-01-09 09:52:45 -08:00
Kosie van der Merwe
d8371ef1f6 Fixing some issues Valgrind found
Summary: Found some issues running Valgrind on `db_test` (there are still some outstanding ones) and fixed them.

Test Plan:
make check

ran `valgrind ./db_test` and saw that errors no longer occur

Reviewers: dhruba, vamsi, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7803
2013-01-08 12:16:40 -08:00
Kosie van der Merwe
4d339d7462 Fixed memory leak in ShardedLRUCache
Summary: `~ShardedLRUCache()` was empty despite `init()` allocating memory on the heap. Fixed the leak by freeing memory allocated by `init()`.

Test Plan:
make check

Ran valgrind on db_test before and after patch and saw leaked memory went down

Reviewers: vamsi, dhruba, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7791
2013-01-08 11:24:15 -08:00
Dhruba Borthakur
628dc2aad9 db_bench should use the default value for max_grandparent_overlap_factor.
Summary:
This was a peformance regression caused by https://reviews.facebook.net/D6729.
The default value of max_grandparent_overlap_factor was erroneously
set to 0 in db_bench.

This was causing compactions to create really really small files because the max_grandparent_overlap_factor was erroneously set to zero in the benchmark.

Test Plan: Run --benchmarks=overwrite

Reviewers: heyongqiang, emayanke, sheki, MarkCallaghan

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7797
2013-01-08 11:21:11 -08:00
Kosie van der Merwe
d6e873f22f Added clearer error message for failure to create db directory in DBImpl::Recover()
Summary:
Changed CreateDir() to CreateDirIfMissing() so a directory that already exists now causes and error.

Fixed CreateDirIfMissing() and added Env.DirExists()

Test Plan:
make check to test for regessions

Ran the following to test if the error message is not about lock files not existing
./db_bench --db=dir/testdb

After creating a file "testdb", ran the following to see if it failed with sane error message:
./db_bench --db=testdb

Reviewers: dhruba, emayanke, vamsi, sheki

Reviewed By: emayanke

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7707
2013-01-07 10:11:18 -08:00
Mark Callaghan
4069f66cc7 Add --seed, --read_range to db_bench
Summary:
Adds the option --seed to db_bench to specify the base for the per-thread RNG.
When not set each thread uses the same value across runs of db_bench which defeats
IO stress testing.

Adds the option --read_range. When set to a value > 1 an iterator is created and
each query done for the randomread benchmark will do a range scan for that many
rows. When not set or set to 1 the existing behavior (a point lookup) is done.

Fixes a bug where a printf format string was missing.

Test Plan: run db_bench

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7749
2013-01-07 09:56:10 -08:00
Kosie van der Merwe
8cd86a7be5 Fixing and adding some comments
Summary:
`MemTableList::Add()` neglected to mention that it took ownership of the reference held by its caller.

The comment in `MemTable::Get()` was wrong in describing the format of the key.

Test Plan: None

Reviewers: dhruba, sheki, emayanke, vamsi

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7755
2013-01-03 17:13:56 -08:00
Abhishek Kona
3f7af03a2d Use a priority queue to merge files.
Summary:
Use a std::priority_queue in merger.cc instead of doing a o(n) search
every time.
Currently only the ForwardIteration uses a Priority Queue.

Test Plan: make all check

Reviewers: dhruba

Reviewed By: dhruba

CC: emayanke, zshao

Differential Revision: https://reviews.facebook.net/D7629
2013-01-02 13:52:25 -08:00
Dhruba Borthakur
d7d43ae21a ExtendOverlappingInputs too slow for large databases.
Summary:
There was a bug in the ExtendOverlappingInputs method so that
the terminating condition for the backward search was incorrect.

Test Plan: make clean check

Reviewers: sheki, emayanke, MarkCallaghan

Reviewed By: MarkCallaghan

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7725
2013-01-02 13:19:06 -08:00
Dhruba Borthakur
2fc394a327 Do not compile thrift for fbcode build.
Summary:
1. The thrift libraries do not need to be built anyore.
2. SSE is dynamically detected via
   1aae609b92

Test Plan: compile and build

Reviewers: sheki, emayanke

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7665
2012-12-27 16:11:11 -08:00
Dhruba Borthakur
5b05417df3 Complication error when using gcc 4.7.1.
Summary:
There is a compilation error while using gcc 4.7.1.
util/ldb_cmd.cc:381:3: error: ‘leveldb::ReadOptions::ReadOptions’ names the constructor, not the type
util/ldb_cmd.cc:381:37: error: expected ‘;’ before ‘read_options’
util/ldb_cmd.cc:381:49: error: statement cannot resolve address of overloaded function

Test Plan: make clean check

Reviewers: sheki, emayanke, zshao

Reviewed By: emayanke

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7659
2012-12-27 12:38:20 -08:00
Zheng Shao
0f762ac573 ldb: Add command "ldb query" to support random read from the database
Summary: The queries will come from stdin.  One key per line.  The output will be in stdout, in the format of "<key> ==> <value>" if found, or "<key>" if not found.  "--hex" uses HEX-encoded keys and values in both input and output.

Test Plan: ldb query --db=leveldb_db --hex

Reviewers: dhruba, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7617
2012-12-26 20:37:42 -08:00
Zheng Shao
dcece4707e ldb: Fix incorrect arg parsing
Summary: We were ignoring additional chars at the end of an arg.  This can create confusion, e.g. --disable_wal=0 will act the same as --disable_wal without any warnings.

Test Plan:
Tried this:
[zshao@dev485 ~/git/rocksdb] ./ldb dump --statsAAA
Failed: Unknown argument:--statsAAA

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7635
2012-12-26 20:09:10 -08:00
Zheng Shao
04832dbc02 sst_dump: Fix incorrect cmd args parsing
Summary: The current parsing logic ignores any additional chars after the arg.

Test Plan: "./sst_dump --verify_checksumAAA" now outputs error.

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7611
2012-12-26 10:57:04 -08:00
Abhishek Kona
396c3aa08e Create a long running test to check GetUpdatesSince.
Summary:
Create an Executable with
* A thread to do Put's
* A thread to use GetUpdatesSince. Check if we miss any sequence
  Numbers.

Test Plan: It runs.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7383
2012-12-21 14:10:06 -08:00
Zheng Shao
127ee2e677 manifest_dump: Fix the help message and make it consistent
Summary: ldb uses --output_hex so make manifest_dump do the same thing.

Test Plan:
[zshao@dev485 ~/git/rocksdb] ./manifest_dump --output_hex --file=/data/users/zshao/test_leveldb/MANIFEST-000034
manifest_file_number 42 next_file_number 43 last_sequence 2311567 log_number 36  prev_log_number 0
--- level 0 --- version# 0 ---
--- level 1 --- version# 0 ---
--- level 2 --- version# 0 ---
--- level 3 --- version# 0 ---
 5:27788699['0000027F4FBE0000' @ 1 : 1 .. '11CE749602C90000' @ 160642 : 1]
 7:27785313['11CE773DA7E00000' @ 160643 : 1 .. '23A4C63EC55D0000' @ 321094 : 1]
 9:27784288['23A4D581FCD30000' @ 321095 : 1 .. '3576291D12D00000' @ 481428 : 1]
 38:64378271['35762BF0E0CE0000' @ 481429 : 1 .. '5E987E0604700000' @ 852910 : 1]
 39:64379046['5E987EB0BDD50000' @ 852911 : 1 .. '87C954330E840000' @ 1224603 : 1]
 40:10169201['87C95507E49C0000' @ 1224604 : 1 .. '8E48DC0933B70000' @ 1283317 : 1]
 21:27798825['8E48DFB0D7CE0000' @ 1283318 : 1 .. 'A00675F8AD7E0000' @ 1443826 : 1]
 23:27793751['A006777536E30000' @ 1443827 : 1 .. 'B1D1787FE8670000' @ 1604553 : 1]
 25:27801659['B1D179289BB30000' @ 1604554 : 1 .. 'C396D3A69DCE0000' @ 1765012 : 1]
 27:27792661['C396DA1E03B10000' @ 1765013 : 1 .. 'D55C9974FCC10000' @ 1925513 : 1]
 29:27789095['D55C9B47CBC00000' @ 1925514 : 1 .. 'E71F67D11CCC0000' @ 2085789 : 1]
 31:27793145['E71F7A667E740000' @ 2085790 : 1 .. 'F8D4712EF3D90000' @ 2246454 : 1]
 41:11246031['F8D4715916A70000' @ 2246455 : 1 .. 'FFFFFCAE97DF0000' @ 2311567 : 1]
--- level 4 --- version# 0 ---
--- level 5 --- version# 0 ---
--- level 6 --- version# 0 ---

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7575
2012-12-21 10:45:24 -08:00
Zheng Shao
01d57a33fe sst_dump: Add --output_hex option and output the same format as ldb dump
Summary: Now sst_dump has the same option --output_hex as "ldb dump" and also share the same output format.  So we can do "sst_dump ... | ldb load ..." for an experiment.

Test Plan:
[zshao@dev485 ~/git/rocksdb] ./sst_dump --file=/data/users/zshao/test_leveldb/000005.sst  --output_hex | head -n 2
0000027F4FBE00000101000000000000 ==> D901000000000000000057596F7520726563656976656420746F6461792773207370656369616C20676966742120436C69636B2041636365707420746F207669657720796F75722047696674206265666F72652069742064697361707065617273210000000000000000
000007F9C2D400000102000000000000 ==> D1010000000000000000544974277320676F6F6420746F206265204B696E67E280A6206F7220517565656E212048657265277320796F75722054756573646179204D79737465727920476966742066726F6D20436173746C6556696C6C65219B7B227A636F6465223A22633566306531633039663764222C227470223A22613275222C227A6B223A22663638663061343262666264303966383435666239626235366365396536643024246234506C345139382E734C4D33522169482D4F31315A64794C4B7A4F4653766D7863746534625F2A3968684E3433786521776C427636504A414355795F70222C227473223A313334343936313737357D0000000000000000

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7587
2012-12-21 10:44:19 -08:00
Zheng Shao
70f0f50731 ldb: file_size and write_buffer_size options.
Summary:
This allows ldb to control the write_buffer_size (which reflects to L0 file size) and file_size (which reflects to L1 file size).  Since the target_file_size_ratio is 1 by default, all other levels will also have the same file size as L1.

As part of the diff, I also cleaned up some unused code and help messages.

Test Plan: ./ldb load --db=/data/users/zshao/test_leveldb --file_size=64000000 --write_buffer_size=32000000 --create_if_missing --input_hex --disable_wal

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7569
2012-12-21 10:40:38 -08:00
Abhishek Kona
1aae609b92 Use CRC32 ss42 instruction. Load it dynamically.
Test Plan: make all check

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb, zshao

Differential Revision: https://reviews.facebook.net/D7503
2012-12-21 10:20:32 -08:00
Zheng Shao
7521a225d1 sst_dump: Error message should include the case that compression algorithms are not supported.
Summary: It took me almost a day to debug this. :(  Although I got to learn the file format as a by-product, this time could be saved if we have better error messages.

Test Plan: gmake clean all; sst_dump --hex --file=000005.sst

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7551
2012-12-20 15:29:51 -08:00
Dhruba Borthakur
551f01fb23 Unit test to test block format.
Summary: This is a standalone unit test to test the format of a block.

Test Plan: ./block_test

Reviewers: sheki

Reviewed By: sheki

Differential Revision: https://reviews.facebook.net/D7533
2012-12-20 14:55:07 -08:00
Zheng Shao
58d144444e Add libevent include and lib directories
Summary:
Without this fix, I see failures like this:
[zshao@dev1049 /data/users/zshao/rocksdb] . fbcode.gcc471.sh; gmake clean libleveldb.a
. . .
./thrift/lib/cpp/async/TEventUtil.h:22:32: fatal error: event.h: No such file or directory

Test Plan: . fbcode.gcc471.sh; make clean libleveldb.a

Reviewers: dhruba, emayanke, sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7497
2012-12-20 14:31:54 -08:00
Dhruba Borthakur
f4c2b7cf97 Enhance ReadOnly mode to process the all committed transactions.
Summary:
Leveldb has an api OpenForReadOnly() that opens the database
in readonly mode. This call had an option to not process the
transaction log.  This patch removes this option and always
processes all transactions that had been committed. It has
been done in such a way that it does not create/write to
any new files in the process. The invariant of "no-writes"
to the leveldb data directory is still true.

This enhancement allows multiple threads to open the same database
in readonly mode and access all trancations that were committed right
upto the OpenForReadOnly call.

I changed the public API to match the new semantics because
there are no users who are currently using this api.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7479
2012-12-19 16:30:46 -08:00
Zheng Shao
be9b862d47 ldb: add "ldb load" command
Summary: This command accepts key-value pairs from stdin with the same format of "ldb dump" command.  This allows us to try out different compression algorithms/block sizes easily.

Test Plan: dump, load, dump, verify the data is the same.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7443
2012-12-19 01:45:59 -08:00
Abhishek Kona
2585979fd7 Release 1.5.6 for Java code + Script to automate it.
Test Plan: it compiles and deploys.

Reviewers: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7341
2012-12-17 12:11:11 -08:00
Dhruba Borthakur
3d1e92b05a Enhancements to rocksdb for better support for replication.
Summary:
1. The OpenForReadOnly() call should not lock the db. This is useful
so that multiple processes can open the same database concurrently
for reading.
2. GetUpdatesSince should not error out if the archive directory
does not exist.
3. A new constructor for WriteBatch that can takes a serialized
string as a parameter of the constructor.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7449
2012-12-17 11:40:19 -08:00
Kosie van der Merwe
62d48571de Added meta-database support.
Summary:
Added kMetaDatabase for meta-databases in db/filename.h along with supporting
fuctions.
Fixed switch in DBImpl so that it also handles kMetaDatabase.
Fixed DestroyDB() that it can handle destroying meta-databases.

Test Plan: make check

Reviewers: sheki, emayanke, vamsi, dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7245
2012-12-17 11:26:59 -08:00
Abhishek Kona
2f0585fb97 Fix a bug. Where DestroyDB deletes a non-existant archive directory.
Summary:
C tests would fail sometimes as DestroyDB would return a Failure Status
message when deleting an archival directory which was not created
(WAL_ttl_seconds = 0).

Fix: Ignore the Status returned on Deleting Archival Directory.

Test Plan: * make check

Reviewers: dhruba, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7395
2012-12-17 10:25:26 -08:00
Zheng Shao
3d9ff0e921 ldb: fix dump command to pad HEX output chars with 0.
Summary: The old code was omitting the 0 if the char is less than 16.

Test Plan:
Tried the following program:
int main() {
  unsigned char c = 1;
  printf("%X\n", c);
  printf("%02X\n", c);
  return 0;
}
The output is:
1
01

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7437
2012-12-16 16:55:38 -08:00
Zheng Shao
7dc8bb710e ldb: support --block_size=<4096|65536|...> and --auto_compaction=<0|1>
Summary: This allows us to use ldb to do more experiments like block_size changes.

Test Plan: run it by hand.

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7431
2012-12-16 09:00:14 -08:00
Zheng Shao
c28097538a manifest_dump: Add --hex=1 option
Summary: Without this option, manifest_dump does not print binary keys for files in a human-readable way.

Test Plan:
./manifest_dump --hex=1 --verbose=0 --file=/data/users/zshao/fdb_comparison/leveldb/fbobj.apprequest-0_0_original/MANIFEST-000002
manifest_file_number 589 next_file_number 590 last_sequence 2311567 log_number 543  prev_log_number 0
--- level 0 --- version# 0 ---
 532:1300357['0000455BABE20000' @ 2183973 : 1 .. 'FFFCA5D7ADE20000' @ 2184254 : 1]
 536:1308170['000198C75CE30000' @ 2203313 : 1 .. 'FFFCF94A79E30000' @ 2206463 : 1]
 542:1321644['0002931AA5E50000' @ 2267055 : 1 .. 'FFF77B31C5E50000' @ 2270754 : 1]
 544:1286390['000410A309E60000' @ 2278592 : 1 .. 'FFFE470A73E60000' @ 2289221 : 1]
 538:1298778['0006BCF4D8E30000' @ 2217050 : 1 .. 'FFFD77DAF7E30000' @ 2220489 : 1]
 540:1282353['00090D5356E40000' @ 2231156 : 1 .. 'FFFFF4625CE40000' @ 2231969 : 1]
--- level 1 --- version# 0 ---
 510:2112325['000007F9C2D40000' @ 1782099 : 1 .. '146F5B67B8D80000' @ 1905458 : 1]
 511:2121742['146F8A3023D60000' @ 1824388 : 1 .. '28BC8FBB9CD40000' @ 1777993 : 1]
 512:801631['28BCD396F1DE0000' @ 2080191 : 1 .. '3082DBE9ADDB0000' @ 1989927 : 1]

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7425
2012-12-16 08:58:28 -08:00
Abhishek Kona
806d4d9346 fixing linters.
Summary:
old version of linters use "lint_engine" instead of "lint.engine"
Some bookeeping in gitignore.
Reviewers: abhishekk
2012-12-14 14:05:27 -08:00
Abhishek Kona
2ba866e0c5 GetSequence API in write batch.
Summary:
WriteBatch is now used by the GetUpdatesSinceAPI. This API is external
and will be used by the rocks server. Rocks Server and others will need
to know about the Sequence Number in the WriteBatch. This public method
will allow for that.

Test Plan: make all check.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7293
2012-12-12 22:21:10 -08:00
Dhruba Borthakur
d0a30935c6 Expose the serialized string that represents a WriteBatch.
Summary:
Expose the serialized string that represents a WriteBatch. This
is helpful to replicate a writebatch operation from one machine
to another.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7317
2012-12-12 16:25:52 -08:00
Abhishek Kona
f20383f095 Make Java Client compilable.
Summary:
Debug and ported changes from the Open Source Github repo to our repo.
Wrote a script to easy build the java Library. future compiling java lib should just be running this script.

Test Plan: it compiles.

Reviewers: dhruba, leveldb

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7323
2012-12-12 14:07:52 -08:00
Abhishek Kona
22c283625b Fix Bug in Binary Search for files containing a seq no. and delete Archived Log Files during Destroy DB.
Summary:
* Fixed implementation bug in Binary_Searvch introduced in https://reviews.facebook.net/D7119
* Binary search is also overflow safe.
* Delete archive log files and archive dir during DestroyDB

Test Plan: make check

Reviewers: dhruba

CC: kosievdmerwe, emayanke

Differential Revision: https://reviews.facebook.net/D7263
2012-12-11 16:15:02 -08:00
Dhruba Borthakur
24fc379273 An public api to fetch the latest transaction id.
Summary:
Implement a interface to retrieve the most current transaction
id from the database.

Test Plan: Added unit test.

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7269
2012-12-10 16:04:19 -08:00
sheki
dcd919a0a4 Port various compaction options to Java.
Summary:
Porting various options, mostly related to Multi-threaded compaction to
Java.

Test Plan:
mvn test.
No clear plan on how else test.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb, emayanke

Differential Revision: https://reviews.facebook.net/D7221
2012-12-10 10:53:48 -08:00
Abhishek Kona
1c6742e32f Refactor GetArchivalDirectoryName to filename.h
Summary:
filename.h has functions to do similar things.
Moving code away from db_impl.cc

Test Plan: make check

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7251
2012-12-10 10:51:07 -08:00
Dhruba Borthakur
38671c4d54 Fix a race condition while processing tasks by background threads.
Summary:
Suppose you submit 100 background tasks one after another. The first
enqueu task finds that the queue is empty and wakes up one worker thread.
Now suppose that all remaining 99 work items are enqueued, they do not
wake up any worker threads because the queue is already non-empty.
This causes a situation when there are 99 tasks in the task queue but
only one worker thread is processing a task while the remaining
worker threads are waiting.
The fix is to always wakeup one worker thread while enqueuing a task.

I also added a check to count the number of elements in the queue
to help in debugging.

Test Plan: make clean check.

Reviewers: chip

Reviewed By: chip

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7203
2012-12-09 17:15:27 -08:00
Zheng Shao
768edfaaed ldb: Add compression and bloom filter options.
Summary:
Added the following two options:
[--bloom_bits=<int,e.g.:14>]
[--compression_type=<no|snappy|zlib|bzip2>]

These options will be used when ldb opens the leveldb database.

Test Plan: Tried by hand for both success and failure cases. We do need a test framework.

Reviewers: dhruba, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7197
2012-12-07 14:26:37 -08:00
Abhishek Kona
8055008909 GetUpdatesSince API to enable replication.
Summary:
How it works:
* GetUpdatesSince takes a SequenceNumber.
* A LogFile with the first SequenceNumber nearest and lesser than the requested Sequence Number is found.
* Seek in the logFile till the requested SeqNumber is found.
* Return an iterator which contains logic to return record's one by one.

Test Plan:
* Test case included to check the good code path.
* Will update with more test-cases.
* Feedback required on test-cases.

Reviewers: dhruba, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7119
2012-12-07 11:42:13 -08:00
Kosie van der Merwe
f69e9f3e04 Fixed off by 1 in tests.
Summary: Added 1 to indices where I shouldn't have so overrun array.

Test Plan: make check

Reviewers: sheki, emayanke, vamsi, dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7227
2012-12-07 10:48:46 -08:00
Kosie van der Merwe
0eb0c9bb82 Added methods to write small ints to bit streams.
Summary: Added BitStreamPutInt() and BitStreamGetInt() which take a stream of chars and can write integers of arbitrary bit sizes to that stream at arbitrary positions. There are also convenience versions of these functions that take std::strings and leveldb::Slices.

Test Plan: make check

Reviewers: sheki, vamsi, dhruba, emayanke

Reviewed By: vamsi

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7071
2012-12-07 10:42:19 -08:00
Dhruba Borthakur
c847a31727 Print compaction score for every compaction run.
Summary:
A compaction is picked based on its score. It is useful to
print the compaction score in the LOG because it aids in
debugging. If one looks at the logs, one can find out why
a compaction was preferred over another.

Test Plan: make clean check

Differential Revision: https://reviews.facebook.net/D7137
2012-12-04 10:03:47 -08:00
Dhruba Borthakur
6eb5ed9a49 rocksdb README file.
Summary:
rocksdb README file.

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2012-11-29 22:39:08 -08:00