andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
115 Commits 152 Branches 236 Tags 271 MiB
e56b2c5a31
Commit Graph

115 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Dhruba Borthakur
e56b2c5a31 Prevent concurrent multiple opens of leveldb database. 
Summary:
The fcntl call cannot detect lock conflicts when invoked multiple times
from the same thread.
Use a static lockedFile Set to record the paths that are locked.
A lockfile request checks to see if htis filename already exists in
lockedFiles, if so, then it triggers an error. Otherwise, it inserts
the filename in the lockedFiles Set.
A unlock file request verifies that the filename is in the lockedFiles
set and removes it from lockedFiles set.

Test Plan: unit test attached

Reviewers: heyongqiang

Reviewed By: heyongqiang

Differential Revision: https://reviews.facebook.net/D4755
 2012-08-20 23:55:04 -07:00
heyongqiang
deb1a1fa9b add disable wal to db_bench 
Summary:
as subject.

./db_bench --benchmarks=fillrandom --num=1000000 --disable_data_sync=1 --write_buffer_size=50000000 --target_file_size_base=100000000 --disable_wal=1

LevelDB:    version 1.4
Date:       Sun Aug 19 16:01:59 2012
CPU:        8 * Intel(R) Xeon(R) CPU           L5630  @ 2.13GHz
CPUCache:   12288 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
------------------------------------------------
fillrandom   :       4.591 micros/op 217797 ops/sec;   24.1 MB/s

./db_bench --benchmarks=fillrandom --num=1000000 --disable_data_sync=1 --write_buffer_size=50000000 --target_file_size_base=100000000

LevelDB:    version 1.4
Date:       Sun Aug 19 16:02:54 2012
CPU:        8 * Intel(R) Xeon(R) CPU           L5630  @ 2.13GHz
CPUCache:   12288 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
------------------------------------------------
fillrandom   :       3.696 micros/op 270530 ops/sec;   29.9 MB/s

Test Plan: db_bench

Reviewers: dhruba

Differential Revision: https://reviews.facebook.net/D4767
 2012-08-19 22:37:51 -07:00
Dhruba Borthakur
2aa514ec8c Utility to dump manifest contents. 
Summary:
./manifest_dump --file=/tmp/dbbench/MANIFEST-000002

Output looks like

manifest_file_number 30 next_file_number 31 last_sequence 388082 log_number 28  prev_log_number 0
--- level 0 ---
--- level 1 ---
--- level 2 ---
 5:3244155['0000000000000000' @ 1 : 1 .. '0000000000028220' @ 28221 : 1]
 7:3244177['0000000000028221' @ 28222 : 1 .. '0000000000056441' @ 56442 : 1]
 9:3244156['0000000000056442' @ 56443 : 1 .. '0000000000084662' @ 84663 : 1]
 11:3244178['0000000000084663' @ 84664 : 1 .. '0000000000112883' @ 112884 : 1]
 13:3244158['0000000000112884' @ 112885 : 1 .. '0000000000141104' @ 141105 : 1]
 15:3244176['0000000000141105' @ 141106 : 1 .. '0000000000169325' @ 169326 : 1]
 17:3244156['0000000000169326' @ 169327 : 1 .. '0000000000197546' @ 197547 : 1]
 19:3244178['0000000000197547' @ 197548 : 1 .. '0000000000225767' @ 225768 : 1]
 21:3244155['0000000000225768' @ 225769 : 1 .. '0000000000253988' @ 253989 : 1]
 23:3244179['0000000000253989' @ 253990 : 1 .. '0000000000282209' @ 282210 : 1]
 25:3244157['0000000000282210' @ 282211 : 1 .. '0000000000310430' @ 310431 : 1]
 27:3244176['0000000000310431' @ 310432 : 1 .. '0000000000338651' @ 338652 : 1]
 29:3244156['0000000000338652' @ 338653 : 1 .. '0000000000366872' @ 366873 : 1]
--- level 3 ---
--- level 4 ---
--- level 5 ---
--- level 6 ---

Test Plan: run on test directory created by dbbench

Reviewers: heyongqiang

Reviewed By: heyongqiang

CC: hustliubo

Differential Revision: https://reviews.facebook.net/D4743
 2012-08-17 22:36:59 -07:00
heyongqiang
680e571c4c add compaction log Summary: 
Summary:
add compaction summary to log

log looks like:

2012/08/17-18:18:32.557334 7fdcaa2bb700 Compaction summary: Base level 0, input file:[11 9 7 ],[]

Test Plan: tested via db_test

Reviewers: dhruba

Differential Revision: https://reviews.facebook.net/D4749
 2012-08-17 19:29:39 -07:00
heyongqiang
29c99fce95 Merge branch 'master' of https://github.com/facebook/leveldb 2012-08-17 16:23:15 -07:00
heyongqiang
20ee76bd34 use ts as suffix for LOG.old files 
Summary: as subject and only maintain 10 log files.

Test Plan: new test in db_test

Reviewers: dhruba

Differential Revision: https://reviews.facebook.net/D4731
 2012-08-17 16:22:04 -07:00
Dhruba Borthakur
b56ff5ea38 Expose new configration variables via the thrift api 
Summary: Expose new configration variables via the thrift api. when compiing for fbcode, always build thrift server

Test Plan: none.

Reviewers: heyongqiang

Reviewed By: heyongqiang

Differential Revision: https://reviews.facebook.net/D4689
 2012-08-17 10:29:12 -07:00
Dhruba Borthakur
7c0b5ec54a leveldb thrift server uses an incorrect root directory 
Summary: The offsets used to speficy hostname and dbdir from the command line were not handled correctly.

Test Plan: none.

Reviewers: heyongqiang

Reviewed By: heyongqiang

Differential Revision: https://reviews.facebook.net/D4683
 2012-08-16 11:10:08 -07:00
heyongqiang
f16e393658 add more options to db_ben 
Summary: as subject

Test Plan: run db_bench with new options

Reviewers: dhruba

Differential Revision: https://reviews.facebook.net/D4677
 2012-08-15 17:42:33 -07:00
heyongqiang
fcb2ea4715 disable data sync options needs to be checked when doing level-0 dump Summary: 
Summary: as subject

Test Plan: use db_bench

Reviewers: dhruba

Differential Revision: https://reviews.facebook.net/D4671
 2012-08-15 16:39:02 -07:00
Dhruba Borthakur
fe6119bd3a "make all check" fails 
Summary:
If I run "make all check" it fails to build the leveldb-thrift-server tests. This is because the vanilla build is not supposed to build thrift and/or hdfs extensions. Remove the server test from auto-running.

The thrift/hdfs build instructions are more elaborate and I would like to avoid it for embedded database builds.

Test Plan: build and run

Reviewers: heyongqiang

Reviewed By: heyongqiang

Differential Revision: https://reviews.facebook.net/D4641
 2012-08-14 15:55:38 -07:00
Dhruba Borthakur
c3096afd61 Introduce a new option disableDataSync for opening the database. If this is set to true, then the data written to newly created data files are not sycned to disk, instead depend on the OS to flush dirty data to stable storage. This option is good for bulk 
Test Plan:
manual tests

Task ID: #

Blame Rev:

Differential Revision: https://reviews.facebook.net/D4515
 2012-08-03 15:23:53 -07:00
Dhruba Borthakur
c44be54dd2 Make leveldb server not exit immediately. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-08-03 01:00:35 -07:00
Dhruba Borthakur
2d04ab5382 Latest release is 1.4.1.fb 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-08-03 00:33:47 -07:00
Dhruba Borthakur
88c515b6ba Implement taoAssocRangeGet(). 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-08-03 00:27:48 -07:00
Dhruba Borthakur
0c98fdcf27 Encode id1Type, id2Type and data version the assoc payload. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-08-02 14:09:13 -07:00
Dhruba Borthakur
a3a8f83482 Fix "gcc -Wall" warnings for assoc-thrift code. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-08-01 17:28:35 -07:00
Dhruba Borthakur
d82cfe1241 Checkin pre-compiled snappy libraries for leveldb. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-08-01 16:55:39 -07:00
Dhruba Borthakur
b40ad060e0 Implement the FB-Assoc API via thrift. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-08-01 16:18:24 -07:00
Dhruba Borthakur
4abf94eeb4 Check in compiled binaries for snappy. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-07-17 00:44:35 -07:00
Dhruba Borthakur
71fa013ad5 Modify symbolic links for snappy builds. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-07-17 00:32:38 -07:00
Dhruba Borthakur
4274e6c6db Added README for snappy builds 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-07-17 00:30:12 -07:00
Dhruba Borthakur
21f2fe4419 Build snappy for leveldb. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-07-17 00:18:23 -07:00
Dhruba Borthakur
d5503208cf Share a single cache for all the DBs served by this server. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-07-11 14:08:46 -07:00
Dhruba Borthakur
bfaa1bd26c Cleanup thrift header files. 
Move thrift api  from thrift/leveldb.thrift to thrift/if/leveldb.thrift

Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-07-10 10:19:49 -07:00
Dhruba Borthakur
d11b637f34 bits_per_key is already configurable. It defines how many bloom bits will be used for every key in the database. 
My change in this patch is to make the Hash code that is used for blooms to be confgurable. In fact,
one can specify a modified HashCode that inspects only parts of the Key to generate the Hash (used
by booms).

Test Plan: none

Differential Revision: https://reviews.facebook.net/D4059
 2012-07-09 23:06:07 -07:00
Dhruba Borthakur
80c663882a Create leveldb server via Thrift. 
Summary:
First draft.
Unit tests pass.

Test Plan: unit tests attached

Reviewers: heyongqiang

Reviewed By: heyongqiang

Differential Revision: https://reviews.facebook.net/D3969
 2012-07-07 09:42:39 -07:00
heyongqiang
22ee777f68 add flush interface to DB 
Summary: as subject. The flush will flush everything in the db.

Test Plan: new test in db_test.cc

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D4029
 2012-07-06 12:11:19 -07:00
heyongqiang
a347d4ac0d add disable WAL option 
Summary: add disable WAL option

Test Plan: new testcase in db_test.cc

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D4011
 2012-07-05 15:06:56 -07:00
heyongqiang
7600228072 fix compile warning 
Summary: as subject

Test Plan: compile

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D3957
 2012-07-02 17:37:45 -07:00
heyongqiang
daa816c4a0 add bzip2 compression 
Summary: add bzip2 compression

Test Plan: testcases in table_test

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D3909
 2012-06-29 10:27:28 -07:00
heyongqiang
054a5657f8 add zlib compression 
Summary: add zlib compression

Test Plan: Will add more testcases

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D3873
 2012-06-28 16:28:57 -07:00
heyongqiang
4e4b6812ff Make some variables configurable for each db instance 
Summary:
Make configurable 'targetFileSize', 'targetFileSizeMultiplier',
'maxBytesForLevelBase', 'maxBytesForLevelMultiplier',
'expandedCompactionFactor', 'maxGrandParentOverlapFactor'

Test Plan: N/A

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D3801
 2012-06-27 14:36:31 -07:00
Dhruba Borthakur
2067d22038 Support for java access to leveldb via leveldbjni. 
Test Plan: unit tests.

Differential Revision: https://reviews.facebook.net/D3735
 2012-06-20 15:17:03 -07:00
Dhruba Borthakur
636e3ec603 Release version 1.4.0.fb. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-06-19 01:47:15 -07:00
Dhruba Borthakur
e9bc777163 While creating the leveldb shared library, use the shared library for jemalloc. 
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
 2012-06-19 00:56:07 -07:00
Dhruba Borthakur
7e6c9b91a5 A version of snappy that uses jemalloc and gcc-4.6.2, It is here for ease of use by leveldb code. 
Test Plan: Run benchmark

Differential Revision: https://reviews.facebook.net/D3687
 2012-06-15 13:54:30 -07:00
Dhruba Borthakur
a35e574344 Make Leveldb save data into HDFS files. You have to set USE_HDFS in your environment variable to compile leveldb with HDFS support. 
Test Plan: Run benchmark.

Differential Revision: https://reviews.facebook.net/D3549
 2012-06-14 00:29:01 -07:00
Dhruba Borthakur
338939e5c1 Print log message when we are throttling writes. 
Summary:
Added option --writes=xxx to specify the number of keys that we want to overwrite in the benchmark.

Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Reviewers: adsharma

CC: sc

Differential Revision: https://reviews.facebook.net/D3465
 2012-06-01 14:03:37 -07:00
Dhruba Borthakur
f50ece60c7 Fix table-cache size bug, gather table-cache statistics and prevent readahead done by fs. Summary: 
Summary:
The db_bench test was not using the specified value for the max-file-open. Fixed.

The fs readhead is switched off.

Gather statistics about the table cache and print it out at the end of the tets run.

Test Plan: Revert Plan:

Reviewers: adsharma, sc

Reviewed By: adsharma

Differential Revision: https://reviews.facebook.net/D3441
 2012-05-30 16:42:45 -07:00
Dhruba Borthakur
8f293b68a9 Support --bufferedio=[0,1] from db_bench. If bufferedio = 0, then the read code path clears the OS page cache after the IO is completed. The default remains as bufferedio=1 
Summary:
Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Differential Revision: https://reviews.facebook.net/D3429
 2012-05-29 13:29:44 -07:00
Dhruba Borthakur
33a3c6ff6c Ability to make the benchmark issue a large number of IOs. This is helpful to populate many gigabytes of data for benchmarking at scale. 
Summary:
Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Differential Revision: https://reviews.facebook.net/D3333
 2012-05-22 12:20:09 -07:00
Dhruba Borthakur
3b86a51cb1 Ability to switch on checksum verification from benchmark. 
Summary:
Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Differential Revision: https://reviews.facebook.net/D3309
 2012-05-19 00:13:50 -07:00
Dhruba Borthakur
a2a0e358cb Add support to specify the number of shards for the Block cache. By default, the block cache is sharded into 16 parts. 
Summary:
Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Differential Revision: https://reviews.facebook.net/D3273
 2012-05-16 17:23:49 -07:00
Arun Sharma
95af128225 SSE4 optimization 
Summary:
This speeds up CRC computation significantly on
hardware that supports it. Enabled via -msse4.

Note: the binary won't be usable on older CPUs
that don't support the instruction.

Test Plan: crc32c_test

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D3201
 2012-05-15 10:10:01 -07:00
Dhruba Borthakur
8d41351666 Ability to switch to gcc 4.1.6 and also use jemalloc. Thanks for Chip for large amounts of help. 
Summary:
Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Reviewers: chip

Reviewed By: chip

CC: sc, adsharma

Differential Revision: https://reviews.facebook.net/D3225
 2012-05-14 22:15:09 -07:00
Arun Sharma
921a48428e Optimize for lp64 
Summary:
Some code reorganization in-preparation for replacing with a hardware
instruction.

* Use u64 for some of the key types
* Use an ALIGN macro so code is easier to read

Test Plan: crc32c_test

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D3135
 2012-05-14 15:40:11 -07:00
Dhruba Borthakur
37d0dcb9b1 Use the elapsed time (instead of the per-thread time) to compute ops/sec. 
Summary:
Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Differential Revision: https://reviews.facebook.net/D3147
 2012-05-11 12:43:31 -07:00
Arun Sharma
90b2924fb2 skiplist: optimize for sequential insert pattern 
Summary:
skiplist doesn't cache the location of the last insert and becomes
CPU bound when the input data has sequential keys.

Notes on thread safety: ::Insert() already requires external
synchronization. So this change is not making it any worse.

Test Plan: skiplist_test

Reviewers: dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D3129
 2012-05-11 09:57:40 -07:00
Dhruba Borthakur
cc6c32535a Support arcdiff. 
Summary:
Task ID: #

Blame Rev:

Test Plan: Revert Plan:

Differential Revision: https://reviews.facebook.net/D3105
 2012-05-09 23:35:05 -07:00