Commit Graph

4909 Commits

Author SHA1 Message Date
Yi Wu
792762c42c Split db_test.cc
Summary: Split db_test.cc into several files. Moving several helper functions into DBTestBase.

Test Plan: make check

Reviewers: sdong, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba, andrewkr, kradhakrishnan, yhchiang, leveldb, sdong

Differential Revision: https://reviews.facebook.net/D56715
2016-04-18 09:42:50 -07:00
Andrew Kryczka
40b840f294 Delete deprecated *BackupableDB interface for backups
Summary:
This interface is redundant and has been deprecated for a while.
It's also unused internally. Let's delete it.

I moved the comments to the corresponding functions in BackupEngine/
BackupEngineReadOnly. This caused the diff tool to not work cleanly.

Test Plan:
unit tests

  $ ./backupable_db_test

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56331
2016-04-18 09:04:14 -07:00
Yi Wu
6affd45d84 Make more tests run in parallel
Summary:
Generate t/run-* scripts to run tests in $PARALLEL_TEST separately, then make check_0 rule execute all of them.

Run `time make check` after running `make all`.
master: 71 sec
with this diff: 63 sec.

It seems moving more tests to $PARALLEL_TEST doesn't help improve test time though.

Test Plan:
Run the following
  make check
  J=16 make check
  J=1 make check
  make valgrind_check
  J=1 make valgrind_check
  J=16 make_valgrind_check

Reviewers: IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: leveldb, kradhakrishnan, dhruba, andrewkr, yhchiang

Differential Revision: https://reviews.facebook.net/D56805
2016-04-17 22:34:56 -07:00
Victor Tyutyunov
47833e0ab9 Merge branch 'master' of github.com:facebook/rocksdb 2016-04-16 01:48:17 -07:00
Victor Tyutyunov
e5c614e1df Fixing snapshot 0 assertion
Summary:
Solution is not to change db sequence number to start from 1 because 0 value is used in multiple other places.
Fix covers only compact_iterator::findEarliestVisibleSnapshot with updated logic to support snapshot's numbering starting from 0.

Test Plan:
run:
  make all check

it should pass all tests

Reviewers: IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: lgalanis, mgalushka, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56601
2016-04-16 01:47:15 -07:00
Victor Tyutyunov
1b1adebe8f Fixing snapshot 0 assertion
Test Plan: TBD

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56601

Cosmetic changes and comment update
2016-04-16 01:45:31 -07:00
sdong
6d436a3f85 DBTest.HardLimit made more deterministic
Summary: In DBTest.HardLimit, multiple flushes may merge into one, based on thread scheduling. Avoid it by waiting each flush to finish before generating the next one.

Test Plan: Run test in parallel several times and see it doesn't fail any more.

Reviewers: yhchiang, kradhakrishnan, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: yiwu, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56853
2016-04-15 17:36:57 -07:00
Islam AbdelRahman
994d9bc82b Make parallel valgrind watch-log more readable
Summary:
Before
{F1131675}

After
{F1131681}

This will have no effect on normal make check

Test Plan: make watch-log

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56847
2016-04-15 16:21:03 -07:00
sdong
9d35ae649e Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more deterministic
Summary: DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels fails one in about 30 runs when running in parallel. We wait for compaction after each flush to make the compaction behavior deterministic.

Test Plan: Run the test 1000 times in parallel and it still passes.

Reviewers: yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: kradhakrishnan, yiwu, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56841
2016-04-15 16:16:53 -07:00
sdong
cea8ed9702 Fix backupable_db_test test cases that can't run by itself
Summary:
Several of backupable_db_test fails if running standalone, because of directory missing. Fix it by:
(1) garbage collector skips shared directory if it doesn't exit
(2) BackupableDBTest.Issue921Test to create the parent directory of the backup directory fist.

Test Plan: Run the tests individually and make sure they pass

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56829
2016-04-15 15:48:57 -07:00
sdong
4b6833aec1 Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too.
Summary: It is useful to print out IO stats in flush jobs too. Extend options.compaction_measure_io_stats to flush jobs and raname it.

Test Plan: Try db_bench and see the stats are printed out.

Reviewers: yhchiang

Reviewed By: yhchiang

Subscribers: kradhakrishnan, yiwu, IslamAbdelRahman, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56769
2016-04-15 10:22:18 -07:00
sdong
3894603fef Allow valgrind_check to run in parallel
Summary:
Extend "J=<parallel>" to valgrind_check.
For DBTest, modify the script to run valgrind. For other tests, prefix launch command with valgrind.

Test Plan: Run valgrind_check with J=1 and J>1 and make sure tests run under valgrind. Manually change codes to introduce memory leak and make sure "make watch-log" correctly report it.

Reviewers: yhchiang, yiwu, andrewkr, kradhakrishnan, IslamAbdelRahman

Reviewed By: kradhakrishnan, IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56727
2016-04-14 18:33:04 -07:00
Dmitri Smirnov
c9d668c584 Fix unit tests issues on Windows (#1078) 2016-04-14 17:33:53 -07:00
krad
083cadc7e9 Minor fix to Java sandcastle job definition
Summary:
proxy config tools are not available in sandcastle environment. Hard
coding the proxy information in the job.

Test Plan: Run manually

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56775
2016-04-14 15:05:54 -07:00
Sherlock
80b74a6c64 Include ldb tool in the windows build (#914) 2016-04-14 10:50:34 -07:00
krad
7c14d11eb9 Minor fix to java build job
Summary:
Java build is downloading junit when not available. Without the proxy
settings exported that would not work.

Test Plan: Run manually

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56751
2016-04-14 10:46:02 -07:00
sdong
535af525d6 BlockBasedTable::PrefixMayMatch() to skip index checking if we can't find a filter block.
Summary:
In the case where we can't find a filter block, there is not much benefit of doing the binary search and see whether the index key has the prefix. With the change, we blindly return true if we can't get the filter.
It also fixes missing row cases for reverse comparator with full bloom.

Test Plan: Add a test case that used to fail.

Reviewers: yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: kradhakrishnan, yiwu, hermanlee4, yoshinorim, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56697
2016-04-13 19:06:48 -07:00
krad
09be5cad5b Minor fix to sandcastle java job definition
Summary: System was not able to locate javac

Test Plan: Run command manually and observe output

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56709
2016-04-13 15:59:05 -07:00
Islam AbdelRahman
1aeca97337 Release RocksDB 4.7
Summary: Bump the version and update HISTORY.md

Test Plan: none

Reviewers: yhchiang, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56469
2016-04-13 15:03:08 -07:00
Islam AbdelRahman
dfc3de8b7d Split Travis unittests Job
Summary:
  - Update Makefile check_some command to run a subset of the tests
  - Update travis config to split the unittests job into 2 jobs
     -- job testing db_test, db_test2
     -- job testing the rest of the unittests

Test Plan:
Run the new travis.yml on my own branch
https://travis-ci.org/facebook/rocksdb/builds/122691453

Reviewers: kradhakrishnan, yhchiang, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56673
2016-04-13 14:22:29 -07:00
krad
c2c8fe47f0 Add Java job for sandcastle
Summary: Adding java build job definition to sandcastle.

Test Plan: ./build_tools/rocksdb-lego-determinator java_build

Reviewers: yhchiang, sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56679
2016-04-13 11:40:51 -07:00
Islam AbdelRahman
19ef3de57e Fix ManualCompactionPartial test flakiness
Summary: The reason for this test flakiness is that we try to verify that number of files in L0 is 3 after flushing the 3rd file although we may have a compaction running in the background that may finish before we do the check and the 3 L0 files are converted to 1 L1 file

Test Plan: Run a modified version of the test that sleep before doing the check

Reviewers: sdong, andrewkr, kradhakrishnan, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56643
2016-04-13 10:38:45 -07:00
Jay Edgar
b345b36620 Add a minimum value for the refill bytes per period value
Summary: If the user specified a small enough value for the rate limiter's bytes per second, the calculation for the number of refill bytes per period could become zero which would effectively cause the server to hang forever.

Test Plan: Existing tests

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56631
2016-04-13 09:01:42 -07:00
sdong
dff4c48ede BlockBasedTable::PrefixMayMatch: no need to find data block after full bloom checking
Summary:
Full block checking should be a good enough indication of prefix existance. No need to further check data block.
This also fixes wrong results when using prefix bloom and reverse bitwise comparator.

Test Plan: Will add a unit test.

Reviewers: yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: hermanlee4, yoshinorim, yiwu, kradhakrishnan, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56625
2016-04-12 16:25:54 -07:00
krad
0353b853cb Propagate sandcastle run error to UI
Summary:
Currently the code does not propagate the sandcastle precommit test run
error status to UI. This can confuse the developer when searching for errors.

With this change, all success should be in green and all errors should be in red

Test Plan: Submit the diff (and hopefully something will fail)

Reviewers: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56595
2016-04-12 16:18:37 -07:00
Islam AbdelRahman
b885f33a56 Parallelize travis jobs
Summary:
Update travis.yml to split the work into 3 sub jobs
- running unittests
- Building and testing java
- building ROCKSDB_LITE

Test Plan: https://travis-ci.org/facebook/rocksdb/builds/122433169

Reviewers: sdong, yhchiang, andrewkr, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56607
2016-04-12 15:05:56 -07:00
Hyunyoung Lee
71303e04e7 Update db_bench_tool.cc (#1073)
* Update db_bench_tool.cc

I fixed the wrong letters, LevelDB -> rocksDB, because I thought of LevelDB as the wrong presentation.

the following show my fix :

fprintf(stderr, "LevelDB:    version %d.%d\n",
            kMajorVersion, kMinorVersion);

----------------->
fprintf(stderr, "rocksDB:    version %d.%d\n",
            kMajorVersion, kMinorVersion);

* Update db_bench_tool.cc

* Update db_bench_tool.cc
2016-04-12 17:05:09 -04:00
sdong
63cf15bb9f Fix option settable tests
Summary: In option settable tests, bytes for pointers are not all skipped, so that they may be the same as the special character and cause false positive.

Test Plan: Run the test. Manually verify the issue is not there any more.

Reviewers: IslamAbdelRahman, andrewkr

Reviewed By: IslamAbdelRahman

Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56553
2016-04-12 14:03:35 -07:00
Shu Zhang
e208575b2d using java7 in runtime for hdfs env (#1072) 2016-04-12 00:08:26 -04:00
Yueh-Hsuan Chiang
13e6c8e97a Relax an assertion in Compaction::ShouldStopBefore
Summary:
In some case, it is possible to have two concesutive SST files might sharing
same boundary keys.  However, in the assertion in Compaction::ShouldStopBefore,
it exclude such possibility.

This patch fix this issue by relaxing the assertion to allow the equal case.

Test Plan: rocksdb tests

Reviewers: IslamAbdelRahman, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D55875
2016-04-11 20:15:52 -07:00
Yueh-Hsuan Chiang
ae21d71e94 Fixed a bug in RocksDB Statistics where flush is considered as compaction
Summary: Fixed a bug in RocksDB Statistics where flush is considered as compaction

Test Plan: unit test

Reviewers: sdong, IslamAbdelRahman, rven, kradhakrishnan, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56271
2016-04-11 19:59:25 -07:00
Andrew Kryczka
8e0e22f76b Fix Windows build by replacing strings.h include
Summary:
strings.h header does not exist on Windows. So, we can try another way
to compare strings ignoring case.

Test Plan:
built and ran:

  $ ./ldb_cmd_test

Reviewers: sdong, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56535
2016-04-11 19:21:00 -07:00
Islam AbdelRahman
5675d5037f Revert travis commit
Summary: Revert travis commit pushed unintentionally

Test Plan: none

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56559
2016-04-11 16:46:55 -07:00
Islam AbdelRahman
91f0f1f5ef fix travis 2016-04-11 16:37:22 -07:00
sdong
a23c6052c8 Don't run DBOptionsAllFieldsSettable under valgrind
Summary: Test DBOptionsAllFieldsSettable sometimes fails under valgrind. Move option settable tests to a separate test file and disable it in valgrind..

Test Plan: Run valgrind test and make sure the test doesn't run.

Reviewers: andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56529
2016-04-11 14:24:55 -07:00
sdong
30d72ee43c PrefixTest.PrefixAndWholeKeyTest should run against a different directory from prefix_test
Summary: PrefixTest.PrefixAndWholeKeyTest runs against the same directory as prefix_test, which sometimes fail parallel tests. Fix it.

Test Plan: Run it in parallel and see it doesn't fail anymore.

Reviewers: andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56541
2016-04-11 13:02:56 -07:00
Andrew Kryczka
0e3cc2cf1f Add column family info to TableProperties::ToString()
Summary: This is used at least by `sst_dump --show_properties`

Test Plan:
- default CF

```
  $ ./sst_dump --show_properties --file=./tmp-db/000007.sst | grep 'column family'
  column family ID: 0
  column family name: default
```

- custom CF

```
  $ ./sst_dump --show_properties --file=./tmp-db/000012.sst | grep 'column family'
  column family ID: 1
  column family name: col-fam-1
```

- no CF

```
  $ ./sst_dump --show_properties --file=./tmp-db/000017.sst | grep 'column family'
  column family ID: N/A
  column family name: N/A
```

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56499
2016-04-08 18:50:18 -07:00
Jay Edgar
2448f80375 Make sure that if use_mmap_reads is on use_os_buffer is also on
Summary: The code assumes that if use_mmap_reads is on then use_os_buffer is also on.  This make sense as by using memory mapped files for reading you are expecting the OS to cache what it needs.  Add code to make sure the user does not turn off use_os_buffer when they turn on use_mmap_reads

Test Plan: New test: DBTest.MMapAndBufferOptions

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56397
2016-04-08 14:30:15 -07:00
Andrew Kryczka
114a1b8792 Fix build errors for windows
Summary:
- Need to use unsigned long long for 64-bit literals on windows
- Need size_t for backup meta-file length since clang doesn't let us assign size_t to int

Test Plan: backupable_db_test and options_test

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56391
2016-04-08 13:09:19 -07:00
Islam AbdelRahman
0522990358 Improve sst_dump help message
Summary:
Current Message

```
sst_dump [--command=check|scan|none|raw] [--verify_checksum] --file=data_dir_OR_sst_file [--output_hex] [--input_key_hex] [--from=<user_key>] [--to=<user_key>] [--read_num=NUM] [--show_properties] [--show_compression_sizes] [--show_compression_sizes [--set_block_size=<block_size>]]
```
New message

```
sst_dump --file=<data_dir_OR_sst_file> [--command=check|scan|raw]
    --file=<data_dir_OR_sst_file>
      Path to SST file or directory containing SST files

    --command=check|scan|raw
        check: Iterate over entries in files but dont print anything except if an error is encounterd (default command)
        scan: Iterate over entries in files and print them to screen
        raw: Dump all the table contents to <file_name>_dump.txt

    --output_hex
      Can be combined with scan command to print the keys and values in Hex

    --from=<user_key>
      Key to start reading from when executing check|scan

    --to=<user_key>
      Key to stop reading at when executing check|scan

    --read_num=<num>
      Maximum number of entries to read when executing check|scan

    --verify_checksum
      Verify file checksum when executing check|scan

    --input_key_hex
      Can be combined with --from and --to to indicate that these values are encoded in Hex

    --show_properties
      Print table properties after iterating over the file

    --show_compression_sizes
      Independent command that will recreate the SST file using 16K block size with different
      compressions and report the size of the file using such compression

    --set_block_size=<block_size>
      Can be combined with --show_compression_sizes to set the block size that will be used
      when trying different compression algorithms
```

Test Plan: none

Reviewers: yhchiang, andrewkr, kradhakrishnan, yiwu, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56325
2016-04-08 12:05:02 -07:00
sdong
0930e5e99f Update comments on include/rocksdb/perf_context.h
Summary: Some grammer mistakes in code comments in include/rocksdb/perf_context.h. Also polish it a liitlebit.

Test Plan: Not needed

Reviewers: IslamAbdelRahman, yhchiang, yiwu, andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56307
2016-04-08 11:27:08 -07:00
Shu Zhang
3b977bcdc8 instructing people to use java7 for hdfs (#1063) 2016-04-07 14:34:28 -07:00
sdong
1518b733eb Change default number of cache shard bit to be 6 and max_file_opening_threads to be 16.
Summary: Cache shard bit 4 is sometimes too small and 6 is a more common value picked by users. Make that default. It shouldn't hurt much to change options.max_file_opening_threads default to be 16, which will reduce the worst case DB open time.

Test Plan: Run all existing tests.

Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, igor, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, MarkCallaghan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D55047
2016-04-07 13:55:10 -07:00
flabby
ada88b63f5 fix wrong assignment of level0_stop_writes_trigger in spatialdb (#1061) 2016-04-07 09:02:28 -07:00
Andrew Kryczka
2391ef7214 Embed column family name in SST file
Summary:
Added the column family name to the properties block. This property
is omitted only if the property is unavailable, such as when RepairDB()
writes SST files.

In a next diff, I will change RepairDB to use this new property for
deciding to which column family an existing SST file belongs. If this
property is missing, it will add it to the "unknown" column family (same
as its existing behavior).

Test Plan:
New unit test:

  $ ./db_table_properties_test --gtest_filter=DBTablePropertiesTest.GetColumnFamilyNameProperty

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D55605
2016-04-06 23:10:32 -07:00
Igor Canadi
ab4c62332e Don't use version in the error message
Summary: We use object `v` in the error message, which is not initialized if the edit is column family manipulation. This doesn't provide much useful info, so this diff is removing it. Instead, it dumps actual VersionEdit contents.

Test Plan: compiles. would be great to get tests in version_set_test.cc that cover cases where a file write fails

Reviewers: sdong, yhchiang, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56349
2016-04-06 15:00:15 -07:00
Islam AbdelRahman
d02eb8d00b Fix unused variable warning
Summary:
My last diff introduced a warning when compiling under release mode
https://reviews.facebook.net/D55539

fix the warning

Test Plan:
DEBUG_LEVEL=0 make db_bench
make check

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56295
2016-04-04 11:10:46 -07:00
Igor Canadi
9278097f86 Merge pull request #1056 from facebook/igorcanadi-patch-1
Create LANGUAGE-BINDINGS.md
2016-04-04 08:08:52 -07:00
Aaron Gao
cc87075d63 No need to limit to 20 files in UpdateAccumulatedStats() if options.max_open_files=-1
Summary:
There is a hardcoded constraint in our statistics collection that prevents reading properties from more than 20 SST files. This means our statistics will be very inaccurate for databases with > 20 files since additional files are just ignored. The purpose of constraining the number of files used is to bound the I/O performed during statistics collection, since these statistics need to be recomputed every time the database reopened.

However, this constraint doesn't take into account the case where option "max_open_files" is -1. In that case, all the file metadata has already been read, so MaybeInitializeFileMetaData() won't incur any I/O cost. so this diff gets rid of the 20-file constraint in case max_open_files == -1.

Test Plan:
write into unit test db/db_properties_test.cc - "ValidateSampleNumber".
We generate 20 files with 2 rows and 10 files with 1 row.
If max_open_files !=-1, the `rocksdb.estimate-num-keys` should be (10*1 + 10*2)/20 * 30 = 45. Otherwise, it should be the ground truth, 50.
{F1089153}

Reviewers: andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56253
2016-04-01 16:19:12 -07:00
Islam AbdelRahman
8a1a603fdb Eliminate std::deque initialization while iterating over merge operands
Summary:
This patch is similar to D52563, When we iterate over a DB with merge operands we keep creating std::queue to store the operands, optimize this by reusing merge_operands_ data member

Before the patch

```
./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq" --db="/dev/shm/bench_merge_memcpy_on_the_fly/" --merge_operator="put" --merge_keys=10000 --num=10000

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       3.757 micros/op 266141 ops/sec;   29.4 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.413 micros/op 2423538 ops/sec;  268.1 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.451 micros/op 2219071 ops/sec;  245.5 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.420 micros/op 2382039 ops/sec;  263.5 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.408 micros/op 2452017 ops/sec;  271.3 MB/s

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       3.947 micros/op 253376 ops/sec;   28.0 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.441 micros/op 2266473 ops/sec;  250.7 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.471 micros/op 2122033 ops/sec;  234.8 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.440 micros/op 2271407 ops/sec;  251.3 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.429 micros/op 2331471 ops/sec;  257.9 MB/s
```

with the patch

```
./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq" --db="/dev/shm/bench_merge_memcpy_on_the_fly/" --merge_operator="put" --merge_keys=10000 --num=10000

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       4.080 micros/op 245092 ops/sec;   27.1 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.308 micros/op 3241843 ops/sec;  358.6 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.312 micros/op 3200408 ops/sec;  354.0 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.332 micros/op 3013962 ops/sec;  333.4 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.300 micros/op 3328017 ops/sec;  368.2 MB/s

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       3.973 micros/op 251705 ops/sec;   27.8 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.320 micros/op 3123752 ops/sec;  345.6 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.335 micros/op 2986641 ops/sec;  330.4 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.339 micros/op 2950047 ops/sec;  326.4 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.319 micros/op 3131565 ops/sec;  346.4 MB/s
```

Test Plan: make check -j64

Reviewers: yhchiang, andrewkr, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56031
2016-04-01 15:48:55 -07:00