Commit Graph

5147 Commits

Author SHA1 Message Date
Yueh-Hsuan Chiang
faa7eb3b99 Improve regression_test.sh
Summary:
This diff makes the following improvement in regression_test.sh:

1. Add NUM_OPS and DELETE_TEST_PATH to regression_test.sh:

  * NUM_OPS:  The number of operations that will be issued
    in EACH thread.
    Default: $NUM_KEYS / $NUM_THREADS

  * DELETE_TEST_PATH: If true, then the test directory
    will be deleted after the script ends.
    Default: 0
2. Add more information in SUMMARY.csv
3. Fix a bug in regression_test.sh where each thread in fillseq will all issue $NUM_KEYS writes.
4. Add --deletes in db_bench, which allows us to control the number of deletes instead of must using FLAGS_num.

Test Plan: run regression test with and without DELETE_TEST_PATH and NUM_OPS

Reviewers: yiwu, sdong, IslamAbdelRahman, gunnarku

Reviewed By: gunnarku

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D60039
2016-06-28 14:10:24 -07:00
sdong
c4cef07f1b Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
Summary: With max_size_amplification_percent = 0 to make sure that DBTestUniversalCompaction.UniversalCompactionSingleSortedRun tests the configuration to compact to one single sorted run.

Test Plan: Run all existing tests

Reviewers: yhchiang, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60021
2016-06-27 15:19:27 -07:00
Reid Horuff
892e9d3047 make transaction WriteOptions modifiable 2016-06-27 12:53:30 -07:00
charsyam
4f2b0946d1 fix simple typos (#1183) 2016-06-25 08:29:40 +01:00
Andrew Kryczka
3b7ed677de ColumnFamilyOptions API [CF + RepairDB part 3/3]
Summary:
Overload RepairDB to take vector-of-ColumnFamilyDescriptor, which tells
us CF name + options. Also takes a ColumnFamilyOptions for unspecified column
families encountered during the repair.

One potentially confusing thing is that we store options in the constructor and
don't invoke AddColumnFamily() until discovering the CF in ScanTable. This is
because we don't know the CF ID until we find a table belonging to that CF.

Depends on D59781.

Test Plan:
  $ ./repair_test

Reviewers: yhchiang, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59853
2016-06-24 16:29:43 -07:00
Andrew Kryczka
56ac686292 Detect column family from properties [CF + RepairDB part 2/3]
Summary:
This diff uses the CF ID and CF name properties in the SST file
to associate recovered data with the proper column family. Depends on D59775.

- In ScanTable(), create column families in VersionSet each time a new one is discovered (via reading SST file properties)
- In ConvertLogToTable(), dump an SST file for every column family with data in the WAL
- In AddTables(), make a VersionEdit per-column family that adds all of that CF's tables

Test Plan:
  $ ./repair_test

Reviewers: yhchiang, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59781
2016-06-24 13:12:13 -07:00
Wanning Jiang
3fc713ed92 delete 2nd level children for default env
Summary: ensure no 2nd level children under test_dir_

Test Plan: env_basic_test on 4 envs

Reviewers: andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59979
2016-06-24 12:31:58 -07:00
Andrew Kryczka
343507afb1 Refactor to use VersionSet [CF + RepairDB part 1/3]
Summary:
To support column families, it is easiest to use VersionSet to manage
our column families (if we don't have Versions then ColumnFamilyData always
behaves as a dummy column family). This diff only refactors the existing repair
logic to use VersionSet; the next two parts will add support for multiple
column families.

Test Plan:
  $ ./repair_test

Reviewers: yhchiang, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59775
2016-06-24 11:19:40 -07:00
Gunnar Kudrjavets
aa432be4b9 Workarounds for continuous build implementation 2016-06-23 19:38:29 -07:00
Gunnar Kudrjavets
8cd9f04feb Test change to verify new commit detection 2016-06-23 12:08:44 -07:00
Gunnar Kudrjavets
8a4ee7e90b Trivial change to test cont. build 2016-06-23 11:53:38 -07:00
Islam AbdelRahman
af6248d8b8 Fix max_bytes_for_level_base comment
Summary: Fix max_bytes_for_level_base comment

Test Plan: none

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59955
2016-06-22 18:28:38 -07:00
Wanning Jiang
0d7b261236 add tests to env_basic_test.cc
Summary: test NativeHdfsEnv

Test Plan: env_basic_test.cc

Reviewers: andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59889
2016-06-22 16:16:21 -07:00
Gunnar Kudrjavets
6576fa05af Fix minor typos and PHP source file name used to trigger the builds
Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed.

Test Plan: - Testing with a private Sandcastle instance.

Reviewers: sdong, mung

Reviewed By: mung

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59919
2016-06-22 11:05:52 -07:00
omegaga
c4e19b77e8 Add a read option to enable background purge when cleaning up iterators
Summary:
Add a read option `background_purge_on_iterator_cleanup` to avoid deleting files in foreground when destroying iterators.
Instead, a job is scheduled in high priority queue and would be executed in a separate background thread.

Test Plan: Add a variant of PurgeObsoleteFileTest. Turn on background purge option in the new test, and use sleeping task to ensure files are deleted in background.

Reviewers: IslamAbdelRahman, sdong

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59499
2016-06-21 18:41:23 -07:00
Islam AbdelRahman
fa813f7478 Update DB::AddFile() to ingest the file to the lowest possible level
Summary:
DB::AddFile() right now always add the ingested file to L0
update the logic to add the file to the lowest possible level

Test Plan: unit tests

Reviewers: jkedgar, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, yoshinorim

Differential Revision: https://reviews.facebook.net/D59637
2016-06-21 17:57:59 -07:00
Islam AbdelRahman
d6b79e2fd0 Remove filter_deletes from crash_test
Summary: filter_deletes option was removed, remove it from crash_test to fix it

Test Plan: make crash_test

Reviewers: yhchiang, andrewkr, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59901
2016-06-21 17:30:21 -07:00
Gunnar Kudrjavets
a52e4d7d02 Framework for enabling continuous RocksDB build and tests
Summary:
The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs.

The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things:

- Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only.

- Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed.

- Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen.

Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff.

Test Plan:
- Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass.
- Private testing of triggering the continuous build script.
- Once the changes will land then author an internal job which will use the script and verify its validity.

Reviewers: sdong, yhchiang, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59811
2016-06-21 11:38:54 -07:00
Islam AbdelRahman
f9bd66779f Makefile warning for invalid paths in make_config.mk
Summary:
Update Makefile to show warnings when we have invalid paths in our make_config.mk file

sample output

```
$ make static_lib -j64
Makefile:150: Warning: /mnt/gvfs/third-party2/libgcc/53e0eac8911888a105aa98b9a35fe61cf1d8b278/4.9.x/gcc-4.9-glibc-2.20/024dbc3/libs dont exist
Makefile:150: Warning: /mnt/gvfs/third-party2/llvm-fb/b91de48a4974ec839946d824402b098d43454cef/stable/centos6-native/7aaccbe/../../src/clang/tools/scan-build/scan-build dont exist
  GEN      util/build_version.cc
```

Test Plan: check that warning is printed visually

Reviewers: sdong, yiwu, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59523
2016-06-21 10:47:57 -07:00
Islam AbdelRahman
88a2776db5 Update SstFileWriter to use bottommost_compression if avaliable
Summary: SstFileWriter ignore Options::bottommost_compression, update it to use bottommost_compression if available

Test Plan:
make check -j64
verified used compression using ./sst_dump

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, yoshinorim

Differential Revision: https://reviews.facebook.net/D59841
2016-06-20 11:26:25 -07:00
goroutine
e87d5df1ae TiKV use-case (#1172) 2016-06-20 09:46:46 -07:00
sdong
7b79238b65 Deprectate filter_deletes
Summary: filter_deltes is not a frequently used feature. Remove it.

Test Plan: Run all test suites.

Reviewers: igor, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59427
2016-06-17 10:30:47 -07:00
sdong
4939fc3892 Bulk load mode shouldn't stop ingest
Summary: We introduced default slow down and stop condition, but didn't reset it in bulk load mode. Fix it.

Test Plan: N/A

Reviewers: igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59757
2016-06-17 09:58:50 -07:00
Yueh-Hsuan Chiang
3a2bccc845 Fixed a crash bug that incorrectly parse deprecated options in options_helper
Summary: Fixed a crash bug that incorrectly parse deprecated options in options_helper

Test Plan:
run db_bench with an old options file with memtable_prefix_bloom_probes
./db_bench --options_file=AN_OLD_OPTIONS_FILE --num=100 --benchmarks=fillseq

Reviewers: sdong, IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59787
2016-06-17 04:53:30 -07:00
Yueh-Hsuan Chiang
cf8adc971e Allow arcanist_util to work with both new and old arc versions
Summary:
Allow arcanist_util to work with both new and old arc versions.
The diff is based on Adam Retter's pull request
https://github.com/facebook/rocksdb/pull/1168

Many thanks to Adam to initiate this work

Test Plan: run arc lint and arc diff using different arc versions

Reviewers: sdong, IslamAbdelRahman, kradhakrishnan, andrewkr, adamretter, igor

Reviewed By: adamretter

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59697
2016-06-17 04:36:52 -07:00
Islam AbdelRahman
30a24f2d3d Add InternalStats and logging for AddFile()
Summary:
We dont report the bytes that we ingested from AddFile which make the write amplification numbers incorrect
Update InternalStats and add logging for AddFile()

Test Plan: Make sure the code compile and existing tests pass

Reviewers: lightmark, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59763
2016-06-16 16:21:41 -07:00
Islam AbdelRahman
d26a848074 Temporarily disable travis on linux
Summary: Travis is broken on linux, disable it temporarily

Test Plan: https://travis-ci.org/facebook/rocksdb/builds/137721712

Reviewers: yhchiang, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59673
2016-06-15 12:08:46 -07:00
sdong
249e796dfc Fix Flaky DBCompactionTest.SkipStatsUpdateTest
Summary: DBCompactionTest.SkipStatsUpdateTest sometimes fails. I don't see any verification related to the deletes issued. Remove them to avoid the uncertainty.

Test Plan: Run the test.

Reviewers: IslamAbdelRahman, andrewkr, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59613
2016-06-15 12:00:51 -07:00
krad
162c9170dd Make sandcastle access secure
Summary: Making access credentials for sandcastle configurable

Test Plan: Submit for review

Reviewers: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59649
2016-06-15 10:55:29 -07:00
Islam AbdelRahman
8366e10ffc Fix clang build
Summary: Fix clang build

Test Plan: USE_CLANG=1 make check -j64

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59667
2016-06-15 00:24:33 -07:00
sdong
0babce57f7 Move away from enum char value -1
Summary: char is not signed in some platforms. Having negative values confuse those compilers.

Test Plan: Run all existing tests.

Reviewers: andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59619
2016-06-14 17:07:34 -07:00
Islam AbdelRahman
812dbfb483 Optimize BlockIter::Prev() by caching decoded entries
Summary:
Right now the way we do BlockIter::Prev() is like this

- Go to the beginning of the restart interval
- Keep moving forward (and decoding keys using ParseNextKey()) until we reach the desired key

This can be optimized by caching the decoded entries in the first pass and reusing them in consecutive BlockIter::Prev() calls

Before caching

```
DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readreverse" --db="/dev/shm/bench_prev_opt/" --use_existing_db --disable_auto_compactions
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.413 micros/op 2423972 ops/sec;  268.2 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.414 micros/op 2413867 ops/sec;  267.0 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.410 micros/op 2440881 ops/sec;  270.0 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.414 micros/op 2417298 ops/sec;  267.4 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.413 micros/op 2421682 ops/sec;  267.9 MB/s
```

After caching

```
DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readreverse" --db="/dev/shm/bench_prev_opt/" --use_existing_db --disable_auto_compactions
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.324 micros/op 3088955 ops/sec;  341.7 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.335 micros/op 2980999 ops/sec;  329.8 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.341 micros/op 2929681 ops/sec;  324.1 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.344 micros/op 2908490 ops/sec;  321.8 MB/s
DB path: [/dev/shm/bench_prev_opt/]
readreverse  :       0.338 micros/op 2958404 ops/sec;  327.3 MB/s
```

Test Plan: COMPILE_WITH_ASAN=1 make check -j64

Reviewers: andrewkr, yiwu, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, yoshinorim

Differential Revision: https://reviews.facebook.net/D59463
2016-06-14 12:27:46 -07:00
krad
550bf895e9 Minor bug fix with log name 2016-06-14 12:20:58 -07:00
Islam AbdelRahman
886af59105 Fix examples/Makefile jemalloc error
Summary: compile and link with jemalloc in examples binaries

Test Plan:
cd examples
make -j64

Reviewers: andrewkr, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, yoshinorim

Differential Revision: https://reviews.facebook.net/D59457
2016-06-14 12:18:32 -07:00
Adam Retter
e3b1e3dfa6 Expose save points in Java WriteBatch and WBWI (#1092)
* Java API - Expose SetSavePoint and RollbackToRestorePoint for WriteBatch and WriteBatchWithIndex

* Minor cleanup
2016-06-14 09:51:14 -07:00
Islam AbdelRahman
f5177c761f Remove wasteful instrumentation in FullMerge (stacked on D59577)
Summary:
[ This diff is stacked on top of D59577 ]

We keep calling timer.ElapsedNanos() on every call to MergeOperator::FullMerge even when statistics are disabled, this is wasteful.

I run the readseq benchmark on a DB containing 100K merge operands for 100K keys (1 operand per key) with 1GB block cache
I see slight performance improvment

Original results

```
$ ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=100000 --num=100000 --db="/dev/shm/100K_merge_compacted/" --cache_size=1073741824 --use_existing_db --disable_auto_compactions
------------------------------------------------
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.498 micros/op 2006597 ops/sec;  222.0 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.295 micros/op 3393627 ops/sec;  375.4 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.285 micros/op 3511155 ops/sec;  388.4 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.286 micros/op 3500470 ops/sec;  387.2 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.283 micros/op 3530751 ops/sec;  390.6 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.289 micros/op 3464811 ops/sec;  383.3 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.277 micros/op 3612814 ops/sec;  399.7 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.283 micros/op 3539640 ops/sec;  391.6 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.285 micros/op 3503766 ops/sec;  387.6 MB/s
```

After patch

```
$ ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=100000 --num=100000 --db="/dev/shm/100K_merge_compacted/" --cache_size=1073741824 --use_existing_db --disable_auto_compactions
------------------------------------------------
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.476 micros/op 2100119 ops/sec;  232.3 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.278 micros/op 3600887 ops/sec;  398.4 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.275 micros/op 3636698 ops/sec;  402.3 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.271 micros/op 3691661 ops/sec;  408.4 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.273 micros/op 3661534 ops/sec;  405.1 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.276 micros/op 3627106 ops/sec;  401.3 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.272 micros/op 3682635 ops/sec;  407.4 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.266 micros/op 3758331 ops/sec;  415.8 MB/s
DB path: [/dev/shm/100K_merge_compacted/]
readseq      :       0.266 micros/op 3761907 ops/sec;  416.2 MB/s
```

Test Plan: make check -j64

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59583
2016-06-13 16:22:14 -07:00
Gunnar Kudrjavets
97fd2a6383 Remove dead Jenkins code and support arc diff --preview in RocksDB
Summary:
Two changes here:

- Remove dead Jenkins related code which is no longer relevant.
- Support `arc diff --preview`. Currently it doesn't work because a step which applies a diff assumes that a revision has been created. Which in case of `--preview` isn't. Therefore diff can't be applied and validation fails. Solution is to use `--nocommit` because for validation purposes performing a commit isn't necessary.

Test Plan:
- Current changes are submitted using `arc diff --preview`.
- All the pre-commit verification tests passed.

Reviewers: kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: leveldb, andrewkr, jtolmer, dhruba

Differential Revision: https://reviews.facebook.net/D59571
2016-06-13 16:20:15 -07:00
Islam AbdelRahman
7c919deccc Reuse TimedFullMerge instead of FullMerge + instrumentation
Summary:
We have alot of code duplication whenever we call FullMerge we keep duplicating the instrumentation and statistics code
This is a simple diff to refactor the code to use TimedFullMerge instead of FullMerge

Test Plan: COMPILE_WITH_ASAN=1 make check -j64

Reviewers: andrewkr, yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59577
2016-06-13 16:17:26 -07:00
sdong
9a33a723ba Remove the comments saying allow_concurrent_memtable_write and enable_write_thread_adaptive_yield are not stable
Summary: allow_concurrent_memtable_write and enable_write_thread_adaptive_yield went through extensive stress test and benchmarks. Should be considered stable.

Test Plan: N/A

Reviewers: ngbronson, yhchiang, IslamAbdelRahman, dhruba

Reviewed By: dhruba

Subscribers: MarkCallaghan, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59415
2016-06-13 16:13:22 -07:00
krad
81f6b33d93 Fix tsan error
Summary:
The tsan error was because the random implementation we have is not
thread safe, using Random::GetTLSInstance

Test Plan: Run tests in Linux

Reviewers: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59559
2016-06-13 12:59:38 -07:00
Yi Wu
bc8af90e8c add option to not flush memtable on open()
Summary:
Add option to not flush memtable on open()
In case the option is enabled, don't delete existing log files by not updating log numbers to MANIFEST.
Will still flush if we need to (e.g. memtable full in the middle). In that case we also flush final memtable.
If wal_recovery_mode = kPointInTimeRecovery, do not halt immediately after encounter corruption. Instead, check if seq id of next log file is last_log_sequence + 1. In that case we continue recovery.

Test Plan: See unit test.

Reviewers: dhruba, horuff, sdong

Reviewed By: sdong

Subscribers: benj, yhchiang, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D57813
2016-06-13 11:34:16 -07:00
Islam AbdelRahman
8100ec2cd1 Fix libgcc broken lib path
Summary: current libgcc lib path is broken, update it and make sure the new one exists

Test Plan:
verify that the new path exists
make check -j64

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59535
2016-06-12 00:11:08 -07:00
Nadav Rotem
7360db39e6 Add a check mode to verify compressed block can be decompressed back
Summary:
Try to decompress compressed blocks when a special flag is set.
assert and crash in debug builds if we can't decompress the just-compressed input.

Test Plan: Run unit-tests.

Reviewers: dhruba, andrewkr, sdong, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59145
2016-06-10 18:20:54 -07:00
sdong
2a79af1c54 Fix Java Break Related to memtable bloom bits to size ratio change
Summary: Need to change several more places for the change to fix Java tests

Test Plan:
make jtest
under java, run "make db_bench"

Reviewers: yhchiang, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59541
2016-06-10 16:54:24 -07:00
sdong
6faddd7c55 Merge db/slice.cc into util/slice.cc
Summary: It confuses some compilers to have slice.cc under multiple directories. Merge them.

Test Plan: Run existing tests

Reviewers: andrewkr, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59409
2016-06-10 16:37:36 -07:00
sdong
5009b5326b BlockBasedTable::FullFilterKeyMayMatch() Should skip prefix bloom if full key bloom exists
Summary: Currently, if users define both of full key bloom and prefix bloom in SST files. During Get(), if full key bloom shows the key may exist, we still go ahead and check prefix bloom. This is wasteful. If bloom filter for full keys exists, we should always ignore prefix bloom in Get().

Test Plan: Run existing tests

Reviewers: yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57825
2016-06-10 16:27:56 -07:00
omegaga
2d05eaeb28 Fix name conflict in delete_shceduler_test and db_sst_test
Summary: delete_scheduler_test and db_sst_test share a same directory name, causing possible fails on both tests when running in parallel. Fixed by changing directory name.

Test Plan: Run the two tests in parallel: `parallel -u ./{} ::: delete_scheduler_test db_sst_test`

Reviewers: sdong, andrewkr

Reviewed By: sdong, andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59529
2016-06-10 15:39:17 -07:00
Islam AbdelRahman
bde7d1055f Fix clang_analyze path in fbcode_config.sh
Summary: current clang_analyze path is broken, update it

Test Plan:
USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make J=64 analyze -j64
make sure it works

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59505
2016-06-10 14:36:38 -07:00
Andrew Kryczka
ca3db54788 Fetch branches from github for format compatibility test
Summary:
The sandcastle setup doesn't provide a remote with our branches. Need
to fetch them directly from github.

Test Plan:
ran script on devserver and the relevant commands on a sandcastle
host.

Reviewers: IslamAbdelRahman, lightmark, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59511
2016-06-10 13:17:14 -07:00
sdong
20699df843 memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
Summary:
memtable_prefix_bloom_probes is not a critical option. Remove it to reduce number of options.
It's easier for users to make mistakes with memtable_prefix_bloom_bits, turn it to memtable_prefix_bloom_bits_ratio

Test Plan: Run all existing tests

Reviewers: yhchiang, igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: gunnarku, yoshinorim, MarkCallaghan, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59199
2016-06-10 12:12:10 -07:00