Commit Graph

6739 Commits

Author SHA1 Message Date
Zhongyi Xie
51c2ea0feb Reduce heavy hitter for Get operation
Summary:
This PR addresses the following heavy hitters in `Get` operation by moving calls to `StatisticsImpl::recordTick` from `BlockBasedTable` to `Version::Get`

- rocksdb.block.cache.bytes.write
- rocksdb.block.cache.add
- rocksdb.block.cache.data.miss
- rocksdb.block.cache.data.bytes.insert
- rocksdb.block.cache.data.add
- rocksdb.block.cache.hit
- rocksdb.block.cache.data.hit
- rocksdb.block.cache.bytes.read

The db_bench statistics before and after the change are:

|1GB block read|Children      |Self  |Command          |Shared Object        |Symbol|
|---|---|---|---|---|---|
|master:     |4.22%     |1.31%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|updated:    |0.51%     |0.21%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|     	     |0.14%     |0.14%  |db_bench  |db_bench  |[.] rocksdb::GetContext::record_counters|

|1MB block read|Children      |Self  |Command          |Shared Object        |Symbol|
|---|---|---|---|---|---|
|master:    |3.48%     |1.08%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|updated:    |0.80%     |0.31%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|    	     |0.35%     |0.35%  |db_bench  |db_bench  |[.] rocksdb::GetContext::record_counters|
Closes https://github.com/facebook/rocksdb/pull/3172

Differential Revision: D6330532

Pulled By: miasantreble

fbshipit-source-id: 2b492959e00a3db29e9437ecdcc5e48ca4ec5741
2017-12-12 21:11:33 -08:00
Islam AbdelRahman
9089373a01 Fix DeleteScheduler::MarkAsTrash() handling existing trash
Summary:
DeleteScheduler::MarkAsTrash() don't handle existing .trash files correctly
This cause rocksdb to not being able to delete existing .trash files on restart
Closes https://github.com/facebook/rocksdb/pull/3261

Differential Revision: D6548003

Pulled By: IslamAbdelRahman

fbshipit-source-id: c3800639412e587a690062c63076a5a08881e0e6
2017-12-12 18:17:13 -08:00
Yi Wu
7393ef779c Fix BlockFetcher ASAN error
Summary:
Some call sites of BlockFetcher create temporary ReadOptions and pass to BlockFetcher. The temporary object will be gone after BlockFetcher construction but BlockFetcher keep its reference, causing stack-use-after-scope. Fixing it.
Closes https://github.com/facebook/rocksdb/pull/3258

Differential Revision: D6547152

Pulled By: yiwu-arbug

fbshipit-source-id: 6b49e9dd46bb72307f5d8f88ea15faacff35b9bc
2017-12-12 12:12:38 -08:00
Souvik Banerjee
4bcb7fb148 Update transaction_test_util.cc
Summary:
Fixes a compile error on gcc 7.2.1 (-Werror=format-truncation=).
Closes https://github.com/facebook/rocksdb/pull/3248

Differential Revision: D6546515

Pulled By: yiwu-arbug

fbshipit-source-id: bd78cca63f2af376faceccb1838d2d4cc9208fef
2017-12-12 12:12:38 -08:00
Yi Wu
e3a06f12d2 WritePrepared Txn: fix compaction filter snapshot checks
Summary:
Add snapshot_checker check whenever we need to check sequence against snapshots and decide what to do with an input key. The changes are related to one of:
* compaction filter
* single delete
* delete at bottom level
* merge
Closes https://github.com/facebook/rocksdb/pull/3251

Differential Revision: D6537850

Pulled By: yiwu-arbug

fbshipit-source-id: 3faba40ed5e37779f4a0cb7ae78af9546659c7f2
2017-12-12 11:12:24 -08:00
Siying Dong
a9c8d4ef15 Fix memory issue introduced by 2f1a3a4d74
Summary: Closes https://github.com/facebook/rocksdb/pull/3256

Differential Revision: D6541714

Pulled By: siying

fbshipit-source-id: 40efd89b68587a9d58cfe6f4eebd771c2d9f1542
2017-12-11 18:27:28 -08:00
Zhongyi Xie
bb5ed4b1d1 exclude DynamicUniversalCompactionOptions from ROCKSDB_LITE
Summary:
since [SetOptions](https://github.com/facebook/rocksdb/blob/master/db/db_impl.cc#L494) is not supported in ROCKSDB_LITE
Right now unit test under lite is broken
Closes https://github.com/facebook/rocksdb/pull/3253

Differential Revision: D6539428

Pulled By: miasantreble

fbshipit-source-id: 13172b8ecbd75682330726498ea198969bc3e637
2017-12-11 16:28:20 -08:00
Siying Dong
0d5692e02b Switch version to 5.10
Summary: Closes https://github.com/facebook/rocksdb/pull/3252

Differential Revision: D6539373

Pulled By: siying

fbshipit-source-id: ce7c3d3fe625852179055295da9cf7bc80755025
2017-12-11 15:42:01 -08:00
Siying Dong
2f1a3a4d74 Refactor ReadBlockContents()
Summary:
Divide ReadBlockContents() to multiple sub-functions. Maintaining the input and intermediate data in a new class BlockFetcher.
I hope in general it makes the code easier to maintain.
Another motivation to do it is to clearly divide the logic before file reading and after file reading. The refactor will help us evaluate how can we make I/O async in the future.
Closes https://github.com/facebook/rocksdb/pull/3244

Differential Revision: D6520983

Pulled By: siying

fbshipit-source-id: 338d90bc0338472d46be7a7682028dc9114b12e9
2017-12-11 15:27:32 -08:00
Yi Wu
9a27ac5d89 Fix drop column family data race
Summary:
A data race is caught by tsan_crash test between compaction and DropColumnFamily:
https://gist.github.com/yiwu-arbug/5a2b4baae05eeb99ae1719b650f30a44 Compaction checks if the column family has been dropped on each key input, while user can issue DropColumnFamily which updates cfd->dropped_, causing the data race. Fixing it by making cfd->dropped_ an atomic.
Closes https://github.com/facebook/rocksdb/pull/3250

Differential Revision: D6535991

Pulled By: yiwu-arbug

fbshipit-source-id: 5571df020beae7fa7db6fff5ad0d598f49962895
2017-12-11 13:57:48 -08:00
Zhongyi Xie
fcc8a6574d Make Universal compaction options dynamic
Summary:
Let me know if more test coverage is needed
Closes https://github.com/facebook/rocksdb/pull/3213

Differential Revision: D6457165

Pulled By: miasantreble

fbshipit-source-id: 3f944abff28aa7775237f1c4f61c64ccbad4eea9
2017-12-11 13:27:06 -08:00
Yi Wu
250a51a3f9 BlobDB: refactor DB open logic
Summary:
Refactor BlobDB open logic. List of changes:

Major:
* On reopen, mark blob files found as immutable, do not use them for writing new keys.
* Not to scan the whole file to find file footer. Instead just seek to the end of the file and try to read footer.

Minor:
* Move most of the real logic from blob_db.cc to blob_db_impl.cc.
* Not to hold shared_ptr of event listeners in global maps in blob_db.cc
* Some changes to BlobFile interface.
* Improve logging and error handling.
Closes https://github.com/facebook/rocksdb/pull/3246

Differential Revision: D6526147

Pulled By: yiwu-arbug

fbshipit-source-id: 9dc4cdd63359a2f9b696af817086949da8d06952
2017-12-11 12:12:38 -08:00
Prashant D
6a183d1ae8 Fix coverity issues compaction_job, compaction_picker
Summary:
db/compaction_job.cc:
  ReportStartedCompaction(compaction);

CID 1419863 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member bottommost_level_ is not initialized in this constructor nor in any functions that it calls.

db/compaction_picker_universal.cc:
7struct InputFileInfo {
   	2. uninit_member: Non-static class member level is not initialized in this constructor nor in any functions that it calls.

CID 1405355 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
4. uninit_member: Non-static class member index is not initialized in this constructor nor in any functions that it calls.
 38  InputFileInfo() : f(nullptr) {}

db/dbformat.h:
 ParsedInternalKey()
 84      : sequence(kMaxSequenceNumber)  // Make code analyzer happy

CID 1168095 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member type is not initialized in this constructor nor in any functions that it calls.
 85  {}  // Intentionally left uninitialized (for speed)
Closes https://github.com/facebook/rocksdb/pull/3091

Differential Revision: D6534558

Pulled By: yiwu-arbug

fbshipit-source-id: 5ada975956196d267b3f149386842af71eda7553
2017-12-11 11:57:15 -08:00
Andrew Kryczka
e3814a8608 revert fbcode build behavior
Summary: Closes https://github.com/facebook/rocksdb/pull/3242

Differential Revision: D6514255

Pulled By: ajkr

fbshipit-source-id: c39fa8e745866b052649d02bf339e794d77e96a3
2017-12-07 16:12:52 -08:00
Dmitri Smirnov
fe608e32ab Fix a race condition in WindowsThread (port::Thread)
Summary:
Fix a race condition when we create a thread and immediately destroy
 This case should be supported.
  What happens is that the thread function needs the Data instance
  to actually run but has no shared ownership and must rely on the
  WindowsThread instance to continue existing.
  To address this we change unique_ptr to shared_ptr and then
  acquire an additional refcount for the threadproc which destroys it
  just before the thread exit.
  We choose to allocate shared_ptr instance on the heap as this allows
  the original thread to continue w/o waiting for the new thread to start
  running.
Closes https://github.com/facebook/rocksdb/pull/3240

Differential Revision: D6511324

Pulled By: yiwu-arbug

fbshipit-source-id: 4633ff7996daf4d287a9fe34f60c1dd28cf4ff36
2017-12-07 13:42:53 -08:00
Prashant D
34aa245dd8 Fix coverity issues version, write_batch
Summary:
db/version_builder.cc:
117        base_vstorage_->InternalComparator();

CID 1351713 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member field level_zero_cmp_.internal_comparator is not initialized in this constructor nor in any functions that it calls.

db/version_edit.h:
145  FdWithKeyRange()
146      : fd(),
147        smallest_key(),
148        largest_key() {

CID 1418254 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member file_metadata is not initialized in this constructor nor in any functions that it calls.
149  }

db/version_set.cc:
120    }

CID 1322789 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
4. uninit_member: Non-static class member curr_file_level_ is not initialized in this constructor nor in any functions that it calls.
121  }

db/write_batch.cc:
 939    assert(cf_mems_);

CID 1419862 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
3. uninit_member: Non-static class member rebuilding_trx_seq_ is not initialized in this constructor nor in any functions that it calls.
 940  }
Closes https://github.com/facebook/rocksdb/pull/3092

Differential Revision: D6505666

Pulled By: yiwu-arbug

fbshipit-source-id: fd2c68948a0280772691a419d72ac7e190951d86
2017-12-07 11:57:36 -08:00
Prashant D
baff91c1ad table: Fix coverity issues
Summary:
table/block.cc:
420  }

CID 1396127 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
7. uninit_member: Non-static class member restart_offset_ is not initialized in this constructor nor in any functions that it calls.
421}

table/block_based_table_builder.cc:

CID 1418259 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
7. uninit_member: Non-static class member compressed_cache_key_prefix_size is not initialized in this constructor nor in any functions that it calls.

table/block_based_table_reader.h:
   	3. uninit_member: Non-static class member index_type is not initialized in this constructor nor in any functions that it calls.

CID 1396147 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
5. uninit_member: Non-static class member hash_index_allow_collision is not initialized in this constructor nor in any functions that it calls.
413        global_seqno(kDisableGlobalSequenceNumber) {}
414

table/cuckoo_table_reader.cc:
 55  if (hash_funs == user_props.end()) {
 56    status_ = Status::Corruption("Number of hash functions not found");
   	5. uninit_member: Non-static class member is_last_level_ is not initialized in this constructor nor in any functions that it calls.
   	7. uninit_member: Non-static class member identity_as_first_hash_ is not initialized in this constructor nor in any functions that it calls.
   	9. uninit_member: Non-static class member use_module_hash_ is not initialized in this constructor nor in any functions that it calls.
   	11. uninit_member: Non-static class member num_hash_func_ is not initialized in this constructor nor in any functions that it calls.
   	13. uninit_member: Non-static class member key_length_ is not initialized in this constructor nor in any functions that it calls.
   	15. uninit_member: Non-static class member user_key_length_ is not initialized in this constructor nor in any functions that it calls.
   	17. uninit_member: Non-static class member value_length_ is not initialized in this constructor nor in any functions that it calls.
   	19. uninit_member: Non-static class member bucket_length_ is not initialized in this constructor nor in any functions that it calls.
   	21. uninit_member: Non-static class member cuckoo_block_size_ is not initialized in this constructor nor in any functions that it calls.
   	23. uninit_member: Non-static class member cuckoo_block_bytes_minus_one_ is not initialized in this constructor nor in any functions that it calls.

CID 1322785 (#2 of 2): Uninitialized scalar field (UNINIT_CTOR)
25. uninit_member: Non-static class member table_size_ is not initialized in this constructor nor in any functions that it calls.
 57    return;

table/plain_table_index.h:
   	2. uninit_member: Non-static class member index_size_ is not initialized in this constructor nor in any functions that it calls.

CID 1322801 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
4. uninit_member: Non-static class member sub_index_size_ is not initialized in this constructor nor in any functions that it calls.
128        huge_page_tlb_size_(huge_page_tlb_size) {}
129
Closes https://github.com/facebook/rocksdb/pull/3113

Differential Revision: D6505719

Pulled By: yiwu-arbug

fbshipit-source-id: 38f44d8f9dfefb4c2e25d83b8df25a5201c75618
2017-12-07 11:57:36 -08:00
Andrew Kryczka
2e3a00987e fix ASAN for DeleteFilesInRange test case
Summary:
error message was

```
==3095==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffd18216c40 at pc 0x0000005edda1 bp 0x7ffd18215550 sp 0x7ffd18214d00
...
Address 0x7ffd18216c40 is located in stack of thread T0 at offset 1952 in frame
     #0 internal_repo_rocksdb/db_compaction_test.cc:1520 rocksdb::DBCompactionTest_DeleteFileRangeFileEndpointsOverlapBug_Test::TestBody()
```

It was unsafe to have slices referring to the temporary string objects' buffers, as those strings were destroyed before the slices were used. Fixed it by assigning the strings returned by `Key()` to local variables.
Closes https://github.com/facebook/rocksdb/pull/3238

Differential Revision: D6507864

Pulled By: ajkr

fbshipit-source-id: dd07de1a0070c6748c1ab4f3d7bd31f9a81889d0
2017-12-07 11:12:43 -08:00
Yi Wu
e1c569c324 Fix clang-analyzer false-positive on ldb_cmd.cc
Summary:
clang-analyzer complaint about db_ being nullptr, but it couldn't be because it checks exec_stats before proceed. Add an assert to get around the false-positive.

Test Plan
`make analyze`
Closes https://github.com/facebook/rocksdb/pull/3236

Differential Revision: D6505417

Pulled By: yiwu-arbug

fbshipit-source-id: e5b65764ea994dd9e4bab3e697b97dc70dc22cab
2017-12-06 22:58:46 -08:00
Sagar Vemuri
bbef8c3884 Log GetCurrentTime failures during Flush and Compaction
Summary:
`GetCurrentTime()` is used to populate `creation_time` table property during flushes and compactions. It is safe to ignore `GetCurrentTime()` failures here but they should be logged.

(Note that `creation_time` property was introduced as part of TTL-based FIFO compaction in #2480.)

Tes Plan:
`make check`
Closes https://github.com/facebook/rocksdb/pull/3231

Differential Revision: D6501935

Pulled By: sagar0

fbshipit-source-id: 376adcf4ab801d3a43ec4453894b9a10909c8eb6
2017-12-06 20:56:53 -08:00
Sagar Vemuri
d51fcb21f4 Blob DB: Add db_bench options
Summary:
Adding more BlobDB db_bench options which are needed for benchmarking.
Closes https://github.com/facebook/rocksdb/pull/3230

Differential Revision: D6500711

Pulled By: sagar0

fbshipit-source-id: 91d63122905854ef7c9148a0235568719146e6c5
2017-12-06 20:44:12 -08:00
Andrew Kryczka
53a516ab58 update history for recent commits
Summary:
I browsed through the history since 5.9 was released and found some changes worth mentioning in HISTORY.md.
Closes https://github.com/facebook/rocksdb/pull/3237

Differential Revision: D6506472

Pulled By: ajkr

fbshipit-source-id: 627ce9f94ca33df9f0f231a9c5ced3624b05506c
2017-12-06 19:56:17 -08:00
Andrew Kryczka
78d1a5ec72 Preserve overlapping file endpoint invariant
Summary:
Fix for #2833.

- In `DeleteFilesInRange`, use `GetCleanInputsWithinInterval` instead of `GetOverlappingInputs` to make sure we get a clean cut set of files to delete.
- In `GetCleanInputsWithinInterval`, support nullptr as `begin_key` or `end_key`.
- In `GetOverlappingInputsRangeBinarySearch`, move the assertion for non-empty range away from `ExtendFileRangeWithinInterval`, which should be allowed to return an empty range (via `end_index < begin_index`).
Closes https://github.com/facebook/rocksdb/pull/2843

Differential Revision: D5772387

Pulled By: ajkr

fbshipit-source-id: e554e8461823c6be82b21a9262a2da02b3957881
2017-12-06 18:56:54 -08:00
Yi Wu
a7d32776f0 Fix write_callback_test compile error
Summary:
Rename shadow variable name db_impl.

Fixing #3227
Closes https://github.com/facebook/rocksdb/pull/3235

Differential Revision: D6504051

Pulled By: yiwu-arbug

fbshipit-source-id: 186c9378dabb11f8d6db56f45c95cc3b029fcb88
2017-12-06 17:12:27 -08:00
Yi Wu
7f04af32a5 ldb to allow db with --try_load_options and without an options file
Summary:
This is to fix tools/check_format_compatible.sh. The tool try to open
old versions of rocksdb with the provided options file. When options
file is missing (e.g. rocksdb 2.2), it should still proceed with default
options.
Closes https://github.com/facebook/rocksdb/pull/3232

Differential Revision: D6503955

Pulled By: yiwu-arbug

fbshipit-source-id: e44cfcce7ddc7d12cf83466ed3f3fe7624aa78b8
2017-12-06 16:42:26 -08:00
Yi Wu
b5798bd324 Add missing recent versions to format compatible test
Summary:
Add recent versions for format compatible test. We should probably update the script to auto include available versions (by looking at include/rocksdb/versions.h and deduce branch names), but we can do it later.
Closes https://github.com/facebook/rocksdb/pull/3233

Differential Revision: D6503631

Pulled By: yiwu-arbug

fbshipit-source-id: e2b01d1ef6e784ff6ffa1bd75d741755e3c69a8c
2017-12-06 16:13:50 -08:00
Yi Wu
20995c5729 Make iterator invalid on Merge error
Summary:
Since #1665, on merge error, iterator will be set to corrupted status, but it doesn't invalidate the iterator. Fixing it.
Closes https://github.com/facebook/rocksdb/pull/3226

Differential Revision: D6499094

Pulled By: yiwu-arbug

fbshipit-source-id: 80222930f949e31f90a6feaa37ddc3529b510d2c
2017-12-06 11:56:39 -08:00
Maysam Yabandeh
36911f55dd WritePrepared Txn: stress test
Summary:
Augment the existing MySQLStyleTransactionTest to check for more core case scenarios. The changes showed effective in revealing the bugs reported in https://github.com/facebook/rocksdb/pull/3205 and https://github.com/facebook/rocksdb/pull/3101
Closes https://github.com/facebook/rocksdb/pull/3222

Differential Revision: D6476862

Pulled By: maysamyabandeh

fbshipit-source-id: 5068497702d67ffc206a58ed96f8578fbb510137
2017-12-06 09:42:28 -08:00
Flarnie Marchan
1b0c58d7e7 Add Code of Conduct
Summary:
In the past Facebook didn't promote including a Code of Conduct when creating new projects, and many projects skipped this important document. Let's fix it. :)

**why make this change?:**
Facebook Open Source provides a Code of Conduct statement for all
projects to follow, to promote a welcoming and safe open source community.

Exposing the COC via a separate markdown file is a standard being
promoted by Github via the Community Profile in order to meet their Open
Source Guide's recommended community standards.

As you can see, adding this file will improve [the rocksdb community profile](https://github.com/facebook/rocksdb/community)
checklist and increase the visibility of our COC.

**test plan:**
Viewing it on my branch -
<img width="1008" alt="screen shot 2017-12-03 at 5 05 45 pm" src="https://user-images.githubusercontent.com/1114467/33532198-66012a56-d84c-11e7-8fab-29ed410bd600.png">
<img width="1015" alt="screen shot 2017-12-03 at 5 05 59 pm" src="https://user-images.githubusercontent.com/1114467/33532199-661813d8-d84c-11e7-941e-94754dd481e5.png">

**issue:**
internal task t23481323
Closes https://github.com/facebook/rocksdb/pull/3219

Reviewed By: yiwu-arbug

Differential Revision: D6494234

Pulled By: flarnie

fbshipit-source-id: 55b59db335cc5546f3a1c968322b9281a3dc3aaf
2017-12-05 18:42:35 -08:00
Adam Novak
a37d734596 Add ROCKSDB_DISABLE_* environment variables
Summary:
Should fix #3036.
Closes https://github.com/facebook/rocksdb/pull/3042

Differential Revision: D6452921

Pulled By: sagar0

fbshipit-source-id: eaf11e43fee1f8747006530cfc0c7a358f1c2f0f
2017-12-05 15:12:46 -08:00
Alex Robinson
4634c735a8 Update DBOptions::IncreaseParallelism to use newer background settings
Summary:
The Options header file recommends using max_background_jobs rather than
directly setting max_background_compactions or max_background_flushes.

I've personally seen a performance problem where stalls were happening
because the one background flushing thread was blocked that was fixed
by this change -
https://github.com/cockroachdb/cockroach/issues/19699#issuecomment-347672485
Closes https://github.com/facebook/rocksdb/pull/3208

Differential Revision: D6473178

Pulled By: ajkr

fbshipit-source-id: 67c892ceb7b1909d251492640cb15a0f2262b7ed
2017-12-04 01:56:15 -08:00
Andrew Kryczka
57056bb606 gflags in cmake on linux
Summary:
We should use it if available otherwise the tools builds never work. Thanks to #3212, we can set -DGFLAGS=1 and it'll be independent of the namespace with which gflags was compiled.
Closes https://github.com/facebook/rocksdb/pull/3214

Differential Revision: D6462214

Pulled By: ajkr

fbshipit-source-id: db4e5f1b905322e3119554a9d01b57532c499384
2017-12-01 18:28:24 -08:00
Andrew Kryczka
63f1c0a57d fix gflags namespace
Summary:
I started adding gflags support for cmake on linux and got frustrated that I'd need to duplicate the build_detect_platform logic, which determines namespace based on attempting compilation. We can do it differently -- use the GFLAGS_NAMESPACE macro if available, and if not, that indicates it's an old gflags version without configurable namespace so we can simply hardcode "google".
Closes https://github.com/facebook/rocksdb/pull/3212

Differential Revision: D6456973

Pulled By: ajkr

fbshipit-source-id: 3e6d5bde3ca00d4496a120a7caf4687399f5d656
2017-12-01 10:42:05 -08:00
Maysam Yabandeh
18dcf7f98d WritePrepared Txn: PreReleaseCallback
Summary:
Add PreReleaseCallback to be called at the end of WriteImpl but before publishing the sequence number. The callback is used in WritePrepareTxn to i) update the commit map, ii) update the last published sequence number in the 2nd write queue. It also ensures that all the commits will go to the 2nd queue.
These changes will ensure that the commit map is updated before the sequence number is published and used by reading snapshots. If we use two write queues, the snapshots will use the seq number published by the 2nd queue. If we use one write queue (the default, the snapshots will use the last seq number in the memtable, which also indicates the last published seq number.
Closes https://github.com/facebook/rocksdb/pull/3205

Differential Revision: D6438959

Pulled By: maysamyabandeh

fbshipit-source-id: f8b6c434e94bc5f5ab9cb696879d4c23e2577ab9
2017-11-30 23:50:45 -08:00
Nikhil Benesch
3e40a5e832 add missing config checks to CMakeLists.txt
Summary:
Bring CMakeLists.txt back up to parity with build_detect_platform.
Closes https://github.com/facebook/rocksdb/pull/3211

Differential Revision: D6452908

Pulled By: ajkr

fbshipit-source-id: 93f5f336ad7eff6ecf65dec47bfaf114dd24cfb2
2017-11-30 22:57:00 -08:00
Andres Suarez
fad14050ae Remove import use from TARGETS
Summary:
We're moving away from `import`. The equivalent internal construct that
gets the directory from `fbcode/` is `package_name()`. This is a
Skylark friendly wrapper around [`get_base_path`].

The additional whitespace change is from running `python ./buckifier/buckify_rocksdb.py`.

[`get_base_path`]: https://buckbuild.com/function/get_base_path.html
Closes https://github.com/facebook/rocksdb/pull/3210

Reviewed By: yiwu-arbug

Differential Revision: D6451242

Pulled By: zertosh

fbshipit-source-id: 445757261de0ec89d5d332c1ba9af097086326dc
2017-11-30 15:27:34 -08:00
Yi Wu
54095d3389 TARGETS file not include tests in opt mode
Summary:
Do not build the tests in opt mode, since SyncPoint and other test code will not be included.
Closes https://github.com/facebook/rocksdb/pull/3204

Differential Revision: D6431154

Pulled By: yiwu-arbug

fbshipit-source-id: c404ef042c1a6f679e5c1dc57600b3d8cb52fc28
2017-11-30 10:56:58 -08:00
zhangjinpeng1987
ffacaaa3ea fix Seek with lower_bound
Summary:
When Seek a key less than `lower_bound`, should return `lower_bound`.
ajkr PTAL
Closes https://github.com/facebook/rocksdb/pull/3199

Differential Revision: D6421126

Pulled By: ajkr

fbshipit-source-id: a06c825830573e0040630704f6bcb3f7f48626f7
2017-11-29 22:56:29 -08:00
Andrew Kryczka
ed3af9ef99 improve ldb CLI option support
Summary:
- Made CLI arguments take precedence over options file when both are provided. Note some of the CLI args are not settable via options file, like `--compression_max_dict_bytes`, so it's necessary to allow both ways of providing options simultaneously.
- Changed `PrepareOptionsForOpenDB` to update the proper `ColumnFamilyOptions` if one exists for the user's `--column_family_name` argument. I supported this only in the base class, `LDBCommand`, so it works for the general arguments. Will defer adding support for subcommand-specific arguments.
- Made the command fail if `--try_load_options` is provided and loading options file returns NotFound. I found the previous behavior of silently continuing confusing.
Closes https://github.com/facebook/rocksdb/pull/3144

Differential Revision: D6270544

Pulled By: ajkr

fbshipit-source-id: 7c2eac9f9b38720523d74466fb9e78db53561367
2017-11-28 17:28:58 -08:00
Andrew Kryczka
c85f8ccca3 convert null terminator in ascii dump
Summary:
The ASCII output is almost always useless to me as the first '\0' byte in the key or value causes it to stop printing. Since all characters are already surrounded by spaces, "\ 0" (how we display a backslash followed by a zero) and "\0" (how this PR displays a null terminator) are distinguishable. My assumption is the value of seeing all the bytes outweighs the value of the alignment we had before, where we always had one character followed by one space.
Closes https://github.com/facebook/rocksdb/pull/3203

Differential Revision: D6428651

Pulled By: ajkr

fbshipit-source-id: aafc978a51e9ea029cfe3e763e2bb0e1751b9ccf
2017-11-28 17:28:58 -08:00
Prashant D
c1ed005a21 tools: Fix coverity issues
Summary:
tools/ldb_cmd.cc:
```
310  ignore_unknown_options_ = IsFlagPresent(flags, ARG_IGNORE_UNKNOWN_OPTIONS);

CID 1322798 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
5. uninit_member: Non-static class member db_ttl_ is not initialized in this constructor nor in any functions that it calls.
311}
```
Closes https://github.com/facebook/rocksdb/pull/3122

Differential Revision: D6428576

Pulled By: sagar0

fbshipit-source-id: d77f04dd201f7f1d9f59ef88a215ee7ad7b934e9
2017-11-28 15:27:41 -08:00
Prashant D
81cf262ff5 utilities/backupable : Fix coverity issues
Summary:
1. Class BackupMeta
```
52      : timestamp_(0), size_(0), meta_filename_(meta_filename),

CID 1168103 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member sequence_number_ is not initialized in this constructor nor in any functions that it calls.
153        file_infos_(file_infos), env_(env) {}
```
2. class BackupEngineImpl
```
513  }
        7. uninit_member: Non-static class member latest_backup_id_ is not initialized in this constructor nor in any functions that it calls.

CID 1322803 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
9. uninit_member: Non-static class member latest_valid_backup_id_ is not initialized in this constructor nor in any functions that it calls.
514}
```
3. struct BackupAfterCopyOrCreateWorkItem
```
368  struct BackupAfterCopyOrCreateWorkItem {
369    std::future<CopyOrCreateResult> result;
        1. member_decl: Class member declaration for shared.
370    bool shared;
        3. member_decl: Class member declaration for needed_to_copy.
371    bool needed_to_copy;
        5. member_decl: Class member declaration for backup_env.
372    Env* backup_env;
373    std::string dst_path_tmp;
374    std::string dst_path;
375    std::string dst_relative;
        2. uninit_member: Non-static class member shared is not initialized in this constructor nor in any functions that it calls.
        4. uninit_member: Non-static class member needed_to_copy is not initialized in this constructor nor in any functions that it calls.

CID 1396122 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
6. uninit_member: Non-static class member backup_env is not initialized in this constructor nor in any functions that it calls.
376    BackupAfterCopyOrCreateWorkItem() {}
```
4. struct CopyOrCreateWorkItem
```
318  struct CopyOrCreateWorkItem {
319    std::string src_path;
320    std::string dst_path;
321    std::string contents;
        1. member_decl: Class member declaration for src_env.
322    Env* src_env;
        3. member_decl: Class member declaration for dst_env.
323    Env* dst_env;
        5. member_decl: Class member declaration for sync.
324    bool sync;
        7. member_decl: Class member declaration for rate_limiter.
325    RateLimiter* rate_limiter;
        9. member_decl: Class member declaration for size_limit.
326    uint64_t size_limit;
327    std::promise<CopyOrCreateResult> result;
328    std::function<void()> progress_callback;
329
        2. uninit_member: Non-static class member src_env is not initialized in this constructor nor in any functions that it calls.
        4. uninit_member: Non-static class member dst_env is not initialized in this constructor nor in any functions that it calls.
        6. uninit_member: Non-static class member sync is not initialized in this constructor nor in any functions that it calls.
        8. uninit_member: Non-static class member rate_limiter is not initialized in this constructor nor in any functions that it calls.

CID 1396123 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
10. uninit_member: Non-static class member size_limit is not initialized in this constructor nor in any functions that it calls.
330    CopyOrCreateWorkItem() {}
```
5. struct RestoreAfterCopyOrCreateWorkItem
```
struct RestoreAfterCopyOrCreateWorkItem {
410    std::future<CopyOrCreateResult> result;
        1. member_decl: Class member declaration for checksum_value.
411    uint32_t checksum_value;

CID 1396153 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member checksum_value is not initialized in this constructor nor in any functions that it calls.
412    RestoreAfterCopyOrCreateWorkItem() {}
```
Closes https://github.com/facebook/rocksdb/pull/3131

Differential Revision: D6428556

Pulled By: sagar0

fbshipit-source-id: a86675444543eff028e3cae6942197a143a112c4
2017-11-28 14:43:28 -08:00
kapitan-k
75d57a5d53 C API: Add some block based table options
Summary: Closes https://github.com/facebook/rocksdb/pull/3159

Differential Revision: D6428220

Pulled By: sagar0

fbshipit-source-id: 60508d09b5281f54b907a1c40e9631fc08343131
2017-11-28 14:12:44 -08:00
Prashant D
b45fbc1175 utilities: Fix coverity issues
Summary:
```
utilities/persistent_cache/block_cache_tier_file.cc
64struct CacheRecordHeader {
   	2. uninit_member: Non-static class member magic_ is not initialized in this constructor nor in any functions that it calls.
   	4. uninit_member: Non-static class member crc_ is not initialized in this constructor nor in any functions that it calls.
   	6. uninit_member: Non-static class member key_size_ is not initialized in this constructor nor in any functions that it calls.

CID 1396161 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
8. uninit_member: Non-static class member val_size_ is not initialized in this constructor nor in any functions that it calls.
 65  CacheRecordHeader() {}
 66  CacheRecordHeader(const uint32_t magic, const uint32_t key_size,
 67                    const uint32_t val_size)
 68      : magic_(magic), crc_(0), key_size_(key_size), val_size_(val_size) {}
 69
   	1. member_decl: Class member declaration for magic_.
 70  uint32_t magic_;
   	3. member_decl: Class member declaration for crc_.
 71  uint32_t crc_;
   	5. member_decl: Class member declaration for key_size_.
 72  uint32_t key_size_;
   	7. member_decl: Class member declaration for val_size_.
 73  uint32_t val_size_;
 74};

utilities/simulator_cache/sim_cache.cc:
157        miss_times_(0),

CID 1396124 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member stats_ is not initialized in this constructor nor in any functions that it calls.
158        hit_times_(0) {}
159
```
Closes https://github.com/facebook/rocksdb/pull/3155

Differential Revision: D6427237

Pulled By: sagar0

fbshipit-source-id: 97e493da5fc043c5b9a3e0d33103442cffb75aad
2017-11-28 13:27:08 -08:00
Prashant D
7b57510a17 utilities: Fix coverity issues in blob_db and col_buf_decoder
Summary:
utilities/blob_db/blob_db_impl.cc
265                    : bdb_options_.blob_dir;
   	3. uninit_member: Non-static class member env_ is not initialized in this constructor nor in any functions that it calls.
   	5. uninit_member: Non-static class member ttl_extractor_ is not initialized in this constructor nor in any functions that it calls.
   	7. uninit_member: Non-static class member open_p1_done_ is not initialized in this constructor nor in any functions that it calls.

CID 1418245 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
9. uninit_member: Non-static class member debug_level_ is not initialized in this constructor nor in any functions that it calls.
266}

   	4. past_the_end: Function end creates an iterator.

CID 1418258 (#1 of 1): Using invalid iterator (INVALIDATE_ITERATOR)
5. deref_iterator: Dereferencing iterator file_nums.end() though it is already past the end of its container.

utilities/col_buf_decoder.h:
     nullable_(nullable),
   	2. uninit_member: Non-static class member remain_runs_ is not initialized in this constructor nor in any functions that it calls.
   	4. uninit_member: Non-static class member run_val_ is not initialized in this constructor nor in any functions that it calls.

CID 1396134 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member last_val_ is not initialized in this constructor nor in any functions that it calls.
 46        big_endian_(big_endian) {}
Closes https://github.com/facebook/rocksdb/pull/3134

Differential Revision: D6340607

Pulled By: sagar0

fbshipit-source-id: 25c52566e2ff979fe6c7abb0f40c27fc16597054
2017-11-28 12:27:57 -08:00
Tamir Duberstein
cf0d6aa007 CMake cross platform Java support and add JNI to travis
Summary:
Rewrite `java/CMakeLists.txt` to take advantage of CMake's cross platform
Java support.

adamretter
Closes https://github.com/facebook/rocksdb/pull/2301

Differential Revision: D5070724

Pulled By: sagar0

fbshipit-source-id: 999aee9bd39da2b24a5fe493a2eb0e9af6072dc7
2017-11-28 12:27:53 -08:00
Yi Wu
78279350aa Blob DB: Add statistics
Summary:
Adding a list of blob db counters.

Also remove WaStats() which doesn't expose the stats and can be substitute by (BLOB_DB_BYTES_WRITTEN / BLOB_DB_BLOB_FILE_BYTES_WRITTEN).
Closes https://github.com/facebook/rocksdb/pull/3193

Differential Revision: D6394216

Pulled By: yiwu-arbug

fbshipit-source-id: 017508c8ff3fcd7ea7403c64d0f9834b24816803
2017-11-28 11:58:49 -08:00
Yi Wu
3cf562be31 Fix IOError on WAL write doesn't propagate to write group follower
Summary:
This is a simpler version of #3097 by removing all unrelated changes.

Fixing the bug where concurrent writes may get Status::OK while it actually gets IOError on WAL write. This happens when multiple writes form a write batch group, and the leader get an IOError while writing to WAL. The leader failed to pass the error to followers in the group, and the followers end up returning Status::OK() while actually writing nothing. The bug only affect writes in a batch group. Future writes after the batch group will correctly return immediately with the IOError.
Closes https://github.com/facebook/rocksdb/pull/3201

Differential Revision: D6421644

Pulled By: yiwu-arbug

fbshipit-source-id: 1c2a455c5b73f6842423785eb8a9dbfbb191dc0e
2017-11-28 11:42:48 -08:00
Andrew Kryczka
1bdb44de95 optimize file ingestion checks for range deletion overlap
Summary:
Before we were checking every file in the level which was unnecessary. We can piggyback onto the code for checking point-key overlap, which already opens all the files that could possibly contain overlapping range deletions. This PR makes us check just the range deletions from those files, so no extra ones will be opened.
Closes https://github.com/facebook/rocksdb/pull/3179

Differential Revision: D6358125

Pulled By: ajkr

fbshipit-source-id: 00e200770fdb8f3cc6b1b2da232b755e4ba36279
2017-11-28 11:27:02 -08:00
Jeffrey Dallatezza
022c598abb Fix minor typo in comment
Summary:
mean -> meant
Closes https://github.com/facebook/rocksdb/pull/3202

Differential Revision: D6426443

Pulled By: sagar0

fbshipit-source-id: adaf07218580ee6903986fa5686de92f43f420e1
2017-11-28 11:27:02 -08:00