Commit Graph

10642 Commits

Author SHA1 Message Date
Akanksha Mahajan
44ac714808 Update History.md for the bug fix in RocksDB implicit prefetching in #9234 (#9237)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9237

Reviewed By: riversand963

Differential Revision: D32769906

Pulled By: akankshamahajan15

fbshipit-source-id: ef9185f57b7f7cb16daf412ae08104a3e2724191
2021-12-01 12:26:28 -08:00
mrambacher
7cd5835a28 Make RateLimiter Customizable (#9141)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9141

Reviewed By: zhichao-cao

Differential Revision: D32432190

Pulled By: mrambacher

fbshipit-source-id: 7930ed88a02412128cd407b5063522484e45c6ce
2021-12-01 06:57:02 -08:00
Akanksha Mahajan
04b2c16f9b Fix bug in rocksdb internal automatic prefetching (#9234)
Summary:
After introducing adaptive_readahead, the original flow got
broken. Readahead size was set to 0 because of which rocksdb wasn't be
able to do automatic prefetching which it enables after seeing
sequential reads. This PR fixes it.

----------------------------------------------------------------------------------------------------

Before this patch:
b_bench -use_existing_db=true -db=/tmp/prefix_scan -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 6.27
Date:       Tue Nov 30 11:56:50 2021
CPU:        24 * Intel Core Processor (Broadwell)
CPUCache:   16384 KB
Keys:       32 bytes each (+ 0 bytes user-defined timestamp)
Values:     512 bytes each (256 bytes after compression)
Entries:    5000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    2594.0 MB (estimated)
FileSize:   1373.3 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/prefix_scan]

seekrandom   : 5356367.174 micros/op 0 ops/sec;   29.4 MB/s (23 of 23 found)

----------------------------------------------------------------------------------------------------

After the patch:
./db_bench -use_existing_db=true -db=/tmp/prefix_scan -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680 -duration=120 -ops_between_duration_checks=1
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 6.27
Date:       Tue Nov 30 14:38:33 2021
CPU:        24 * Intel Core Processor (Broadwell)
CPUCache:   16384 KB
Keys:       32 bytes each (+ 0 bytes user-defined timestamp)
Values:     512 bytes each (256 bytes after compression)
Entries:    5000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    2594.0 MB (estimated)
FileSize:   1373.3 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/prefix_scan]
seekrandom   :  456504.277 micros/op 2 ops/sec;  359.8 MB/s (264 of 264 found)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9234

Test Plan:
Ran ./db_bench -db=/data/mysql/rocksdb/prefix_scan
-benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000000 -use_d
irect_io_for_flush_and_compaction=true -target_file_size_base=16777216
and then ./db_bench -use_existing_db=true
-db=/data/mysql/rocksdb/prefix_scan -benchmarks="seekrandom"
-key_size=32 -value_siz
e=512 -num=5000000 -use_direct_reads=true -seek_nexts=327680
-duration=120 -ops_between_duration_checks=1
 and compared the results.

Reviewed By: anand1976

Differential Revision: D32743965

Pulled By: akankshamahajan15

fbshipit-source-id: b950fba68c91963b7deb5c20acdf471bc60251f5
2021-11-30 22:53:10 -08:00
Yanqin Jin
924616526a Update WriteBatch::AssignTimestamp() and Add (#9205)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9205

Update WriteBatch::AssignTimestamp() APIs so that they take an
additional argument, i.e. a function object called `checker` indicating the user-specified logic of performing
checks on timestamp sizes.

WriteBatch is a building block used by multiple other RocksDB components, each of which may track
timestamp information in different data structures. For example, transaction can either write to
`WriteBatchWithIndex` which is a `WriteBatch` with index, or write directly to raw `WriteBatch` if
`Transaction::DisableIndexing()` is called.
`WriteBatchWithIndex` keeps mapping from column family id to comparator, and transaction needs
to keep similar information for the `WriteBatch` if user calls `Transaction::DisableIndexing()` (dynamically)
so that we will know the size of each timestamp later. The bookkeeping info maintained by `WriteBatchWithIndex`
and `Transaction` should not overlap.
When we later call `WriteBatch::AssignTimestamp()`, we need to use these data structures to guarantee
that we do not accidentally assign timestamps for keys from column families that disable timestamp.

Reviewed By: ltamasi

Differential Revision: D31735186

fbshipit-source-id: 8b1709ed880ac72f995aa9e012e5873b290840a7
2021-11-30 22:33:00 -08:00
Yanqin Jin
29954b8b57 Add initial CMake support to plugin (#9214)
Summary:
Not a CMake expert, and the current CMake build support added by this PR is
unlikely the best way of doing it. Sending out the PR to demonstrate it
can work.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9214

Test Plan:
Will need to update https://github.com/ajkr/dedupfs with CMake build.
Also, PR https://github.com/facebook/rocksdb/issues/9170 and PR https://github.com/facebook/rocksdb/issues/9206 both include CMake support for their
plugins, and can be used as a proof of concept.

Reviewed By: ajkr

Differential Revision: D32738273

Pulled By: riversand963

fbshipit-source-id: da87fb4377c716bbbd577a69763b48d22483f845
2021-11-30 17:16:53 -08:00
Artem Krylysov
552256cb1a Add rocksdb_livefiles_column_family_name C API (#9232)
Summary:
Extend C API to add new function `rocksdb_livefiles_column_family_name`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9232

Reviewed By: akankshamahajan15

Differential Revision: D32736516

Pulled By: ajkr

fbshipit-source-id: a854256a0f4652c903ab5ad8355ded051ac19987
2021-11-30 16:54:27 -08:00
Andreas Hindborg
074562226f Allow plugins to add pkg-config dependencies to rocksdb.pc (#9198)
Summary:
This patch fixes an issue that occur when dependencies of plugins are not
installed to the same prefix as librocksdb. Because plugin dependencies are
declared in the `Libs` field of rocksdb.pc, programs that link against
librocksdb with `pkg-config --libs rocksdb` will link with `-L` flag for the
path of librocksdb only. This patch allows plugin dependencies to be declared in
the `Requires` field of rocksdb.pc, so that pkg-config will correctly provide
`-L` flags for dependencies of plugins that are installed in other locations.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9198

Reviewed By: akankshamahajan15

Differential Revision: D32596620

Pulled By: ajkr

fbshipit-source-id: e17b2b6452b5f2e955b430140197c57e26a4a518
2021-11-30 15:14:44 -08:00
leipeng
c712b68f5b Fix num files in single compaction for universal compaction (#9168)
Summary:
https://github.com/facebook/rocksdb/issues/9026 fixed histogram NUM_FILES_IN_SINGLE_COMPACTION for level compaction, but missed fix for universal compaction.

This PR fixed NUM_FILES_IN_SINGLE_COMPACTION for universal compaction.

Quote from https://github.com/facebook/rocksdb/issues/9026:
> currently histogram `NUM_FILES_IN_SINGLE_COMPACTION` just counted files in first level of compaction input, this fix counts files in all levels of compaction input.

Thanks for ajkr pointed this missed fix!

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9168

Reviewed By: akankshamahajan15

Differential Revision: D32434494

Pulled By: ajkr

fbshipit-source-id: 93ea092af4afbd8dce67898ffb350cf26b065ed2
2021-11-30 15:11:21 -08:00
Peter Dillinger
e8b5d05e93 HISTORY for #9208 (#9227)
Summary:
Update HISTORY for bug fix. This is going into 6.27 initial
release. (Technically 6.27.1)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9227

Test Plan: n/a

Reviewed By: ajkr

Differential Revision: D32727912

Pulled By: pdillinger

fbshipit-source-id: 75e7a81749a188a590d44ef47e261eaaa8667152
2021-11-30 15:01:59 -08:00
Aravind Ramesh
9c932816cf db_stress: db_stress segmentation fault (#9219)
Summary:
db_stress asserts/seg-faults with below command (on debug and release builds)

```
"rm -rf /tmp/rocksdbtest*; db_stress --ops_per_thread=1000 --reopen=5"
=======================================
Error opening unique id file for append: IO error: No such file or directory:
While open a file for appending: /tmp/rocksdbtest-0/dbstress/.unique_ids:
No such file or directory
Choosing random keys with no overwrite
Creating 2621440 locks
Starting continuous_verification_thread
2021/11/15-08:46:49  Initializing worker threads
2021/11/15-08:46:49  Starting database operations
2021/11/15-08:46:49  Reopening database for the 1th time
WARNING: prefix_size is non-zero but memtablerep != prefix_hash
DB path: [/tmp/rocksdbtest-0/dbstress]
Segmentation fault
=======================================

```
StressTest() constructor deletes the directory "dbstress" because
the option --destroy_db_initially is true by default in db_stress.

This Seg fault happens on a new database, UniqueIdVerifier's constructor
tries to read the ".unique_ids" file, if the file is not present,
ReopenWritableFile() tries to create .unique_ids file, but fails
as the directory db_stress is not available. The data_file_writer_
is set as an invalid(null) pointer and in subsequent calls (~UniqueIdVerifier()
and UniqueIdVerifier::Verify()) it accesses this null pointer and crashes.

This patch creates db_stress directory if it is missing, so the .unique_ids file
is created.

Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9219

Reviewed By: ajkr

Differential Revision: D32730151

Pulled By: pdillinger

fbshipit-source-id: f47baba56b380d93c3ba5608904756e86bbf14f5
2021-11-30 14:56:13 -08:00
mrambacher
7aa31ba4a9 Fix GetOptionsPtr for Wrapped Customizable; Allow null options map (#9213)
Summary:
1.  Fix GetOptionsPtr for Wrapped (Inner() != nullptr) Customizable objects.  This allows the inner options to be returned via this method.

2.  Allow the option type map to be nullptr.  This allows objects to be registered as options (for GetOptionsPtr) but not be used by the configuration methods.

Added tests as appropriate.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9213

Reviewed By: zhichao-cao

Differential Revision: D32718882

Pulled By: mrambacher

fbshipit-source-id: 563203d1f006a2629060feb31c5dff9a233e1e83
2021-11-30 13:23:25 -08:00
Yanqin Jin
42fef0224f Fix build for msvc (#9230)
Summary:
Test plan

With Visual Studio 2017.
```
cd rocksdb
mkdir build && cd build
cmake -G "Visual Studio 15 Win64" -DWITH_GFLAGS=1 ..
MSBuild rocksdb.sln /m /TARGET:cache_bench /TARGET:db_bench /TARGET:db_stress
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9230

Reviewed By: akankshamahajan15

Differential Revision: D32705095

Pulled By: riversand963

fbshipit-source-id: 101e3533f5178b24c0535ddc47a39347ccfcf92c
2021-11-29 14:27:48 -08:00
Peter Dillinger
fc3a6eb74a Fix/improve 'must free heap allocations' code (#9209)
Summary:
Added missing include, and cleaned up to make same mistake less
likely in future (minimize conditional compilation)

Fixes https://github.com/facebook/rocksdb/issues/9183

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9209

Test Plan: added to existing test

Reviewed By: mrambacher

Differential Revision: D32631390

Pulled By: pdillinger

fbshipit-source-id: 63a0501855cf5fac9e22ca1e5c4f53725dbf3f93
2021-11-29 10:53:52 -08:00
Peter Dillinger
735fe61e8f Fix flaky CassandraFunctionalTest...ExpiredColumnsToTombstone (#9226)
Summary:
You could easily reproduce the failure by injecting sleep(11)
before `store.Flush()`. Fixed by setting TTL time to approximately test
timeout time.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9226

Test Plan: manual

Reviewed By: akankshamahajan15

Differential Revision: D32698105

Pulled By: pdillinger

fbshipit-source-id: 40529af9d9f2389585988b7c81dffb120e2795a2
2021-11-29 09:53:07 -08:00
Peter Dillinger
2a67d475f1 Fix bug affecting GetSortedWalFiles, Backups, Checkpoint (#9208)
Summary:
Saw error like this:
`Backup failed -- IO error: No such file or directory: While opening a
file for sequentially reading:
/dev/shm/rocksdb/rocksdb_crashtest_blackbox/004426.log: No such file or
directory`

Unfortunately, GetSortedWalFiles (used by Backups, Checkpoint, etc.)
relies on no file deletions happening while its operating, which
means not only disabling (more) deletions, but ensuring any pending
deletions are completed. Two fixes related to this:

* There was a gap in several places between decrementing
pending_purge_obsolete_files_ and incrementing bg_purge_scheduled_ where
the db mutex would be released and GetSortedWalFiles (and others) could
get false information that no deletions are pending.

* The fix to https://github.com/facebook/rocksdb/issues/8591 (disabling deletions in GetSortedWalFiles) seems
incomplete because it doesn't prevent pending deletions from occuring
during the operation (if deletions not already disabled, the case that
was to be fixed by the change).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9208

Test Plan:
existing tests (it's hard to write a test for interleavings
that are now excluded - this is what stress test is for)

Reviewed By: ajkr

Differential Revision: D32630675

Pulled By: pdillinger

fbshipit-source-id: a121e3da648de130cd24d44c524232f4eb22f178
2021-11-24 14:52:00 -08:00
jsteemann
5384f0af6e Fix compile warnings (#9199)
Summary:
* added missing override specifiers for overriden methods
  this fixes compiler warnings emitted by g++ and clang++ when compile option `-Wsuggest-override` is turned on.
* fix compile warning with -Wmaybe-uninitialized
  g++-11 warns about a _potentially_ uninitialized variable when using `-Wmaybe_uninitialized`:
  ```
      env/env.cc: In member function ‘virtual rocksdb::Status rocksdb::Env::GetHostNameString(std::string*)’:
      env/env.cc:738:66: error: ‘hostname_buf’ may be used uninitialized [-Werror=maybe-uninitialized]
        738 |   Status s = GetHostName(hostname_buf.data(), hostname_buf.size());
            |                                                                  ^
      In file included from /usr/include/c++/11/tuple:39,
                       from /usr/include/c++/11/functional:54,
                       from ./include/rocksdb/env.h:22,
                       from env/env.cc:10:
      /usr/include/c++/11/array:176:7: note: by argument 1 of type ‘const std::array<char, 256>*’ to ‘constexpr std::array<_Tp, _Nm>::size_type std::array<_Tp, _Nm>::size() const [with _Tp = char; long unsigned int _Nm = 256]’ declared here
        176 |       size() const noexcept { return _Nm; }
            |       ^~~~
      env/env.cc:737:37: note: ‘hostname_buf’ declared here
        737 |   std::array<char, kMaxHostNameLen> hostname_buf;
  ```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9199

Reviewed By: jay-zhuang

Differential Revision: D32630703

Pulled By: pdillinger

fbshipit-source-id: 9ea3010b1105a582548e3c3c0db4475b201e4a10
2021-11-24 11:19:06 -08:00
Peter Dillinger
ea3aa60dcc More improvements to output for CircleCI (#9201)
Summary:
More follow-up to https://github.com/facebook/rocksdb/issues/9193 + https://github.com/facebook/rocksdb/issues/9188
* Even though we need to print ETA updates to avoid hitting the 10min
timeout, we need to avoid printing an update if there's no actual
progress, so that hung tests will timeout after 10 min rather than 5
hours.
* When there is a hung test, it's really annoying to track down which
test is hung, so if no progress is observed for 1 minute, we run ps once
to show what is running.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9201

Test Plan: manual and CircleCI

Reviewed By: jay-zhuang

Differential Revision: D32612028

Pulled By: pdillinger

fbshipit-source-id: 00f8ea70fc5fec9ede28ff74287d90fc73854aad
2021-11-23 22:10:27 -08:00
Yanqin Jin
dc0ee3e5ca Fix internal build error (#9195)
Summary:
Internal build reported:
```
rocksdb/listener.h:470:3: error: extra ';' inside a struct [-Werror,-Wextra-semi]
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9195

Test Plan: import to fbcode and compile.

Reviewed By: akankshamahajan15

Differential Revision: D32590138

Pulled By: riversand963

fbshipit-source-id: ca4ed9cca210a1a9a12d3de17c789ef9151c57e8
2021-11-22 09:38:02 -08:00
Yanqin Jin
12e98add68 Print file checksum in hex (#9196)
Summary:
Printing file checksum (usually an integer) in non-hex format is barely useful. To make the matter
worse, it can mess with the output format. If you use `less` to redirect the output of `ldb manifest_dump`,
non-hex file checksum can cause `less` not to function as expected.

Also output some additional fields to json output.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9196

Test Plan: manually test `ldb manifest_dump`.

Reviewed By: ajkr

Differential Revision: D32590253

Pulled By: riversand963

fbshipit-source-id: de434b7e60dd05b0b7cb76eff2240b21f9ae4b32
2021-11-22 09:30:47 -08:00
Peter Dillinger
d561432de8 Fix some CI output (#9193)
Summary:
Address some issues with https://github.com/facebook/rocksdb/issues/9188
* Internal CI doesn't render \r as anything, so use \n for "not
connected to terminal" case
* CircleCI apparently uses a pseudo-tty for output and although
rerdirect stdout (because of EAGAIN bug) we don't redirect stderr, so it
is detected as a terminal-connected case. Fix by redirecting stderr
also.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9193

Test Plan: manual, CI

Reviewed By: akankshamahajan15

Differential Revision: D32581128

Pulled By: pdillinger

fbshipit-source-id: 5ae7c3209128d8dbd4153c5b9fdb2b810e6deb2e
2021-11-20 10:21:58 -08:00
Zhichao Cao
4340e1ff6d Disable the QPS verification in test temporally (#9190)
Summary:
Disable the QPS verification in test temporally, which causes the test failure due to different system delays.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9190

Test Plan: make check

Reviewed By: siying

Differential Revision: D32576289

Pulled By: zhichao-cao

fbshipit-source-id: 1df972e77dd82eed5af3462e5db5e141aadf8fae
2021-11-19 23:47:43 -08:00
Yanqin Jin
8101643611 Update HISTORY and version.h for 6.27 release (#9192)
Summary:
As title.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9192

Reviewed By: ltamasi

Differential Revision: D32578141

Pulled By: riversand963

fbshipit-source-id: 16216451c87e383ca8fd309acf15106e46172aaa
2021-11-19 22:11:56 -08:00
Levi Tamasi
3a9f557451 Update HISTORY for PR 9187 (#9191)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9191

Reviewed By: riversand963

Differential Revision: D32577939

Pulled By: ltamasi

fbshipit-source-id: 3c52067a0c3e9219c1aafdb711718dfcce5dedf5
2021-11-19 20:07:11 -08:00
Yanqin Jin
43ac7a2774 Fix an assertion failure when ManifestTailer switches to new Manifest in multi-cf mode (#9143)
Summary:
Original unit test fail to test the case of multi-cf mode switching to new manifest. The assertion
failure will trigger when the primary instance reopens and secondary continues to tail the
newly-created MANIFEST. Fix the assertion failure and update existing unit tests.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9143

Test Plan: make check

Reviewed By: ltamasi

Differential Revision: D32574233

Pulled By: riversand963

fbshipit-source-id: 857ddbe994019091276458abebcf8e2b65340468
2021-11-19 19:53:40 -08:00
Levi Tamasi
dc5de45af8 Support readahead during compaction for blob files (#9187)
Summary:
The patch adds a new BlobDB configuration option `blob_compaction_readahead_size`
that can be used to enable prefetching data from blob files during compaction.
This is important when using storage with higher latencies like HDDs or remote filesystems.
If enabled, prefetching is used for all cases when blobs are read during compaction,
namely garbage collection, compaction filters (when the existing value has to be read from
a blob file), and `Merge` (when the value of the base `Put` is stored in a blob file).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9187

Test Plan: Ran `make check` and the stress/crash test.

Reviewed By: riversand963

Differential Revision: D32565512

Pulled By: ltamasi

fbshipit-source-id: 87be9cebc3aa01cc227bec6b5f64d827b8164f5d
2021-11-19 17:53:47 -08:00
Peter Dillinger
cd4ea675e3 Fix backward compatibility with 2.5 through 2.7 (#9189)
Summary:
A bug in https://github.com/facebook/rocksdb/issues/9163 can cause checksum verification to fail if
parsing a properties block fails.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9189

Test Plan:
check_format_compatible.sh (never quite works locally but
this particular case seems fixed using variants of SHORT_TEST=1).
And added new unit test case.

Reviewed By: ajkr

Differential Revision: D32574626

Pulled By: pdillinger

fbshipit-source-id: 6fa5c8595737b71a3c3d011a52daf6d6c08715d7
2021-11-19 17:31:01 -08:00
Jay Zhuang
6cde8d2190 Deprecating iter_start_seqnum and preserve_deletes (#9091)
Summary:
`ReadOptions::iter_start_seqnum` and `DBOptions::preserve_deletes` are
deprecated, please try using user defined timestamp feature instead.
The feature is used to support differential snapshots, but not well
maintained (https://github.com/facebook/rocksdb/issues/6837, https://github.com/facebook/rocksdb/issues/8472) and the interface is not user friendly which
returns an internal key from the iterator. The user defined timestamp
feature is a more flexible feature to support similar usecase, please
switch to that if you have such usecase.
The deprecated feature will be removed in a future release.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9091

Test Plan:
check LOG

Fix https://github.com/facebook/rocksdb/issues/9090

Reviewed By: ajkr

Differential Revision: D32071750

Pulled By: jay-zhuang

fbshipit-source-id: b882c4668dd1bf26ce03c4c192f1bba584bf6104
2021-11-19 16:55:45 -08:00
Peter Dillinger
3ce4d4f558 Print failures in parallel make check (#9188)
Summary:
Generating megabytes of successful test output has caused
issues / inconveniences for CI and internal sandcastle runs. This
changes their configuration to only print output from failed tests.
(Successful test output is still available in files under t/.)

This likewise changes default behavior of parallel `make check` as
a quick team poll showed interest in that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9188

Test Plan:
Seed some test failures and observe
* `make -j24 check` (new behavior)
* `PRINT_PARALLEL_OUTPUTS=1 make -j24 check` (old CI behavior)
* `QUIET_PARALLEL_TESTS=1 make -j24 check` (old manual run behavior)

Reviewed By: siying

Differential Revision: D32567392

Pulled By: pdillinger

fbshipit-source-id: 8d8fb64aebd16bca103b11e3bd1f13c488a69611
2021-11-19 16:55:45 -08:00
Adam Retter
ad40b0bee2 Some small changes to RocksJava build (#9186)
Summary:
1. Added a target for building a bundle jar for Sonatype Nexus - sometimes if the OSS Maven Central is misbehaving, it is quicker to upload a bundle to be processed for release.

2. Simplify the publish code by using a for-loop.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9186

Reviewed By: jay-zhuang

Differential Revision: D32564469

Pulled By: ajkr

fbshipit-source-id: aaceac27e9143fb65b61dad2a46df346586672cd
2021-11-19 12:23:15 -08:00
slk
e12753eb71 Track each SST's timestamp information as user properties (#9093)
Summary:
Track each SST's timestamp information as user properties https://github.com/facebook/rocksdb/issues/8959

Rockdb has supported user-defined timestamp feature. Application can specify a timestamp
when writing each k-v pair. When data flush from memory to disk file called SST files.
Each SST files consist of multiple data blocks and several metadata blocks. Among the metadata
blocks, there is one called Properties block that tracks some pre-defined properties of this SST file.

This PR is for collecting the properties of min and max timestamps of all keys in the file. With those
properties the SST file is more convenient to tell whether the keys in the SST have timestamps or not.

The changes involved are as follows:

1) Add a class TimestampTablePropertiesCollector to collect min/max timestamp when add keys to table,
   The way TimestampTablePropertiesCollector use to compare timestamp of key should defined by
   user by implementing the Comparator::CompareTimestamp function in the user defined comparator.
2) Add corresponding unit tests.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9093

Reviewed By: ltamasi

Differential Revision: D32406927

Pulled By: riversand963

fbshipit-source-id: 25922971b7e67bacf4d53a1fb67c4c5ddaa61573
2021-11-19 11:37:06 -08:00
sdong
12117b26a3 Fix flaky DBTest2.RateLimitedCompactionReads (#9185)
Summary:
DBTest2.RateLimitedCompactionReads sometime shows following failure:

  what():  db/db_test2.cc:3976: Failure
Expected equality of these values:
  i + 1
    Which is: 4
  NumTableFilesAtLevel(0)
    Which is: 0

The assertion itself doesn't appear to be correct. Fix it.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9185

Test Plan: Removing an assertion shouldn't break anything.

Reviewed By: ajkr

Differential Revision: D32549530

fbshipit-source-id: 9993372d8af89161f903337a13f3e316e690a6b8
2021-11-19 10:08:59 -08:00
Yanqin Jin
1e8322c0f5 Fix a bug in FlushJob picking more memtables beyond synced WALs (#9142)
Summary:
After RocksDB 6.19 and before this PR, RocksDB FlushJob may pick more memtables to flush beyond synced WALs.
This can be problematic if there are multiple column families, since it can prematurely advance the flushed column
family's log_number. Should subsequent attempts fail to sync the latest WALs and the database goes
through a recovery, it may detect corrupted WAL number below the flushed column family's log number
and complain about column family inconsistency.
To fix, we record the maximum memtable ID of the column family being flushed. Then we call SyncClosedLogs()
so that all closed WALs at the time when memtable ID is recorded will be synced.
I also disabled a unit test temporarily due to reasons described in https://github.com/facebook/rocksdb/issues/9151

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9142

Test Plan: make check

Reviewed By: ajkr

Differential Revision: D32299956

Pulled By: riversand963

fbshipit-source-id: 0da75888177d91905cf8c9d00605b73afb5970a7
2021-11-19 09:56:00 -08:00
Andrew Kryczka
8cf4294e25 Adhere to per-DB concurrency limit when bottom-pri compactions exist (#9179)
Summary:
- Fixed bug where bottom-pri manual compactions were counting towards `bg_compaction_scheduled_` instead of `bg_bottom_compaction_scheduled_`. It seems to have no negative effect.
- Fixed bug where automatic compaction scheduling did not consider `bg_bottom_compaction_scheduled_`. Now automatic compactions cannot be scheduled that exceed the per-DB compaction concurrency limit (`max_compactions`) when some existing compactions are bottommost.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9179

Test Plan: new unit test for manual/automatic. Also verified the existing automatic/automatic test ("ConcurrentBottomPriLowPriCompactions") hanged until changing it to explicitly enable concurrency.

Reviewed By: riversand963

Differential Revision: D32488048

Pulled By: ajkr

fbshipit-source-id: 20c4c0693678e81e43f85ed3cc3402fcf26e3310
2021-11-18 17:31:50 -08:00
Akanksha Mahajan
4a7c1dc375 Add listener API that notifies on IOError (#9177)
Summary:
Add a new API in listener.h that notifies about IOErrors on
Read/Write/Append/Flush etc. The API reports about IOStatus, filename, Operation
name, offset and length.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9177

Test Plan: Added new unit tests

Reviewed By: anand1976

Differential Revision: D32470627

Pulled By: akankshamahajan15

fbshipit-source-id: 189a717033590ae227b3beae8b1e7e185e4cdc12
2021-11-18 17:11:19 -08:00
Adam Retter
d94932323a Check that newIteratorWithBase regardless of WBWI Overwrite Mode (#8134)
Summary:
The behaviour of WBWI has changed when calling newIteratorWithBase when overwrite is set to true or false. This PR simply adds tests to assert the new correct behaviour.

Closes https://github.com/facebook/rocksdb/issues/7370
Closes https://github.com/facebook/rocksdb/pull/8134

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9107

Reviewed By: pdillinger

Differential Revision: D32099475

Pulled By: mrambacher

fbshipit-source-id: 245f483f73db866cc8a51219a2bff2e09e59faa0
2021-11-18 11:53:09 -08:00
Peter Dillinger
230660be73 Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)

Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163

Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.

Reviewed By: ajkr, mrambacher

Differential Revision: D32514757

Pulled By: pdillinger

fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 11:43:44 -08:00
Zhichao Cao
f4294669e0 Fix the analyzer test failure caused by inaccurate timing wait (#9181)
Summary:
Fix the analyzer test failure caused by inaccurate timing wait. The wait time at different system might be different or cause the delay, now we do not accurately count the lines. Only in a very rare extreme case, test will ignore the part exceed the timing of 1 second.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9181

Test Plan: make check

Reviewed By: pdillinger

Differential Revision: D32511319

Pulled By: zhichao-cao

fbshipit-source-id: e694c8cb465c750cfa5a43dab3eff6707b9a11c8
2021-11-18 11:28:38 -08:00
Hui Xiao
74544d582f Account Bloom/Ribbon filter construction memory in global memory limit (#9073)
Summary:
Note: This PR is the 4th part of a bigger PR stack (https://github.com/facebook/rocksdb/pull/9073) and will rebase/merge only after the first three PRs (https://github.com/facebook/rocksdb/pull/9070, https://github.com/facebook/rocksdb/pull/9071, https://github.com/facebook/rocksdb/pull/9130) merge.

**Context:**
Similar to https://github.com/facebook/rocksdb/pull/8428, this PR is to track memory usage during (new) Bloom Filter (i.e,FastLocalBloom) and Ribbon Filter (i.e, Ribbon128) construction, moving toward the goal of [single global memory limit using block cache capacity](https://github.com/facebook/rocksdb/wiki/Projects-Being-Developed#improving-memory-efficiency). It also constrains the size of the banding portion of Ribbon Filter during construction by falling back to Bloom Filter if that banding is, at some point, larger than the available space in the cache under `LRUCacheOptions::strict_capacity_limit=true`.

The option to turn on this feature is `BlockBasedTableOptions::reserve_table_builder_memory = true` which by default is set to `false`. We [decided](https://github.com/facebook/rocksdb/pull/9073#discussion_r741548409) not to have separate option for separate memory user in table building therefore their memory accounting are all bundled under one general option.

**Summary:**
- Reserved/released cache for creation/destruction of three main memory users with the passed-in `FilterBuildingContext::cache_res_mgr` during filter construction:
   - hash entries (i.e`hash_entries`.size(), we bucket-charge hash entries during insertion for performance),
   - banding (Ribbon Filter only, `bytes_coeff_rows` +`bytes_result_rows` + `bytes_backtrack`),
   - final filter (i.e, `mutable_buf`'s size).
      - Implementation details: in order to use `CacheReservationManager::CacheReservationHandle` to account final filter's memory, we have to store the `CacheReservationManager` object and `CacheReservationHandle` for final filter in `XXPH3BitsFilterBuilder` as well as  explicitly delete the filter bits builder when done with the final filter in block based table.
- Added option fo run `filter_bench` with this memory reservation feature

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9073

Test Plan:
- Added new tests in `db_bloom_filter_test` to verify filter construction peak cache reservation under combination of  `BlockBasedTable::Rep::FilterType` (e.g, `kFullFilter`, `kPartitionedFilter`), `BloomFilterPolicy::Mode`(e.g, `kFastLocalBloom`, `kStandard128Ribbon`, `kDeprecatedBlock`) and `BlockBasedTableOptions::reserve_table_builder_memory`
  - To address the concern for slow test: tests with memory reservation under `kFullFilter` + `kStandard128Ribbon` and `kPartitionedFilter` take around **3000 - 6000 ms** and others take around **1500 - 2000 ms**, in total adding **20000 - 25000 ms** to the test suit running locally
- Added new test in `bloom_test` to verify Ribbon Filter fallback on large banding in FullFilter
- Added test in `filter_bench` to verify that this feature does not significantly slow down Bloom/Ribbon Filter construction speed. Local result averaged over **20** run as below:
   - FastLocalBloom
      - baseline `./filter_bench -impl=2 -quick -runs 20 | grep 'Build avg'`:
         - **Build avg ns/key: 29.56295** (DEBUG_LEVEL=1), **29.98153** (DEBUG_LEVEL=0)
      - new feature (expected to be similar as above)`./filter_bench -impl=2 -quick -runs 20 -reserve_table_builder_memory=true | grep 'Build avg'`:
         - **Build avg ns/key: 30.99046** (DEBUG_LEVEL=1), **30.48867** (DEBUG_LEVEL=0)
      - new feature of RibbonFilter with fallback  (expected to be similar as above) `./filter_bench -impl=2 -quick -runs 20 -reserve_table_builder_memory=true -strict_capacity_limit=true | grep 'Build avg'` :
         - **Build avg ns/key: 31.146975** (DEBUG_LEVEL=1), **30.08165** (DEBUG_LEVEL=0)

    - Ribbon128
       - baseline `./filter_bench -impl=3 -quick -runs 20 | grep 'Build avg'`:
           - **Build avg ns/key: 129.17585** (DEBUG_LEVEL=1), **130.5225** (DEBUG_LEVEL=0)
       - new feature  (expected to be similar as above) `./filter_bench -impl=3 -quick -runs 20 -reserve_table_builder_memory=true | grep 'Build avg' `:
           - **Build avg ns/key: 131.61645** (DEBUG_LEVEL=1), **132.98075** (DEBUG_LEVEL=0)
       - new feature of RibbonFilter with fallback (expected to be a lot faster than above due to fallback) `./filter_bench -impl=3 -quick -runs 20 -reserve_table_builder_memory=true -strict_capacity_limit=true | grep 'Build avg'` :
          - **Build avg ns/key: 52.032965** (DEBUG_LEVEL=1), **52.597825** (DEBUG_LEVEL=0)
          - And the warning message of `"Cache reservation for Ribbon filter banding failed due to cache full"` is indeed logged to console.

Reviewed By: pdillinger

Differential Revision: D31991348

Pulled By: hx235

fbshipit-source-id: 9336b2c60f44d530063da518ceaf56dac5f9df8e
2021-11-18 09:42:20 -08:00
Peter Dillinger
4f678b52e7 Don't allow parallel crash_test in Makefile (#9180)
Summary:
Using deps for running blackbox and whitebox allows them to be
parallelized, which doesn't seem to be working well.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9180

Test Plan: make -j24 crash_test

Reviewed By: siying

Differential Revision: D32500851

Pulled By: pdillinger

fbshipit-source-id: 364288c8d023b93e7ca2724ea40edae2f4eb0407
2021-11-17 11:19:48 -08:00
Davide Angelocola
c9539ede76 Fix integer overflow in TraceOptions (#9157)
Summary:
Hello from a happy user of rocksdb java :-)

Default constructor of TraceOptions is supposed to initialize size to 64GB but the expression contains an integer overflow.

Simple test case with JShell:
```
jshell> 64 * 1024 * 1024 * 1024
$1 ==> 0

jshell> 64L * 1024 * 1024 * 1024
$2 ==> 68719476736
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9157

Reviewed By: pdillinger, zhichao-cao

Differential Revision: D32369273

Pulled By: mrambacher

fbshipit-source-id: 6a0c95fff7a91f27ff15d65b662c6b101756b450
2021-11-17 08:41:48 -08:00
Andrew Kryczka
2225f063d4 Remove incremental ID from background thread pool names (#9165)
Summary:
`pthread_setname_np()` fails on attempts to assign oversized names like
"rocksdb:bottom10", which resulted in some thread name updates being
lost. We do not need the ID suffix so I removed it.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9165

Test Plan:
```
$ TEST_TMPDIR=/dev/shm ./db_bench -benchmarks=fillrandom -max_background_flushes=123 -max_background_compactions=456 -num_bottom_pri_threads=789 -duration=60
```

While above is running:
```
$ ps -o 'comm' -Lp `pidof db_bench` | grep '^rocksdb:' | sort | uniq -c
    789 rocksdb:bottom
    123 rocksdb:high
    456 rocksdb:low
```

Reviewed By: pdillinger

Differential Revision: D32415077

Pulled By: ajkr

fbshipit-source-id: a0e013101e26a78bc5eca73509293ef4bf22254f
2021-11-16 18:26:12 -08:00
Peter Dillinger
d95ffbaf4f Parallelize sandcastle tests (#9178)
Summary:
Some tests are timing out, so increase parallelism but also
keep test output on console

Large credit to jay-zhuang who is currently unavailable to revise & land https://github.com/facebook/rocksdb/issues/9135

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9178

Test Plan:
make check and valgrind_test, with and without PRINT_PARALLEL_OUTPUTS=1
https://www.internalfb.com/intern/sandcastle/group/nonce/6211105410048528/

Reviewed By: riversand963

Differential Revision: D32476680

Pulled By: pdillinger

fbshipit-source-id: 8844947416e5baf4435e1ef6e33aeecc45621a89
2021-11-16 16:45:13 -08:00
Zhichao Cao
b694cd0e0d Add tiered storage related read bytes stats to Statistic (#9123)
Summary:
Add the 3 read bytes counter to the Statistic, which will be used by storage tiering and get the information for files with different temperature.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9123

Test Plan: added new testing cases.

Reviewed By: siying

Differential Revision: D32154745

Pulled By: zhichao-cao

fbshipit-source-id: b7905d6dae469a72428742364ec07b634b6f15da
2021-11-16 15:17:17 -08:00
Sahir Hoda
1178d348aa Fix portable mac shared_library ld flags (#9149)
Summary:
Move the 'macosx-version-min' arg to the front of PLATFORM_SHARED_LDFLAGS so that it doesn't get concatenated with the library name. Fixes https://github.com/facebook/rocksdb/issues/9146

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9149

Reviewed By: mrambacher

Differential Revision: D32396101

Pulled By: pdillinger

fbshipit-source-id: aefcf53384e64d399049f158779acc3a4e54a8fe
2021-11-16 12:17:17 -08:00
Peter Dillinger
f8c685c4fc Check for and disallow shared key space in block caches (#9172)
Summary:
We have three layers of block cache that often use the same key
but map to different physical data:
* BlockBasedTableOptions::block_cache
* BlockBasedTableOptions::block_cache_compressed
* BlockBasedTableOptions::persistent_cache

If any two of these happen to share an underlying implementation and key
space (insertion into one shows up in another), then memory safety is
broken. The simplest case is block_cache == block_cache_compressed.
(Credit mrambacher for asking about this case in a review.)

With this change, we explicitly check for overlap and preemptively and
safely fail with a Status code.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9172

Test Plan: test added. Crashes without new check

Reviewed By: anand1976

Differential Revision: D32465659

Pulled By: pdillinger

fbshipit-source-id: 3876b45b6dce6167e5a7a642725ddc86b96f8e40
2021-11-16 11:16:05 -08:00
Adam Simpkins
28f54e71f3 fix compile errors in db/kv_checksum.h (#9173)
Summary:
When defining a template class, the constructor should be specified
simply using the class name; it does not take template arguments.a

Apparently older versions of gcc and clang did not complain about this
syntax, but gcc 11.x and recent versions of clang both complain about
this file.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9173

Test Plan:
When building with platform010 I got compile errors in this file both
in `mode/dev` (clang) and in `mode/opt-gcc`.  This diff fixes the
compile failures.

Reviewed By: ajkr

Differential Revision: D32455881

Pulled By: simpkins

fbshipit-source-id: 0682910d9e2cdade94ce1e77973d47ac04d9f7e2
2021-11-16 10:20:50 -08:00
Peter Dillinger
230f18b6e1 Improve parallel test suite runner (#9160)
Summary:
* Parallel `make check` would pass if a test binary failed to list gtest
tests. This is now likely to report as a failure.
* Crazy perl was generating some extra incorrect test names causing
extra files and binary invocations. Fixed with cleaner awk.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9160

Test Plan:
For first part, add an 'assert(false);' to start of hash_test main and
see 'make check' pass before, and fail after.

For second part, inspect t/ directory before vs. after. Number of
executed tests is same:

    $ cat log* | grep 'PASSED.*test' | awk '{ tot += $4; } END { print tot; }'
    10469

Reviewed By: ajkr

Differential Revision: D32372006

Pulled By: pdillinger

fbshipit-source-id: 185b3db2b67e3f9198eb75322e4d0493e4fc1beb
2021-11-16 09:59:21 -08:00
Hui Xiao
cff7819dff Fix BackupEngine's internal callers of GenericRateLimiter::Request() not honoring bytes <= GetSingleBurstBytes() (#9063)
Summary:
**Context:**
Some existing internal calls of `GenericRateLimiter::Request()` in backupable_db.cc and newly added internal calls in https://github.com/facebook/rocksdb/pull/8722/ do not make sure `bytes <= GetSingleBurstBytes()` as required by rate_limiter https://github.com/facebook/rocksdb/blob/master/include/rocksdb/rate_limiter.h#L47.

**Impacts of this bug include:**
(1) In debug build, when `GenericRateLimiter::Request()` requests bytes greater than `GenericRateLimiter:: kMinRefillBytesPerPeriod = 100` byte, process will crash due to assertion failure. See https://github.com/facebook/rocksdb/pull/9063#discussion_r737034133 and for possible scenario
(2) In production build, although there will not be the above crash due to disabled assertion, the bug can lead to a request of small bytes being blocked for a long time by a request of same priority with insanely large bytes from a different thread. See updated https://github.com/facebook/rocksdb/wiki/Rate-Limiter ("Notice that although....the maximum bytes that can be granted in a single request have to be bounded...") for more info.

There is an on-going effort to move rate-limiting to file wrapper level so rate limiting in `BackupEngine` and this PR might be made obsolete in the future.

**Summary:**
- Implemented loop-calling `GenericRateLimiter::Request()` with `bytes <= GetSingleBurstBytes()` as a static private helper function `BackupEngineImpl::LoopRateLimitRequestHelper`
   -- Considering make this a util function in `RateLimiter` later or do something with `RateLimiter::RequestToken()`
- Replaced buggy internal callers with this helper function wherever requested byte is not pre-limited by `GetSingleBurstBytes()`
- Removed the minimum refill bytes per period enforced by `GenericRateLimiter` since it is useless and prevents testing `GenericRateLimiter` for extreme case with small refill bytes per period.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9063

Test Plan:
- Added a new test that failed the assertion before this change and now passes
  - It exposed bugs in [the write during creation in `CopyOrCreateFile()`](df7cc66e17/utilities/backupable/backupable_db.cc (L2034-L2043)), [the read of table properties in `GetFileDbIdentities()`](df7cc66e17/utilities/backupable/backupable_db.cc (L2372-L2378)), [some read of metadata in `BackupMeta::LoadFromFile()`](df7cc66e17/utilities/backupable/backupable_db.cc (L2726))
- Passing Existing tests

Reviewed By: ajkr

Differential Revision: D31824535

Pulled By: hx235

fbshipit-source-id: d2b3dea7a64e2a4b1e6a59fca322f0800a4fcbcc
2021-11-16 09:52:16 -08:00
Yanqin Jin
2035798834 Update TransactionUtil::CheckKeyForConflict to also use timestamps (#9162)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9162

Existing TransactionUtil::CheckKeyForConflict() performs only seq-based
conflict checking. If user-defined timestamp is enabled, it should perform
conflict checking based on timestamps too.

Update TransactionUtil::CheckKey-related methods to verify the timestamp of the
latest version of a key is smaller than the read timestamp. Note that
CheckKeysForConflict() is not updated since it's used only by optimistic
transaction, and we do not plan to update it in this upcoming batch of diffs.

Existing GetLatestSequenceForKey() returns the sequence of the latest
version of a specific user key. Since we support user-defined timestamp, we
need to update this method to also return the timestamp (if enabled) of the
latest version of the key. This will be needed for snapshot validation.

Reviewed By: ltamasi

Differential Revision: D31567960

fbshipit-source-id: 2e4a14aed267435a9aa91bc632d2411c01946d44
2021-11-15 12:52:18 -08:00
Andrew Kryczka
9bb13c56b3 Use system-wide thread ID in info log lines (#9164)
Summary:
This makes it easier to debug with tools like `ps`. The change only
applies to builds with glibc 2.30+ and _GNU_SOURCE extensions enabled.
We could adopt it in more cases by using the syscall but this is enough
for our build.

Replaces https://github.com/facebook/rocksdb/issues/2973.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9164

Test Plan:
- ran some benchmarks and correlated logged thread IDs with those shown by `ps -L`.
- verified no noticeable regression in throughput for log heavy (more than 700k log lines and over 5k / second) scenario.

Benchmark command:

```
$ TEST_TMPDIR=/dev/shm ./db_bench -benchmarks=filluniquerandom -compression_type=none -max_bytes_for_level_multiplier=2 -write_buffer_size=262144 -num_levels=7 -max_bytes_for_level_base=2097152 -target_file_size_base=524288 -level_compaction_dynamic_level_bytes=true -max_background_jobs=12 -num=20000000
```

Results before: 15.9MB/s, 15.8MB/s, 16.0MB/s
Results after: 16.3MB/s, 16.3MB/s, 15.8MB/s

- Rely on CI to test the fallback behavior

Reviewed By: riversand963

Differential Revision: D32399660

Pulled By: ajkr

fbshipit-source-id: c24d44fdf7782faa616ef0a0964eaca3539d9c24
2021-11-12 19:46:06 -08:00