Commit Graph

6287 Commits

Author SHA1 Message Date
Andrew Kryczka
a5c8b54340 fix a header include
Summary:
need to consistently include "rocksdb/persistent_cache.h" to fix internal build
Closes https://github.com/facebook/rocksdb/pull/2034

Differential Revision: D4768101

Pulled By: ajkr

fbshipit-source-id: 2ecb07f
2017-03-23 19:24:22 -07:00
Siying Dong
91b5feb37b Fix Windows Build broken by a recent commit
Summary: Closes https://github.com/facebook/rocksdb/pull/2032

Differential Revision: D4766260

Pulled By: siying

fbshipit-source-id: 415daa4
2017-03-23 18:09:57 -07:00
Warren Falk
41ccae6d26 Add C API functions (and tests) for WriteBatchWithIndex
Summary:
I've added functions to the C API to support WriteBatchWithIndex as requested in #1833.

I've also added unit tests to c_test

I've implemented the WriteBatchWithIndex variation of every function available for regular WriteBatch.  And added additional functions unique to WriteBatchWithIndex.

For now, the following is omitted:
  1. The ability to create WriteBatchWithIndex's custom batch-only iterator as I'm not sure what its purpose is.  It should be possible to add later if anyone wants it.
  2. The ability to create the batch with a fallback comparator, since it appears to be unnecessary.  I believe the column family comparator will be used for this, meaning those using a custom comparator can just use the column family variations.
Closes https://github.com/facebook/rocksdb/pull/1985

Differential Revision: D4760039

Pulled By: siying

fbshipit-source-id: 393227e
2017-03-23 15:54:13 -07:00
Siying Dong
88bb6f6bfa non_shm CI should run tests on /tmp
Summary:
Since non_shn CI was made to run in parallel, /dev/shm is automatically used. It defeated the purpose of the test to cover a non-ramfs file system.
Closes https://github.com/facebook/rocksdb/pull/2031

Differential Revision: D4764804

Pulled By: siying

fbshipit-source-id: 5666bda
2017-03-23 15:24:12 -07:00
Sagar Vemuri
8888de2b13 Update .gitignore file in examples
Summary:
options_file_example should be added in .gitignore so that it does not show up as an untracked file in `git status`.
Closes https://github.com/facebook/rocksdb/pull/2026

Differential Revision: D4759402

Pulled By: sagar0

fbshipit-source-id: d7fe133
2017-03-23 11:24:14 -07:00
Ayappan
203136e793 Fix Compilation errors when using IBM Java
Summary:
PR to fix this issue -> https://github.com/facebook/rocksdb/issues/1926
Closes https://github.com/facebook/rocksdb/pull/1965

Differential Revision: D4682411

Pulled By: siying

fbshipit-source-id: a519be1
2017-03-22 18:09:10 -07:00
Daniel Black
f4fce4751e Fix clang compile error - [-Werror,-Wunused-lambda-capture]
Summary:
Errors where:

db/version_set.cc:1535:20: error: lambda capture 'this' is not used [-Werror,-Wunused-lambda-capture]
                  [this](const Fsize& f1, const Fsize& f2) -> bool {
                   ^
db/version_set.cc:1541:20: error: lambda capture 'this' is not used [-Werror,-Wunused-lambda-capture]
                  [this](const Fsize& f1, const Fsize& f2) -> bool {
                   ^
db/db_test.cc:2983:27: error: lambda capture 'kNumPutsBeforeWaitForFlush' is not required to be captured for this use [-Werror,-Wunused-lambda-capture]
  auto gen_l0_kb = [this, kNumPutsBeforeWaitForFlush](int size) {
                          ^
Closes https://github.com/facebook/rocksdb/pull/1972

Differential Revision: D4685991

Pulled By: siying

fbshipit-source-id: 9125379
2017-03-22 18:09:10 -07:00
Siying Dong
a084b26a5a Blog post for releasing 5.2.1
Summary: Closes https://github.com/facebook/rocksdb/pull/2025

Differential Revision: D4759957

Pulled By: siying

fbshipit-source-id: 5b5b625
2017-03-22 17:54:18 -07:00
Siying Dong
15950fe3a0 Remove ASSERT_EQ(boolean, ...)
Summary: Closes https://github.com/facebook/rocksdb/pull/2024

Differential Revision: D4755420

Pulled By: siying

fbshipit-source-id: 7332ab1
2017-03-22 15:54:12 -07:00
Aaron Gao
3e56c7e0c4 make total_log_size_ atomic
Summary:
make total_log_size_ atomic to avoid overflow caused by data race.
Closes https://github.com/facebook/rocksdb/pull/2019

Differential Revision: D4751391

Pulled By: siying

fbshipit-source-id: fac01dd
2017-03-22 11:54:40 -07:00
Siying Dong
909028e214 HISTORY.md for log_size_for_flush in CreateCheckpoint()
Summary: Closes https://github.com/facebook/rocksdb/pull/2021

Differential Revision: D4755324

Pulled By: siying

fbshipit-source-id: c8d7955
2017-03-22 11:24:12 -07:00
Dmitri Smirnov
be723a8d8c Optionally construct Post Processing Info map in MemTableInserter
Summary:
MemTableInserter default constructs Post processing info
  std::map. However, on Windows with 2015 STL the default
  constructed map still dynamically allocates one node
  which shows up on a profiler and we loose ~40% throughput
  on fillrandom benchmark.
  Solution: declare a map as std::aligned storage and optionally
  construct.

This addresses https://github.com/facebook/rocksdb/issues/1976

Before:
-------------------------------------------------------------------
  Initializing RocksDB Options from command-line flags
  DB path: [k:\data\BulkLoadRandom_10M_fillonly]
  fillrandom   :       2.775 micros/op 360334 ops/sec;  280.4 MB/s
  Microseconds per write:
  Count: 10000000 Average: 2.7749  StdDev: 39.92
  Min: 1  Median: 2.0826  Max: 26051
  Percentiles: P50: 2.08 P75: 2.55 P99: 3.55 P99.9: 9.58 P99.99: 51.5**6
  ------------------------------------------------------

  After:

  Initializing RocksDB Options from command-line flags
  DB path: [k:\data\BulkLoadRandom_10M_fillon
Closes https://github.com/facebook/rocksdb/pull/2011

Differential Revision: D4740823

Pulled By: siying

fbshipit-source-id: 1daaa2c
2017-03-22 11:24:12 -07:00
Siying Dong
e474df9470 db_bench: not need to check mmap for PlainTable
Summary:
PlainTable now supports non-mmap mode. We don't need to check it anymore.
Closes https://github.com/facebook/rocksdb/pull/1882

Differential Revision: D4751643

Pulled By: siying

fbshipit-source-id: ab14540
2017-03-22 11:09:13 -07:00
Maysam Yabandeh
8b0097b49b Readers for partition filter
Summary:
This is the last split of this pull request: https://github.com/facebook/rocksdb/pull/1891 which includes the reader part as well as the tests.
Closes https://github.com/facebook/rocksdb/pull/1961

Differential Revision: D4672216

Pulled By: maysamyabandeh

fbshipit-source-id: 6a2b829
2017-03-22 09:24:15 -07:00
Siying Dong
9ef3627fd3 Allow checkpointing without flushing
Summary:
Add a parameter to Checkpoint::CreateCheckpoint() so that flush can be skipped if total log file size is within a threshold.
Closes https://github.com/facebook/rocksdb/pull/1993

Differential Revision: D4719842

Pulled By: siying

fbshipit-source-id: 4f9d9e1
2017-03-21 18:09:13 -07:00
Siying Dong
17866ecc3a Allow Users to change customized ldb tools' header in help printing
Summary: Closes https://github.com/facebook/rocksdb/pull/2018

Differential Revision: D4748448

Pulled By: siying

fbshipit-source-id: a54c2f9
2017-03-21 17:39:12 -07:00
Leonidas Galanis
a2a883318b remove deleted option from benchmark.sh
Summary:
Removed max_grandparent_overlap_factor from benchmark.sh since it is not a valid option anymore.
Closes https://github.com/facebook/rocksdb/pull/2015

Differential Revision: D4748229

Pulled By: lgalanis

fbshipit-source-id: c3869ea
2017-03-21 12:54:13 -07:00
Aaron Gao
78cb195595 add checkpoint to ldb
Summary: Closes https://github.com/facebook/rocksdb/pull/2017

Differential Revision: D4747656

Pulled By: lightmark

fbshipit-source-id: c52f160
2017-03-21 11:54:11 -07:00
Raza Hussain
4b04addfce updated solution if "make format" command fails
Summary: Closes https://github.com/facebook/rocksdb/pull/2016

Differential Revision: D4747003

Pulled By: yiwu-arbug

fbshipit-source-id: c1e2c5a
2017-03-21 11:09:10 -07:00
Siying Dong
8f5bf04468 Flush triggered by DB write buffer size picks the oldest unflushed CF
Summary:
Previously, when DB write buffer size triggers, we always pick the CF with most data in its memtable to flush. This approach can minimize total flush happens. Change the behavior to always pick the oldest unflushed CF, which makes it the same behavior when max_total_wal_size hits. This approach will minimize size used by max_total_wal_size.
Closes https://github.com/facebook/rocksdb/pull/1987

Differential Revision: D4703214

Pulled By: siying

fbshipit-source-id: 9ff8b09
2017-03-21 11:09:10 -07:00
Raza Hussain
6908e24b56 dynamic setting of stats_dump_period_sec through SetDBOption()
Summary:
Resolved the following issue: https://github.com/facebook/rocksdb/issues/1930
Closes https://github.com/facebook/rocksdb/pull/2004

Differential Revision: D4736764

Pulled By: yiwu-arbug

fbshipit-source-id: 64fe0b7
2017-03-20 22:54:13 -07:00
Aaron Gao
93c68b642e change regression bash file with debug mode
Summary:
add debug mode for better debugging
refactor some regex
change micros/op to ops/sec
Closes https://github.com/facebook/rocksdb/pull/1999

Differential Revision: D4742806

Pulled By: lightmark

fbshipit-source-id: 0fe3ae6
2017-03-20 17:39:17 -07:00
Maysam Yabandeh
21d8c31796 remove LIKELY from public headers
Summary:
Fixes #2008
Closes https://github.com/facebook/rocksdb/pull/2009

Differential Revision: D4739885

Pulled By: maysamyabandeh

fbshipit-source-id: 36fba5d
2017-03-20 12:54:14 -07:00
Tamir Duberstein
36ad75778b INSTALL: document USE_SSE
Summary:
Fixes #1790.

adamretter siying
Closes https://github.com/facebook/rocksdb/pull/1996

Differential Revision: D4730449

Pulled By: IslamAbdelRahman

fbshipit-source-id: 19944fa
2017-03-17 11:54:11 -07:00
Aaron Gao
9272e12f19 avoid ftruncate twice in buffered io
Summary:
in buffered io, the filesize_ is the real size.
Closes https://github.com/facebook/rocksdb/pull/1991

Differential Revision: D4711433

Pulled By: lightmark

fbshipit-source-id: ad604b9
2017-03-17 11:39:13 -07:00
Islam AbdelRahman
d52f334cbd Break stalls when no bg work is happening
Summary:
Current stall will keep sleeping even if there is no Flush/Compactions to wait for, I changed the logic to break the stall if we are not flushing or compacting

db_bench command used
```
# fillrandom
# memtable size = 10MB
# value size = 1 MB
# num = 1000
# use /dev/shm
./db_bench --benchmarks="fillrandom,stats" --value_size=1048576 --write_buffer_size=10485760 --num=1000 --delayed_write_rate=XXXXX  --db="/dev/shm/new_stall" | grep "Cumulative stall"
```

```
Current results

# delayed_write_rate = 1000 Kb/sec
Cumulative stall: 00:00:9.031 H:M:S

# delayed_write_rate = 200 Kb/sec
Cumulative stall: 00:00:22.314 H:M:S

# delayed_write_rate = 100 Kb/sec
Cumulative stall: 00:00:42.784 H:M:S

# delayed_write_rate = 50 Kb/sec
Cumulative stall: 00:01:23.785 H:M:S

# delayed_write_rate = 25 Kb/sec
Cumulative stall: 00:02:45.702 H:M:S
```

```
New results

# delayed_write_rate = 1000 Kb/sec
Cumulative stall: 00:00:9.017 H:M:S

# delayed_write_rate = 200 Kb/sec
Cumulative stall: 00
Closes https://github.com/facebook/rocksdb/pull/1884

Differential Revision: D4585439

Pulled By: IslamAbdelRahman

fbshipit-source-id: aed2198
2017-03-16 18:24:17 -07:00
Andrew Kryczka
e66221add4 fix db_bench rate limiter callsites
Summary:
pass nullptr as stats object for db_bench-specific rate limiters since its stats are intended to capture background write activity only.
Closes https://github.com/facebook/rocksdb/pull/1997

Differential Revision: D4726806

Pulled By: ajkr

fbshipit-source-id: 8e4b225
2017-03-16 17:54:12 -07:00
Siying Dong
dbae438a0e Replace "DEPRECATED" comment to "not supported"
Summary:
"DEPRECATED" is ambiguous. Make it clear that those options not supported won't take effect.
Closes https://github.com/facebook/rocksdb/pull/1995

Differential Revision: D4724241

Pulled By: siying

fbshipit-source-id: 1e812b8
2017-03-16 13:54:11 -07:00
Islam AbdelRahman
995618a821 Support SstFileManager::SetDeleteRateBytesPerSecond()
Summary:
Update DeleteScheduler component to support changing delete rate in runtime by introducing
SstFileManager::SetDeleteRateBytesPerSecond()
Closes https://github.com/facebook/rocksdb/pull/1994

Differential Revision: D4719906

Pulled By: IslamAbdelRahman

fbshipit-source-id: e6b8d9e
2017-03-16 12:09:15 -07:00
Islam AbdelRahman
e19163688b Add macros to include file name and line number during Logging
Summary:
current logging
```
2017/03/14-14:20:30.393432 7fedde9f5700 (Original Log Time 2017/03/14-14:20:30.393414) [default] Level summary: base level 1 max bytes base 268435456 files[1 0 0 0 0 0 0] max score 0.25
2017/03/14-14:20:30.393438 7fedde9f5700 [JOB 2] Try to delete WAL files size 61417909, prev total WAL file size 73820858, number of live WAL files 2.
2017/03/14-14:20:30.393464 7fedde9f5700 [DEBUG] [JOB 2] Delete /dev/shm/old_logging//MANIFEST-000001 type=3 #1 -- OK
2017/03/14-14:20:30.393472 7fedde9f5700 [DEBUG] [JOB 2] Delete /dev/shm/old_logging//000003.log type=0 #3 -- OK
2017/03/14-14:20:31.427103 7fedd49f1700 [default] New memtable created with log file: #9. Immutable memtables: 0.
2017/03/14-14:20:31.427179 7fedde9f5700 [JOB 3] Syncing log #6
2017/03/14-14:20:31.427190 7fedde9f5700 (Original Log Time 2017/03/14-14:20:31.427170) Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
2017/03/14-14:20:31.
Closes https://github.com/facebook/rocksdb/pull/1990

Differential Revision: D4708695

Pulled By: IslamAbdelRahman

fbshipit-source-id: cb8968f
2017-03-15 19:39:12 -07:00
Aaron Gao
d525718a93 cleanup direct io flag in WritableFileWriter
Summary:
remove unnecessary field `direct_io_`, use `use_direct_io()` instead.
Closes https://github.com/facebook/rocksdb/pull/1992

Differential Revision: D4712195

Pulled By: lightmark

fbshipit-source-id: 57d34f9
2017-03-14 22:39:09 -07:00
slyrz
5fa927aa97 Add Xpress and ZSTD CompressionType values to C header
Summary:
This PR copies the missing CompressionType values
```cpp
enum CompressionType : unsigned char {
  ...
  kXpressCompression = 0x6,
  kZSTD = 0x7,
  ...
};
```
into the C header.
Closes https://github.com/facebook/rocksdb/pull/1989

Differential Revision: D4706265

Pulled By: siying

fbshipit-source-id: e65e62a
2017-03-14 12:09:21 -07:00
Maysam Yabandeh
11526252cc Pinnableslice (2nd attempt)
Summary:
PinnableSlice

    Summary:
    Currently the point lookup values are copied to a string provided by the
    user. This incures an extra memcpy cost. This patch allows doing point lookup
    via a PinnableSlice which pins the source memory location (instead of
    copying their content) and releases them after the content is consumed
    by the user. The old API of Get(string) is translated to the new API
    underneath.

    Here is the summary for improvements:

    value 100 byte: 1.8% regular, 1.2% merge values
    value 1k byte: 11.5% regular, 7.5% merge values
    value 10k byte: 26% regular, 29.9% merge values
    The improvement for merge could be more if we extend this approach to
    pin the merge output and delay the full merge operation until the user
    actually needs it. We have put that for future work.

    PS:
    Sometimes we observe a small decrease in performance when switching from
    t5452014 to this patch but with the old Get(string) API. The d
Closes https://github.com/facebook/rocksdb/pull/1756

Differential Revision: D4391738

Pulled By: maysamyabandeh

fbshipit-source-id: 6f3edd3
2017-03-13 11:54:10 -07:00
Aaron Gao
e5bd8def1e update history.md for fixing the bug that skips keys
Summary: Closes https://github.com/facebook/rocksdb/pull/1986

Differential Revision: D4699152

Pulled By: siying

fbshipit-source-id: b18c32c
2017-03-13 11:39:13 -07:00
Sagar Vemuri
1ffbdfd9a7 Add a new SstFileWriter constructor without explicit comparator
Summary:
The comparator param in SstFileWriter constructor is redundant as it already exists as a field in options. So the current SstFileWriter constructor should be deprecated in favor of a new one which does not take a comparator.
Note that the jni/java apis have not been touched yet.
Closes https://github.com/facebook/rocksdb/pull/1978

Differential Revision: D4685629

Pulled By: sagar0

fbshipit-source-id: 372ce96
2017-03-13 11:39:13 -07:00
Reid Horuff
ebd5639b6d Add ability to search for key prefix in sst_dump tool
Summary:
Add the flag --prefix to the sst_dump tool
This flag is similar to, and exclusive from, the --from flag.

--prefix=0x00FF will return all rows prefixed with 0x00FF.
The --to flag may also be specified and will work as expected.

These changes were used to help in debugging the power cycle corruption issue and theses changes were tested by scanning through a udb.
Closes https://github.com/facebook/rocksdb/pull/1984

Differential Revision: D4691814

Pulled By: reidHoruff

fbshipit-source-id: 027f261
2017-03-13 10:39:12 -07:00
Maysam Yabandeh
e6725e8c8d Fix some bugs in MockEnv
Summary:
Fixing some bugs in MockEnv so it be actually used.
Closes https://github.com/facebook/rocksdb/pull/1914

Differential Revision: D4609923

Pulled By: maysamyabandeh

fbshipit-source-id: ca25735
2017-03-13 09:54:11 -07:00
Min Wei
900c62be61 fix compile for VS2015
Summary:
Without the cast, the build will break on Windows.
Closes https://github.com/facebook/rocksdb/pull/1982

Differential Revision: D4690462

Pulled By: ajkr

fbshipit-source-id: c493b6c
2017-03-10 11:24:09 -08:00
Andrew Kryczka
fe1835617a release 5.3
Summary: Closes https://github.com/facebook/rocksdb/pull/1971

Differential Revision: D4683851

Pulled By: ajkr

fbshipit-source-id: 967116e
2017-03-09 12:39:10 -08:00
Maysam Yabandeh
5dae019477 Revert "Report cpu usage using time command"
Summary:
This reverts commit d43adf21bb.

The patch has caused problems in regression tests. Will revert it for now until we figure how to debug the problems regression tests.
Closes https://github.com/facebook/rocksdb/pull/1975

Differential Revision: D4682880

Pulled By: maysamyabandeh

fbshipit-source-id: 84df83a
2017-03-09 11:09:13 -08:00
Andrew Kryczka
f2817fb7f9 avoid ASSERT_EQ(false, ...);
Summary:
lately it fails on travis due to a compiler bug (see https://github.com/google/googletest/issues/322#issuecomment-125645145). interestingly it seems to affect occurrences of `ASSERT_EQ(false, ...);` but not `ASSERT_EQ(true, ...);`.
Closes https://github.com/facebook/rocksdb/pull/1958

Differential Revision: D4680742

Pulled By: ajkr

fbshipit-source-id: 291fe41
2017-03-08 22:24:16 -08:00
Andrew Kryczka
5b11124e39 add max to histogram stats
Summary:
Domas enlightened me about p100 (i.e., max) stats. Let's add them to our histograms.
Closes https://github.com/facebook/rocksdb/pull/1968

Differential Revision: D4678716

Pulled By: ajkr

fbshipit-source-id: 65e7118
2017-03-08 22:24:15 -08:00
Maysam Yabandeh
d43adf21bb Report cpu usage using time command
Summary:
It augments the regression benchmarks with a time command, parses the output, and print them to the SUMMARY.csv file.

I tested a variation of the script locally. Any idea how to do run a test that also involves writing to scuba tables?
Closes https://github.com/facebook/rocksdb/pull/1967

Differential Revision: D4679470

Pulled By: maysamyabandeh

fbshipit-source-id: 44dac30
2017-03-08 17:54:11 -08:00
Andrew Kryczka
18fc1bc0e0 minor changes for rate limiter test flakiness
Summary:
the 50%+ drained constraint wasn't working consistently in some of our test environments, maybe their resources are too low. relax the constraints a bit.
Closes https://github.com/facebook/rocksdb/pull/1970

Differential Revision: D4679419

Pulled By: ajkr

fbshipit-source-id: 3789cd8
2017-03-08 17:54:11 -08:00
Aaron Gao
12ba00ea65 Reset DBIter::saved_key_ with proper user key anywhere before pass to DBIter::FindNextUserEntry
Summary:
fix db_iter bug introduced by [facebook#1413](https://github.com/facebook/rocksdb/pull/1413)
Closes https://github.com/facebook/rocksdb/pull/1962

Differential Revision: D4672369

Pulled By: lightmark

fbshipit-source-id: 6a22953
2017-03-08 17:24:11 -08:00
Dmitri Smirnov
c9df05d1e4 Fix random access alignment
Summary:
This fixes an issue when the most recent readers assume that alignment is always set even if direct io is off.
Also adjust slightly appveyor script to run db_basic_test cases concurrently.
Closes https://github.com/facebook/rocksdb/pull/1959

Differential Revision: D4671972

Pulled By: IslamAbdelRahman

fbshipit-source-id: 1886620
2017-03-08 17:09:11 -08:00
Islam AbdelRahman
f64991537e Add Bulkoading IngestExternalFile blog post
Summary:
new blog post for bulkoading
Closes https://github.com/facebook/rocksdb/pull/1883

Differential Revision: D4671984

Pulled By: IslamAbdelRahman

fbshipit-source-id: 3450860
2017-03-07 18:24:10 -08:00
Maysam Yabandeh
54b434110e Builders for partition filter
Summary:
This is the second split of this pull request: https://github.com/facebook/rocksdb/pull/1891 which includes only the builder part. The testing will be included in the third split, where the reader is also included.
Closes https://github.com/facebook/rocksdb/pull/1952

Differential Revision: D4660272

Pulled By: maysamyabandeh

fbshipit-source-id: 36b3cf0
2017-03-07 13:54:12 -08:00
Sagar Vemuri
97edc72d39 Add a memtable-only iterator
Summary:
This PR is to support a way to iterate over all the keys that are just in memtables.
Closes https://github.com/facebook/rocksdb/pull/1953

Differential Revision: D4663500

Pulled By: sagar0

fbshipit-source-id: 144e177
2017-03-07 11:54:10 -08:00
Leonidas Galanis
72202962f9 fix db_sst_test flakiness
Summary:
db_sst_test had been flaky occasionally in the following way: reached_max_space_on_compaction can in very rare cases be 0. This happens when the limit on maximum allowable space set using SetMaxAllowedSpaceUsage is hit during flush for all test db sizes (1,2,4,8 and 10MB).The fix clears the error returned when the the space limit is reached during flush. This ensures that the compaction call back will always be called. The runtime is increased slightly because the 1MB loop writes more data and hits the limit during multiple flushes until compaction is scheduled.
Closes https://github.com/facebook/rocksdb/pull/1861

Differential Revision: D4557396

Pulled By: lgalanis

fbshipit-source-id: ff778d1
2017-03-07 11:24:13 -08:00