Commit Graph

5033 Commits

Author SHA1 Message Date
Hyunyoung Lee
71303e04e7 Update db_bench_tool.cc (#1073)
* Update db_bench_tool.cc

I fixed the wrong letters, LevelDB -> rocksDB, because I thought of LevelDB as the wrong presentation.

the following show my fix :

fprintf(stderr, "LevelDB:    version %d.%d\n",
            kMajorVersion, kMinorVersion);

----------------->
fprintf(stderr, "rocksDB:    version %d.%d\n",
            kMajorVersion, kMinorVersion);

* Update db_bench_tool.cc

* Update db_bench_tool.cc
2016-04-12 17:05:09 -04:00
sdong
63cf15bb9f Fix option settable tests
Summary: In option settable tests, bytes for pointers are not all skipped, so that they may be the same as the special character and cause false positive.

Test Plan: Run the test. Manually verify the issue is not there any more.

Reviewers: IslamAbdelRahman, andrewkr

Reviewed By: IslamAbdelRahman

Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56553
2016-04-12 14:03:35 -07:00
Shu Zhang
e208575b2d using java7 in runtime for hdfs env (#1072) 2016-04-12 00:08:26 -04:00
Yueh-Hsuan Chiang
13e6c8e97a Relax an assertion in Compaction::ShouldStopBefore
Summary:
In some case, it is possible to have two concesutive SST files might sharing
same boundary keys.  However, in the assertion in Compaction::ShouldStopBefore,
it exclude such possibility.

This patch fix this issue by relaxing the assertion to allow the equal case.

Test Plan: rocksdb tests

Reviewers: IslamAbdelRahman, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D55875
2016-04-11 20:15:52 -07:00
Yueh-Hsuan Chiang
ae21d71e94 Fixed a bug in RocksDB Statistics where flush is considered as compaction
Summary: Fixed a bug in RocksDB Statistics where flush is considered as compaction

Test Plan: unit test

Reviewers: sdong, IslamAbdelRahman, rven, kradhakrishnan, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56271
2016-04-11 19:59:25 -07:00
Andrew Kryczka
8e0e22f76b Fix Windows build by replacing strings.h include
Summary:
strings.h header does not exist on Windows. So, we can try another way
to compare strings ignoring case.

Test Plan:
built and ran:

  $ ./ldb_cmd_test

Reviewers: sdong, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56535
2016-04-11 19:21:00 -07:00
Islam AbdelRahman
5675d5037f Revert travis commit
Summary: Revert travis commit pushed unintentionally

Test Plan: none

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56559
2016-04-11 16:46:55 -07:00
Islam AbdelRahman
91f0f1f5ef fix travis 2016-04-11 16:37:22 -07:00
sdong
a23c6052c8 Don't run DBOptionsAllFieldsSettable under valgrind
Summary: Test DBOptionsAllFieldsSettable sometimes fails under valgrind. Move option settable tests to a separate test file and disable it in valgrind..

Test Plan: Run valgrind test and make sure the test doesn't run.

Reviewers: andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56529
2016-04-11 14:24:55 -07:00
sdong
30d72ee43c PrefixTest.PrefixAndWholeKeyTest should run against a different directory from prefix_test
Summary: PrefixTest.PrefixAndWholeKeyTest runs against the same directory as prefix_test, which sometimes fail parallel tests. Fix it.

Test Plan: Run it in parallel and see it doesn't fail anymore.

Reviewers: andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56541
2016-04-11 13:02:56 -07:00
Andrew Kryczka
0e3cc2cf1f Add column family info to TableProperties::ToString()
Summary: This is used at least by `sst_dump --show_properties`

Test Plan:
- default CF

```
  $ ./sst_dump --show_properties --file=./tmp-db/000007.sst | grep 'column family'
  column family ID: 0
  column family name: default
```

- custom CF

```
  $ ./sst_dump --show_properties --file=./tmp-db/000012.sst | grep 'column family'
  column family ID: 1
  column family name: col-fam-1
```

- no CF

```
  $ ./sst_dump --show_properties --file=./tmp-db/000017.sst | grep 'column family'
  column family ID: N/A
  column family name: N/A
```

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56499
2016-04-08 18:50:18 -07:00
Jay Edgar
2448f80375 Make sure that if use_mmap_reads is on use_os_buffer is also on
Summary: The code assumes that if use_mmap_reads is on then use_os_buffer is also on.  This make sense as by using memory mapped files for reading you are expecting the OS to cache what it needs.  Add code to make sure the user does not turn off use_os_buffer when they turn on use_mmap_reads

Test Plan: New test: DBTest.MMapAndBufferOptions

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56397
2016-04-08 14:30:15 -07:00
Andrew Kryczka
114a1b8792 Fix build errors for windows
Summary:
- Need to use unsigned long long for 64-bit literals on windows
- Need size_t for backup meta-file length since clang doesn't let us assign size_t to int

Test Plan: backupable_db_test and options_test

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56391
2016-04-08 13:09:19 -07:00
Islam AbdelRahman
0522990358 Improve sst_dump help message
Summary:
Current Message

```
sst_dump [--command=check|scan|none|raw] [--verify_checksum] --file=data_dir_OR_sst_file [--output_hex] [--input_key_hex] [--from=<user_key>] [--to=<user_key>] [--read_num=NUM] [--show_properties] [--show_compression_sizes] [--show_compression_sizes [--set_block_size=<block_size>]]
```
New message

```
sst_dump --file=<data_dir_OR_sst_file> [--command=check|scan|raw]
    --file=<data_dir_OR_sst_file>
      Path to SST file or directory containing SST files

    --command=check|scan|raw
        check: Iterate over entries in files but dont print anything except if an error is encounterd (default command)
        scan: Iterate over entries in files and print them to screen
        raw: Dump all the table contents to <file_name>_dump.txt

    --output_hex
      Can be combined with scan command to print the keys and values in Hex

    --from=<user_key>
      Key to start reading from when executing check|scan

    --to=<user_key>
      Key to stop reading at when executing check|scan

    --read_num=<num>
      Maximum number of entries to read when executing check|scan

    --verify_checksum
      Verify file checksum when executing check|scan

    --input_key_hex
      Can be combined with --from and --to to indicate that these values are encoded in Hex

    --show_properties
      Print table properties after iterating over the file

    --show_compression_sizes
      Independent command that will recreate the SST file using 16K block size with different
      compressions and report the size of the file using such compression

    --set_block_size=<block_size>
      Can be combined with --show_compression_sizes to set the block size that will be used
      when trying different compression algorithms
```

Test Plan: none

Reviewers: yhchiang, andrewkr, kradhakrishnan, yiwu, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56325
2016-04-08 12:05:02 -07:00
sdong
0930e5e99f Update comments on include/rocksdb/perf_context.h
Summary: Some grammer mistakes in code comments in include/rocksdb/perf_context.h. Also polish it a liitlebit.

Test Plan: Not needed

Reviewers: IslamAbdelRahman, yhchiang, yiwu, andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56307
2016-04-08 11:27:08 -07:00
Shu Zhang
3b977bcdc8 instructing people to use java7 for hdfs (#1063) 2016-04-07 14:34:28 -07:00
sdong
1518b733eb Change default number of cache shard bit to be 6 and max_file_opening_threads to be 16.
Summary: Cache shard bit 4 is sometimes too small and 6 is a more common value picked by users. Make that default. It shouldn't hurt much to change options.max_file_opening_threads default to be 16, which will reduce the worst case DB open time.

Test Plan: Run all existing tests.

Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, igor, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, MarkCallaghan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D55047
2016-04-07 13:55:10 -07:00
flabby
ada88b63f5 fix wrong assignment of level0_stop_writes_trigger in spatialdb (#1061) 2016-04-07 09:02:28 -07:00
Andrew Kryczka
2391ef7214 Embed column family name in SST file
Summary:
Added the column family name to the properties block. This property
is omitted only if the property is unavailable, such as when RepairDB()
writes SST files.

In a next diff, I will change RepairDB to use this new property for
deciding to which column family an existing SST file belongs. If this
property is missing, it will add it to the "unknown" column family (same
as its existing behavior).

Test Plan:
New unit test:

  $ ./db_table_properties_test --gtest_filter=DBTablePropertiesTest.GetColumnFamilyNameProperty

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D55605
2016-04-06 23:10:32 -07:00
Igor Canadi
ab4c62332e Don't use version in the error message
Summary: We use object `v` in the error message, which is not initialized if the edit is column family manipulation. This doesn't provide much useful info, so this diff is removing it. Instead, it dumps actual VersionEdit contents.

Test Plan: compiles. would be great to get tests in version_set_test.cc that cover cases where a file write fails

Reviewers: sdong, yhchiang, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56349
2016-04-06 15:00:15 -07:00
Islam AbdelRahman
d02eb8d00b Fix unused variable warning
Summary:
My last diff introduced a warning when compiling under release mode
https://reviews.facebook.net/D55539

fix the warning

Test Plan:
DEBUG_LEVEL=0 make db_bench
make check

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56295
2016-04-04 11:10:46 -07:00
Igor Canadi
9278097f86 Merge pull request #1056 from facebook/igorcanadi-patch-1
Create LANGUAGE-BINDINGS.md
2016-04-04 08:08:52 -07:00
Aaron Gao
cc87075d63 No need to limit to 20 files in UpdateAccumulatedStats() if options.max_open_files=-1
Summary:
There is a hardcoded constraint in our statistics collection that prevents reading properties from more than 20 SST files. This means our statistics will be very inaccurate for databases with > 20 files since additional files are just ignored. The purpose of constraining the number of files used is to bound the I/O performed during statistics collection, since these statistics need to be recomputed every time the database reopened.

However, this constraint doesn't take into account the case where option "max_open_files" is -1. In that case, all the file metadata has already been read, so MaybeInitializeFileMetaData() won't incur any I/O cost. so this diff gets rid of the 20-file constraint in case max_open_files == -1.

Test Plan:
write into unit test db/db_properties_test.cc - "ValidateSampleNumber".
We generate 20 files with 2 rows and 10 files with 1 row.
If max_open_files !=-1, the `rocksdb.estimate-num-keys` should be (10*1 + 10*2)/20 * 30 = 45. Otherwise, it should be the ground truth, 50.
{F1089153}

Reviewers: andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56253
2016-04-01 16:19:12 -07:00
Islam AbdelRahman
8a1a603fdb Eliminate std::deque initialization while iterating over merge operands
Summary:
This patch is similar to D52563, When we iterate over a DB with merge operands we keep creating std::queue to store the operands, optimize this by reusing merge_operands_ data member

Before the patch

```
./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq" --db="/dev/shm/bench_merge_memcpy_on_the_fly/" --merge_operator="put" --merge_keys=10000 --num=10000

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       3.757 micros/op 266141 ops/sec;   29.4 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.413 micros/op 2423538 ops/sec;  268.1 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.451 micros/op 2219071 ops/sec;  245.5 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.420 micros/op 2382039 ops/sec;  263.5 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.408 micros/op 2452017 ops/sec;  271.3 MB/s

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       3.947 micros/op 253376 ops/sec;   28.0 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.441 micros/op 2266473 ops/sec;  250.7 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.471 micros/op 2122033 ops/sec;  234.8 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.440 micros/op 2271407 ops/sec;  251.3 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.429 micros/op 2331471 ops/sec;  257.9 MB/s
```

with the patch

```
./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq" --db="/dev/shm/bench_merge_memcpy_on_the_fly/" --merge_operator="put" --merge_keys=10000 --num=10000

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       4.080 micros/op 245092 ops/sec;   27.1 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.308 micros/op 3241843 ops/sec;  358.6 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.312 micros/op 3200408 ops/sec;  354.0 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.332 micros/op 3013962 ops/sec;  333.4 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.300 micros/op 3328017 ops/sec;  368.2 MB/s

DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
mergerandom  :       3.973 micros/op 251705 ops/sec;   27.8 MB/s ( updates:10000)
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.320 micros/op 3123752 ops/sec;  345.6 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.335 micros/op 2986641 ops/sec;  330.4 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.339 micros/op 2950047 ops/sec;  326.4 MB/s
DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/]
readseq      :       0.319 micros/op 3131565 ops/sec;  346.4 MB/s
```

Test Plan: make check -j64

Reviewers: yhchiang, andrewkr, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56031
2016-04-01 15:48:55 -07:00
Islam AbdelRahman
f38540b12a WriteBatchWithIndex micro optimization
Summary:
  - Put key offset and key size in WriteBatchIndexEntry
  - Use vector for comparators in WriteBatchEntryComparator

I use a slightly modified version of @yoshinorim code to benchmark
https://gist.github.com/IslamAbdelRahman/b120f4fba8d6ff7d58d2

For Put I create a transaction that put a 1000000 keys and measure the time spent without commit.
For GetForUpdate I read the keys that I added in the Put transaction.

Original time:

```
 rm -rf /dev/shm/rocksdb-example/
 ./txn_bench put 1000000
 1000000 OK Ops | took      3.679 seconds
 ./txn_bench get_for_update 1000000
 1000000 OK Ops | took      3.940 seconds
```

New Time

```
  rm -rf /dev/shm/rocksdb-example/
 ./txn_bench put 1000000
 1000000 OK Ops | took      2.727 seconds
 ./txn_bench get_for_update 1000000
 1000000 OK Ops | took      3.880 seconds
```

It looks like there is no significant improvement in GetForUpdate() but we can see ~30% improvement in Put()

Test Plan: unittests

Reviewers: yhchiang, anthony, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, yoshinorim

Differential Revision: https://reviews.facebook.net/D55539
2016-04-01 15:23:46 -07:00
Adam Retter
200654067a Merge pull request #1053 from adamretter/benchmark-java-comparator
Benchmark Java comparator vs C++ comparator
2016-04-01 13:53:15 -07:00
Andrew Kryczka
f2c43a4a27 Stderr info logger
Summary:
Adapted a stderr logger from the option tests. Moved it to a separate
header so we can reuse it, e.g., from ldb subcommands for faster debugging. This
is especially useful to make errors/warnings more visible when running
"ldb repair", which involves potential data loss.

Test Plan:
ran options_test and "ldb repair"

  $ ./ldb repair --db=./tmp/
  [WARN] **** Repaired rocksdb ./tmp/; recovered 1 files; 588bytes. Some data may have been lost. ****
  OK

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56151
2016-04-01 11:06:06 -07:00
Uddipta Maity
b55e2165be Rocksdb backup can store optional application specific metadata
Summary:
Rocksdb backup engine maintains metadata about backups in separate files. But,
there was no way to add extra application specific data to it. Adding support
for that.
In some use cases, applications decide to restore a backup based on some
metadata. This will help those cases to cheaply decide whether to restore or
not.

Test Plan:
Added a unit test. Existing ones are passing

Sample meta file for BinaryMetadata test-

```

1459454043
0
metadata 6162630A64656600676869
2
private/1/MANIFEST-000001 crc32 1184723444
private/1/CURRENT crc32 3505765120

```

Reviewers: sdong, ldemailly, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, ldemailly

Differential Revision: https://reviews.facebook.net/D56007
2016-04-01 10:56:52 -07:00
Marton Trencseni
9b51987521 Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.
Summary:
When a block based table file is opened, if prefetch_index_and_filter is true, it will prefetch the index and filter blocks, putting them into the block cache.
What this feature adds: when a L0 block based table file is opened, if pin_l0_filter_and_index_blocks_in_cache is true in the options (and prefetch_index_and_filter is true), then the filter and index blocks aren't released back to the block cache at the end of BlockBasedTableReader::Open(). Instead the table reader takes ownership of them, hence pinning them, ie. the LRU cache will never push them out. Meanwhile in the table reader, further accesses will not hit the block cache, thus avoiding lock contention.

Test Plan:
'export TEST_TMPDIR=/dev/shm/ && DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32' is OK.
I didn't run the Java tests, I don't have Java set up on my devserver.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56133
2016-04-01 10:42:39 -07:00
sdong
2feafa3db9 Change some RocksDB default options
Summary: Change some RocksDB default options to make it more friendly to server workloads.

Test Plan: Run all existing tests

Reviewers: yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: sumeet, muthu, benj, MarkCallaghan, igor, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D55941
2016-03-31 17:12:18 -07:00
Yueh-Hsuan Chiang
a558830f8f Fixed compile warnings in posix_logger.h and coding.h
Summary:
Fixed the following compile warnings:

/Users/yhchiang/rocksdb/util/posix_logger.h:32:11: error: unused variable 'kDebugLogChunkSize' [-Werror,-Wunused-const-variable]
const int kDebugLogChunkSize = 128 * 1024;
          ^
/Users/yhchiang/rocksdb/util/coding.h:24:20: error: unused variable 'kMaxVarint32Length' [-Werror,-Wunused-const-variable]
const unsigned int kMaxVarint32Length = 5;
                   ^
2 errors generated.

Test Plan: make clean rocksdb

Reviewers: igor, sdong, anthony, IslamAbdelRahman, rven, kradhakrishnan, adamretter

Reviewed By: adamretter

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56223
2016-03-31 16:01:47 -07:00
Yueh-Hsuan Chiang
51c9464dfc Merge pull request #980 from adamretter/java-arm
ARM for the Java API
2016-03-31 13:56:45 -07:00
Igor Canadi
925b5d0025 Merge pull request #1054 from DCEngines/magic12
Remove the Magic number 12 used in record size checks
2016-03-30 21:38:19 -07:00
Sandeep Joshi
63e8f1b55b Formatted lines to adhere to 80 char limit 2016-03-31 08:26:55 +05:30
Gunnar Kudrjavets
994b3bd693 Add support for UBsan builds to RocksDB
Summary:
Undefined Behavior Sanitizer (ubsan) is //a good thing// which will help us to find sneaky bugs with low cost. Please see http://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ for more details and official GCC documentation for more context: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html.

Changes itself are quite simple and pretty much imitating whatever is implemented for ASan.

Hooking the UBsan validation build to Sandcastle is a separate step and will be dealt as separate diff because code is in internal repository.

Test Plan: Make sure that that there no regressions when it comes to builds and test pass rate.

Reviewers: leveldb, sdong, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56049
2016-03-30 15:59:24 -07:00
Laurent Demailly
21700a5106 to/from hex refactor
Summary:
Expose the inverse of ToString(hex=true) on Slice: Slice::DecodeHex
Refactor the other implementation of to/from hex in ldb_cmd.h to use the Slice
version
(Difference between the 2 is whether 0x is expected/produced in front of the hex
string or not)
Eliminated support for invalid odd length hex string - this is now invalid
instead of having 1/2 byte set
Added (inverse of HexToString) test for LDBCommand::StringToHex which also
indirectly tests Slice::ToString(true)

After moving the original implementation from ldb_cmd.h, updated it to much simpler/efficient version
(originally/inspired from https://github.com/facebook/wdt/blob/master/util/EncryptionUtils.cpp#L140-L169 )

Test Plan: run tests

Reviewers: uddipta, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56121
2016-03-30 14:36:48 -07:00
Sandeep Joshi
24420947d9 Replace kHeader by WriteBatchInternal::kHeader in few more places
kHeader was moved from write_batch.cc to header file because
it is being used wherever the number "12" was being used to
check for record size
2016-03-30 23:13:00 +05:30
Sandeep Joshi
3bdbe89614 Merge branch 'magic12' of https://github.com/dcengines/rocksdb into magic12 2016-03-30 23:09:39 +05:30
zensan
78711524b7 In all the places where log records are read, there was a check that
record.size() should not be less than 12.

This "magic number" seems to be the WriteBatch header (8 byte sequence
and 4 byte count).   Replaced all the places where "12" was used
by WriteBatchInternal::kHeader.
2016-03-30 23:05:22 +05:30
Islam AbdelRahman
e7c64fb115 Imporve sst_file_manager comment
Summary: Improve the comment for sst_file_manager

Test Plan: none

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56001
2016-03-29 13:20:26 -07:00
Islam AbdelRahman
99ffb3d533 Fix perf_context::merge_operator_time_nanos calculation
Summary: We were not measuring the time spent in merge_operator when called from Version::Get()

Test Plan: added a unittest

Reviewers: sdong, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D55905
2016-03-25 18:29:43 -07:00
sdong
07bb12d97d Update internal jemalloc and other versions
Summary: Update jemalloc to newer versions, as well as some other dependencies. Only effective for FB internal.

Test Plan: See all tests run

Reviewers: IslamAbdelRahman, MarkCallaghan

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D55947
2016-03-25 10:42:20 -07:00
Yueh-Hsuan Chiang
ad2fdaa823 Correct a typo in a comment
Summary: Correct a typo in a comment

Test Plan: No code change.

Reviewers: sdong, kradhakrishnan, IslamAbdelRahman

Reviewed By: kradhakrishnan, IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D55803
2016-03-24 19:39:13 -07:00
Yueh-Hsuan Chiang
be9816b3d9 Fix data race issue when sub-compaction is used in CompactionJob
Summary:
When subcompaction is used, all subcompactions share the same Compaction
pointer in CompactionJob while each subcompaction all keeps their mutable
stats in SubcompactionState.  However, there're still some mutable part
that is currently store in the shared Compaction pointer.

This patch makes two changes:

1. Make the shared Compaction pointer const so that it can never be modified
   during the compaction.
2. Move necessary states from Compaction to SubcompactionState.
3. Make functions of Compaction const if the function does not modify
   its internal state.

Test Plan: rocksdb and MyRocks test

Reviewers: sdong, kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, yoshinorim, gunnarku, leveldb

Differential Revision: https://reviews.facebook.net/D55923
2016-03-24 19:36:39 -07:00
Islam AbdelRahman
e3802531f1 Merge pull request #1050 from yuslepukhin/support_db_test2
Add support for db_test2 for dev and CI runs
2016-03-23 17:59:30 -07:00
Dmitri Smirnov
e7cc49cbdf Add support for db_test2 for dev and CI runs 2016-03-23 16:33:57 -07:00
sdong
3996770d0b Add comments to perf_context skip counters
Summary: Document the skipped counters in perf context more clearly.

Test Plan: Comment only.

Reviewers: IslamAbdelRahman, yhchiang, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D55833
2016-03-23 14:26:25 -07:00
Mike Kolupaev
4e85b74790 Make WritableFileWrapper not screw up preallocation
Summary:
Without this diff, this is what happens to compaction output file if it's a subclass of WritableFileWrapper:
- during compaction, all `PrepareWrite()` calls update `last_preallocated_block_` of the `WritableFileWrapper` itself, not of `target_`, since `PrepareWrite()` is not virtual,
- `PrepareWrite()` calls `Allocate()`, which is virtual; it does `fallocate()` on `target_`,
- after writing data, `target_->Close()` calls `GetPreallocationStatus()` of `target_`; it returns `last_preallocated_block_` of `target_`, which is zero because it was never touched before,
- `target_->Close()` doesn't call `ftruncate()`; file remains big.

This diff fixes it in a straightforward way, by making the methods virtual. `WritableFileWrapper` ends up having the useless fields `last_preallocated_block_` and `preallocation_block_size_`. I think ideally the preallocation logic should be outside `WritableFile`, the same way as `log_writer.h` and `file_reader_writer.h` moved some non-platform-specific logic out of Env, but that's probably not worth the effort now.

Test Plan: `make -j check`; I'm going to deploy it on our test tier and see if it fixes space reclamation problem there

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, sdong

Differential Revision: https://reviews.facebook.net/D54681
2016-03-23 09:14:56 -07:00
Siying Dong
ec458dcdeb Merge pull request #1047 from PraveenSinghRao/wal_filter_ex
Avoid overloaded virtual function
2016-03-22 21:48:39 -07:00
Praveen Rao
583157f710 Avoid overloaded virtual function 2016-03-22 17:10:31 -07:00