Commit Graph

3352 Commits

Author SHA1 Message Date
Herman Lee
51c8133a72 Fix make unity build compiler warning about "stats" shadowing global variable
Summary:
Fix the make unity build. The local stats variable name was shadowing a
global stats variable.

Test Plan:
Run the build
OPT=-DTRAVIS V=1 make unity

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D36285
2015-04-01 10:48:42 -07:00
Igor Canadi
df71c6b9ed Script to trigger jenkins test
Summary: After you run `arc diff`, just run `build_tools/trigger_jenkins_test.sh` and Jenkins will test your diff!

Test Plan: Triggered a build to jenkins

Reviewers: sdong, rven, IslamAbdelRahman, anthony, yhchiang, meyering

Reviewed By: meyering

Subscribers: meyering, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36021
2015-03-31 08:56:40 -07:00
Tian Xia
38a01ed1b0 Update COMMIT.md 2015-03-30 17:48:16 -07:00
sdong
76d63b4525 Fix one non-determinism of DBTest.DynamicCompactionOptions
Summary:
After recent change of DBTest.DynamicCompactionOptions, occasionally hit another non-deterministic case where L0 showdown is triggered while timeout should not triggered for hard limit.
Fix it by increasing L0 slowdown trigger at the same time.

Test Plan: Run the failed test.

Reviewers: igor, rven

Reviewed By: rven

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D36219
2015-03-30 15:53:44 -07:00
sdong
b23bbaa82a Universal Compactions with Small Files
Summary:
With this change, we use L1 and up to store compaction outputs in universal compaction.
The compaction pick logic stays the same. Outputs are stored in the largest "level" as possible.

If options.num_levels=1, it behaves all the same as now.

Test Plan:
1) convert most of existing unit tests for universal comapaction to include the option of one level and multiple levels.
2) add a unit test to cover parallel compaction in universal compaction and run it in one level and multiple levels
3) add unit test to migrate from multiple level setting back to one level setting
4) add a unit test to insert keys to trigger multiple rounds of compactions and verify results.

Reviewers: rven, kradhakrishnan, yhchiang, igor

Reviewed By: igor

Subscribers: meyering, leveldb, MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D34539
2015-03-30 15:12:02 -07:00
Igor Canadi
2511b7d947 Makefile minor cleanup
Summary:
Just couple of small changes:
1. removed signal_test, since it doesn't seem useful and we don't even run it as part of `make check`
2. moved perf_context_test to TESTS instead of PROGRAMS
3. `make release` probably shouldn't compile benchmarks. We currently rely on `make release` building db_bench (via Jenkins), so I left db_bench there.

This is just a minor cleanup. We need to rethink our targets since they are a bit messy right now. We can do this during our tech debt week.

Test Plan: make release

Reviewers: anthony, rven, yhchiang, sdong, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36171
2015-03-30 16:05:35 -04:00
Mark Callaghan
1bd70fb54a Add --stats_interval_seconds to db_bench
Summary:
The --stats_interval_seconds determines interval for stats reporting
and overrides --stats_interval when set. I also changed tools/benchmark.sh
to report stats every 60 seconds so I can avoid trying to figure out a
good value for --stats_interval per test and per storage device.

Task ID: #6631621

Blame Rev:

Test Plan:
run tools/run_flash_bench, look at output

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D36189
2015-03-30 12:58:32 -07:00
Igor Canadi
fd3dbef22b Clean up old log files in background threads
Summary:
Cleaning up log files can do heavy IO, since we call ftruncate() in the destructor. We don't want to call ftruncate() in user threads.

This diff moves cleaning to background threads (flush and compaction)

Test Plan: make check, will also run valgrind

Reviewers: yhchiang, rven, MarkCallaghan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36177
2015-03-30 15:04:10 -04:00
Mark Callaghan
99ec2412e5 Make the benchmark scripts configurable and add tests
Summary:
This makes run_flash_bench.sh configurable. Previously it was hardwired for 1B keys and tests
ran for 12 hours each. That kept me from using it. This makes it configuable, adds more tests,
makes the duration per-test configurable and refactors the test scripts.

Adds the seekrandomwhilemerging test to db_bench which is the same as seekrandomwhilewriting except
the writer thread does Merge rather than Put.

Forces the stall-time column in compaction IO stats to use a fixed format (H:M:S) which makes
it easier to scrape and parse. Also adds an option to AppendHumanMicros to force a fixed format.
Sometimes automation and humans want different format.

Calls thread->stats.AddBytes(bytes); in db_bench for more tests to get the MB/sec summary
stats in the output at test end.

Adds the average ingest rate to compaction IO stats. Output now looks like:
https://gist.github.com/mdcallag/2bd64d18be1b93adc494

More information on the benchmark output is at https://gist.github.com/mdcallag/db43a58bd5ac624f01e1

For benchmark.sh changes default RocksDB configuration to reduce stalls:
* min_level_to_compress from 2 to 3
* hard_rate_limit from 2 to 3
* max_grandparent_overlap_factor and max_bytes_for_level_multiplier from 10 to 8
* L0 file count triggers from 4,8,12 to 4,12,20 for (start,stall,stop)

Task ID: #6596829

Blame Rev:

Test Plan:
run tools/run_flash_bench.sh

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D36075
2015-03-30 11:28:25 -07:00
Igor Canadi
2158e0f831 Fix clang build
Summary: as title

Test Plan: clang builds

Reviewers: leveldb

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36183
2015-03-30 10:03:23 -07:00
Igor Canadi
d61cb0b9de db_bench can now disable flashcache for background threads
Summary: Most of the approach is copied from WebSQL's MySQL branch. It's nice that we can do this without touching core RocksDB code.

Test Plan: Compiles and runs. Didn't test flashback code, as I don't have flashback device and most if it is c/p

Reviewers: MarkCallaghan, sdong

Reviewed By: sdong

Subscribers: rven, lgalanis, kradhakrishnan, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35391
2015-03-30 09:51:11 -07:00
Jim Meyering
1c47c433ba build: always attempt to update util/build_version.cc
Summary:
This fixes two bugs: "make clean" would never remove the generated
file, util/build_version.cc, and since D33591, would be regenerated
only if it were absent.
* Makefile (clean): Remove the generated file.
(util/build_version.cc): Depend on the no-prereq FORCE target,
so that this target's rules are always run.
Since this is a generated file, make it read-only.
Also, be sure to remove the temporary file when it is the same
as the original.

Test Plan:
Ensure that we attempt regeneration every time.
Make it empty with an up-to-date time stamp and demonstrate
that it is rebuilt with the expected content:

  $ : > util/build_version.cc
  $ make util/build_version.o
   GEN      util/build_version.cc
   GEN      util/build_version.d
   GEN      util/build_version.cc
   CC       util/build_version.o
  $ cat util/build_version.cc
  #include "build_version.h"
  const char* rocksdb_build_git_sha = "rocksdb_build_git_sha:v3.10-2-gb30e72a";
  const char* rocksdb_build_git_date = "rocksdb_build_git_date:2015-03-27";
  const char* rocksdb_build_compile_date = __DATE__;

Reviewers: igor.sugak, sdong, ljin, igor, rven

Reviewed By: rven

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D36087
2015-03-27 19:21:50 -07:00
Herman Lee
e018892bb6 Formalize the DB properties string definitions.
Summary:
Assign the string properties to const string variables under the
DB::Properties namespace. This helps catch typos during compilation and
also consolidates the property definition in one place.

Test Plan: Run rocksdb unit tests

Reviewers: sdong, yoshinorim, igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35991
2015-03-27 14:50:20 -07:00
Yueh-Hsuan Chiang
cfa576402c Make auto_sanity_test always use the db_sanity_test.cc of the newer commit.
Summary:
Whenever we add new tests in db_sanity_test.cc, the verification test
will fail since the old version db_sanity_test.cc does not have the
newly added test.  This patch makes auto_sanity_test.sh always use
the db_sanity_test.cc of the newer commit.

As a result, a macro guard is added to allow db_sanity_test.cc to be
backward compatible.

Test Plan: tools/auto_sanity_check.sh

Reviewers: sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35997
2015-03-27 11:32:49 -07:00
Igor Canadi
e9fddb7a2b Merge pull request #560 from xiaoxichen/patch-1
Fix interger overflow on i386 arch
2015-03-27 09:56:48 -04:00
xiaoxichen
bcd8a71a28 Fix interger overflow on i386 arch
The error was:

util/logging.cc: In function 'int rocksdb::AppendHumanMicros(uint64_t, char*, int)':
error: util/logging.cc:41:39: integer overflow in expression [-Werror=overflow]
} else if (micros < 1000000l * 60 * 60) {
^
error: util/logging.cc:41:39: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
2015-03-27 08:38:53 +08:00
Igor Canadi
030859eb5d Dump compression info on startup
Summary: It's useful to know if we have compression support or no

Test Plan:
Observed this in my LOG:

      2015/03/26-10:34:35.460681 7f5b322b7840 Snappy supported
      2015/03/26-10:34:35.460682 7f5b322b7840 Zlib supported
      2015/03/26-10:34:35.460686 7f5b322b7840 Bzip supported
      2015/03/26-10:34:35.460687 7f5b322b7840 LZ4 NOT supported

Reviewers: sdong, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35955
2015-03-26 11:22:20 -07:00
Igor Canadi
3539e06448 Merge pull request #558 from aamihailov/master
fix compilation error (same as fix #284)
2015-03-26 10:02:09 -04:00
Alexander.Mikhaylov
a3e4b32483 fix compilation error (same as fix #284)
[maa@srv2-nskb-devg2 rocksdb-master]$ CXX=/usr/local/CC/gcc-4.7.4/bin/g++ EXTRA_CXXFLAGS=-std=c++11 DISABLE_WARNING_AS_ERROR=1  make db_bench
  CC       db/db_bench.o
db/db_bench.cc: In member function 'rocksdb::Slice rocksdb::Benchmark::AllocateKey(std::unique_ptr<const char []>*)':
db/db_bench.cc:1434:41: error: use of deleted function 'void std::unique_ptr<_Tp [], _Dp>::reset(_Up) [with _Up = char*; _Tp = const char; _Dp = std::default_delete<const char []>]'
In file included from /usr/local/CC/gcc-4.7.4/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/memory:86:0,
                 from ./include/rocksdb/db.h:14,
                 from ./db/dbformat.h:14,
                 from ./db/db_impl.h:21,
                 from db/db_bench.cc:33:
2015-03-26 14:53:42 +06:00
Jim Meyering
ff1ff7c624 TSAN: avoid new link failure with -pg
Summary:
* Makefile (COMPILE_WITH_TSAN): Avoid a link failure by disabling
-pg when building with TSAN enabled.
Now that "make check" builds all $(PROGRAMS), it is linking
a few programs that were not normally linked before.
For example, this would fail to link with the following diagnostic:

  COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench

    CCLD     log_and_apply_bench
  ld: /usr/lib/../lib64/gcrt1.o: relocation R_X86_64_32S against `__libc_csu_fini' can not be used when making a shared object; recompile with -fPIC
  /usr/lib/../lib64/gcrt1.o: error adding symbols: Bad value
  collect2: error: ld returned 1 exit status
  Makefile:511: recipe for target 'log_and_apply_bench' failed
  make: *** [log_and_apply_bench] Error 1

Since removing -pg is sufficient to get past this link
failure, and no one cares about profiling TSAN-enabled
binaries anyway, we will refrain from linking with -pg
when TSAN testing is enabled.  Use a new variable, "pg"
which is set to "-pg" in most cases, but that is made
empty when COMPILE_WITH_TSAN is set.

Test Plan:
  Now, this succeeds:

    rm -f log_and_apply_bench
    COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench

Reviewers: igor.sugak, rven, sdong, ljin, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35943
2015-03-25 14:40:41 -07:00
Yueh-Hsuan Chiang
39d508e34c Add a missing section title in HISTORY.md
Summary:
Add a missing section title in HISTORY.md

Test Plan:
no code change
2015-03-25 14:14:26 -07:00
Yueh-Hsuan Chiang
727684bf97 Fixed a typo in RocksDBSample.java
Summary:
Fixed a typo in RocksDBSample.java

Test Plan:
make clean
make rocksdbjava -j32
make jtest
2015-03-25 11:09:30 -07:00
Jim Meyering
ca25e86efc build: cause the "check" rule to depend on $(PROGRAMS)
Summary:
* Makefile (check): Cause "make check" to build all $(PROGRAMS),
so that it verifies that the few benchmark-only source files that
not already built via "make check" do compile and link successfully.

Test Plan:
run "make clean; make check", and verify that
table/table_reader_bench.cc is now compiled.
Before, it was not, which led to an incomplete fix
for a build break.

Reviewers: ljin, igor.sugak, rven, igor, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35883
2015-03-24 22:09:40 -07:00
Anurag Indu
986bdc6801 Merge branch 'master' of github.com:facebook/rocksdb 2015-03-24 17:00:41 -07:00
Anurag Indu
1e57f2bf2b Fix build
Test Plan: Running make all

Reviewers: sdong

Reviewed By: sdong

Subscribers: rven, yhchiang, igor, meyering, dhruba

Differential Revision: https://reviews.facebook.net/D35889
2015-03-24 17:00:28 -07:00
Yueh-Hsuan Chiang
2d417e52df Update HISTORY.md for 3.10.0
Summary: Update HISTORY.md for 3.10.0

Test Plan: no code chagne.

Reviewers: sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35871
2015-03-24 16:39:39 -07:00
Anurag Indu
cbd6a20738 Merge branch 'master' of github.com:facebook/rocksdb 2015-03-24 16:27:39 -07:00
Anurag Indu
211ca26aee Fixing build issue
Summary: Fixing issues with get context function.

Test Plan: Run make commit-prereq

Reviewers: sdong, meyering, yhchiang

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35853
2015-03-24 16:27:24 -07:00
Yueh-Hsuan Chiang
2495f93966 Merge pull request #555 from pshareghi/staticLZ4
[RocksJava] Added static lz4 support for roccksjavastatic
2015-03-24 15:39:42 -07:00
Yueh-Hsuan Chiang
f06de5f232 Merge pull request #556 from fyrz/RocksJava-Maven-Fix
[RocksJava] Maven build fix
2015-03-24 14:58:00 -07:00
Yueh-Hsuan Chiang
cd987c383a Fix compile error when NROCKSDB_THREAD_STATUS is not used.
Summary: Fix compile error when NROCKSDB_THREAD_STATUS is not used.

Test Plan: make dbg OPT=-DNROCKSDB_THREAD_STATUS -j32

Reviewers: sdong, igor, rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35847
2015-03-24 14:52:59 -07:00
fyrz
8f104ae5e0 [RocksJava] Maven build fix 2015-03-24 22:52:30 +01:00
Anurag Indu
3d1a924ff3 Adding stats for the merge and filter operation
Summary:
We have addded new stats and perf_context for measuring the merge and filter operation time consumption.
We have bounded all the merge operations within the GUARD statment and collected the total time for these operations in the DB.

Test Plan: WIP

Reviewers: rven, yhchiang, kradhakrishnan, igor, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D34377
2015-03-24 14:42:04 -07:00
Yueh-Hsuan Chiang
afc51649e2 Merge pull request #546 from fyrz/RocksJava-MemEnv
[RocksJava] Expose MemEnv in RocksJava
2015-03-24 14:01:12 -07:00
Pooya Shareghi
4806cc1269 Added static lz4 support for roccksjavastatic
Usage:
make clean rocksdbjavastatic

Dependency:
LZ4

To install LZ4:
git clone https://github.com/Cyan4973/lz4.git && \
cd lz4/lib && \
make install

Alternativel installation method:
curl -O https://codeload.github.com/Cyan4973/lz4/tar.gz/r127
mv r127 lz4-r127.tar.gz
tar xvzf lz4-r127.tar.gz
cd lz4-r127/lib && \
make install

Signed-off-by: Pooya Shareghi <shareghi@gmail.com>
2015-03-24 13:48:53 -07:00
Yueh-Hsuan Chiang
6284eef4c0 Merge pull request #545 from fyrz/RocksJava-Level-Compression
[RocksJava] Add compression per level to API
2015-03-24 12:14:09 -07:00
Yueh-Hsuan Chiang
248c063ba1 Report elapsed time in micros in ThreadStatus instead of start time.
Summary:
Report elapsed time of a thread operation in micros in ThreadStatus
instead of start time of a thread operation in seconds since the
Epoch, 1970-01-01 00:00:00 (UTC).

Test Plan:
./db_bench --benchmarks=fillrandom --num=100000 --threads=40 \
--max_background_compactions=10 --max_background_flushes=3 \
--thread_status_per_interval=1000 --key_size=16 --value_size=1000 \
--num_column_families=10

Sample Output:
            ThreadID ThreadType                    cfName    Operation  ElapsedTime                                         Stage        State
     140667724562496   High Pri column_family_name_000002        Flush   772.419 ms                    FlushJob::WriteLevel0Table
     140667728756800   High Pri                   default        Flush   617.845 ms                    FlushJob::WriteLevel0Table
     140667732951104   High Pri column_family_name_000005        Flush   772.078 ms                    FlushJob::WriteLevel0Table
     140667875557440    Low Pri column_family_name_000008   Compaction  1409.216 ms                        CompactionJob::Install
     140667737145408    Low Pri
     140667749728320    Low Pri
     140667816837184    Low Pri column_family_name_000007   Compaction  1071.815 ms      CompactionJob::ProcessKeyValueCompaction
     140667787477056    Low Pri column_family_name_000009   Compaction   772.516 ms      CompactionJob::ProcessKeyValueCompaction
     140667741339712    Low Pri
     140667758116928    Low Pri column_family_name_000004   Compaction   620.739 ms      CompactionJob::ProcessKeyValueCompaction
     140667753922624    Low Pri
     140667842003008    Low Pri column_family_name_000006   Compaction  1260.079 ms      CompactionJob::ProcessKeyValueCompaction
     140667745534016    Low Pri

Reviewers: sdong, igor, rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35769
2015-03-24 11:32:25 -07:00
Igor Canadi
315abac945 Undeprecate GetLiveFiles()
Summary: There is no alternative to GetLiveFiles() function

Test Plan: none

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35805
2015-03-24 09:42:38 -07:00
Yueh-Hsuan Chiang
a057bb2a8e Improve ThreadStatusSingleCompaction
Summary:
Improve ThreadStatusSingleCompaction in two ways:
1. Use SYNC_POINT to ensure compaction won't happen
   before the test finishes its "Put Phase" instead of
   using sleep.
2. In Put Phase, it continues until we have sufficient
   number of L0 files.  Note that during the put phase,
   there won't be any compaction that consumes L0 files
   because of item 1.

Test Plan: ./db_test  --gtest_filter="*ThreadStatusSingleCompaction*"

Reviewers: sdong, igor, rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35727
2015-03-23 15:30:45 -07:00
krad
868968c8e5 Merge branch 'master' of github.com:facebook/rocksdb 2015-03-23 15:04:48 -07:00
krad
689391406a Make SSTDumpTest.GetProperties less noisy
Summary:
Limiting verbose printing to "command=scan"

Test Plan:
Run make check and manual testing of sst_dump_test

Reviewers: sdong

CC: leveldb

Task ID: #6575982

Blame Rev:
2015-03-23 14:30:11 -07:00
Yueh-Hsuan Chiang
8d8656243e Merge pull request #551 from fyrz/RocksJava-JavaDoc-Fix
[RocksJava] Add missing JavaDoc annotations
2015-03-23 13:25:37 -07:00
Yueh-Hsuan Chiang
f8c505b23a Merge pull request #550 from fyrz/RocksJava-Sample-Fix
[RocksJava] Java sample bugfix
2015-03-23 13:24:37 -07:00
fyrz
46443bfa94 [RocksJava] Add missing JavaDoc annotations 2015-03-23 21:17:20 +01:00
fyrz
864b7e88f4 [RocksJava] Java sample bugfix
One of the latest commits broke the sample code. This
resolves the introduced bug.
2015-03-23 21:03:28 +01:00
fyrz
2b2394cbbb [RocksJava] DBBenchmark option for RocksMemEnv
Extended DBBenchmark with option to enable RocksMemEnv.
2015-03-23 18:59:32 +01:00
fyrz
fd8804f979 [RocksJava] Expose MemEnv in RocksJava
Summary:
In 3.10 the C++ code was extended with a MemEnv implementation. This
is now also available in RocksJava.

Changes:
- Extraced abstract super class Env
- Introduced RocksMemEnv
- Remove unnecessary disposeInternal method. The disposal of the default environment is managed by C++ so there needs to be no disposeInternal method in Java.
- Introduced a RocksMemEnvTest, which is aligned with the C++ equivalent.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35619
2015-03-23 18:59:31 +01:00
fyrz
004b89fba4 [RocksJava] Add compression per level to API
Summary:
RocksDB offers the possibility to set different compression types
on a per level basis. This shall be also available using RocksJava.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35577
2015-03-23 18:58:56 +01:00
Adam Retter
51da3aab4a Merge pull request #536 from fyrz/RocksJava-32-Bit-adjustment
[RocksJava] 32-Bit adjustments
2015-03-22 23:33:35 +00:00
Adam Retter
b7e88cfb5f Merge pull request #543 from fyrz/RocksJava-Logger-Comment
[RocksJava] Enhanced Logger comment
2015-03-22 23:16:02 +00:00