Commit Graph

2901 Commits

Author SHA1 Message Date
Mark Callaghan
32a0a03844 Add Moved(GB) to Compaction IO stats
Summary:
Adds counter for bytes moved (files pushed down a level rather than compacted) to compaction
IO stats as Moved(GB). From the output removed these infrequently used columns: RW-Amp, Rn(cnt), Rnp1(cnt),
Wnp1(cnt), Wnew(cnt).
Example old output:
Level   Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s)  Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt)  Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) RecordIn RecordDrop
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0     0/0          0   0.0      0.0     0.0      0.0    2130.8   2130.8    0.0   0.0      0.0    109.1        0         0         0         0      20002     25068    0.798      28.75     182059    0.16       0          0
  L1   142/0        509   1.0   4618.5  2036.5   2582.0    4602.1   2020.2    4.5   2.3     88.5     88.1    24220    701246   1215528    514282      53466      4229   12.643       0.00          0    0.002032745988  300688729

Example new output:
Level   Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms)     RecordIn   RecordDrop
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0     7/0         13   1.8      0.0     0.0      0.0       0.6      0.6       0.0   0.0      0.0     14.7        44       353    0.124       0.03        626    0.05            0            0
  L1     9/0         16   1.6      0.0     0.0      0.0       0.0      0.0       0.6   0.0      0.0      0.0         0         0    0.000       0.00          0    0.00            0            0

Task ID: #

Blame Rev:

Test Plan:
make check, run db_bench --fillseq --stats_per_interval --stats_interval and look at output

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D29787
2014-12-03 18:28:39 -08:00
Jonah Cohen
a14b7873ee Enforce write buffer memory limit across column families
Summary:
Introduces a new class for managing write buffer memory across column
families.  We supplement ColumnFamilyOptions::write_buffer_size with
ColumnFamilyOptions::write_buffer, a shared pointer to a WriteBuffer
instance that enforces memory limits before flushing out to disk.

Test Plan: Added SharedWriteBuffer unit test to db_test.cc

Reviewers: sdong, rven, ljin, igor

Reviewed By: igor

Subscribers: tnovak, yhchiang, dhruba, xjin, MarkCallaghan, yoshinorim

Differential Revision: https://reviews.facebook.net/D22581
2014-12-02 12:09:20 -08:00
fyrz
3e684aa685 Integrated changes from D29571 2014-12-02 19:56:45 +01:00
Igor Canadi
37d73d597e Fix linters
Summary:
Two fixes:
1. if cpplint is not present on the system, don't return a confusing error in the linter
2. Add include_alpha, which means our includes should be sorted lexicographically

Test Plan: Tried unsorting our includes, lint complained

Reviewers: rven, ljin, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28845
2014-12-02 13:53:39 -05:00
fyrz
a15169f2e9 Fixed a Lint problem 2014-12-02 09:58:20 +01:00
fyrz
b7f9e644cc [RocksJava] Quality improvements
Summary:
- Addressed some FindBugs issues.
- Remove obsolete dbFolder cleanup
- Comparator tests for CF
 - Added AbstractComparatorTest.
 - Fixed a bug in the JNI Part about Java comparators
- Minor test improvements

Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D29571
2014-12-02 09:58:19 +01:00
fyrz
e002a6122f [RocksJava] Comparator tests for CF
- Added AbstractComparatorTest.
- Fixed a bug in the JNI Part about Java comparators
2014-12-02 09:58:19 +01:00
fyrz
335e6ad5cd [RocksJava] Remove obsolete dbFolder cleanup 2014-12-02 09:58:18 +01:00
fyrz
b036804ac1 RocksJava - FindBugs issues
Addressed some FindBugs issues.
2014-12-02 09:58:17 +01:00
Igor Canadi
9a632b4a92 Merge pull request #429 from fyrz/RocksJava-MacOSX-strip-fix
[RocksJava] MacOSX strip support
2014-12-01 13:13:02 -05:00
fyrz
b426675061 [RocksJava] MacOSX strip support 2014-12-01 19:01:29 +01:00
Igor Canadi
e463cb0bcf Merge pull request #424 from eile/master
Tweak Makefile for building on BG/Q
2014-12-01 10:20:12 -05:00
Stefan Eilemann
91d8981639 Tweak Makefile for building on BG/Q 2014-12-01 09:01:54 +01:00
Haneef Mubarak
c6f31a2893 minor memory leak in C example 2014-11-29 21:42:42 -08:00
Igor Canadi
703ef66a86 Merge pull request #426 from fyrz/RocksJava-Restore-PrecisionFix
[RocksJava] Fixed MacOS build of RocksJava
2014-11-27 20:39:39 -05:00
Haneef Mubarak
ac4ed1e305 fix examples/makefile for C example 2014-11-27 15:20:55 -08:00
Haneef Mubarak
d7f5ccb0c2 add c example to makefile and fix "make clean" 2014-11-27 15:06:12 -08:00
Haneef Mubarak
9c34d5e361 fix type in C simple example 2014-11-27 13:53:04 -08:00
Haneef Mubarak
0a9a7e753c added C version of simple_example 2014-11-27 13:49:19 -08:00
Yueh-Hsuan Chiang
bcf9086899 Block Universal and FIFO compactions in ROCKSDB_LITE
Summary: Block Universal and FIFO compactions in ROCKSDB_LITE

Test Plan:
make shared_lib -j32
make OPT=-DROCKSDB_LITE shared_lib

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29589
2014-11-26 15:45:11 -08:00
fyrz
67cb7ca758 [RocksJava] Fixed MacOS build of RocksJava
There were still some precision loss problems
remainging in RocksJava. This pull request resolve
these.
2014-11-26 20:53:23 +01:00
Yueh-Hsuan Chiang
b8136a7d27 Merge pull request #398 from fyrz/RocksJava-CreateCheckPoint
[RocksJava] Support for stored snapshots
2014-11-26 11:40:41 -08:00
Yueh-Hsuan Chiang
533592a27d Merge pull request #401 from fyrz/RocksJava-Sigsegv-MergeOperatorName
[RocksJava] Fixes in MergeOperatorByName
2014-11-26 11:40:13 -08:00
Yueh-Hsuan Chiang
73d72ed5c7 Block ReadOnlyDB in ROCKSDB_LITE
Summary:
db_imp_readonly.o is one of the big obj file.  If it's not a necessary
feature, we should probably block it in ROCKSDB_LITE.

    1322704 Nov 24 16:55 db/db_impl_readonly.o

Test Plan:
make shared_lib -j32
make ROCKSDB_LITE shared_lib -j32

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29583
2014-11-26 11:37:59 -08:00
Igor Canadi
e47f0fa9ed Merge pull request #425 from adamretter/macosx-clean-fix
Mac OS X jclean fix
2014-11-26 13:40:32 -05:00
Adam Retter
ff0cb90d1c Do not delete Java Fatal Error Log, developers may still want these for
reference
2014-11-26 18:18:04 +00:00
Adam Retter
2a792cd300 There will also be a librocksdbjni-osx.jnilib.dSYM folder on MacOSX
builds to be deleted
2014-11-26 18:17:54 +00:00
Yueh-Hsuan Chiang
beb74c14ca Fix travis-build error
Summary:
Fix travis-build error
2014-11-26 09:42:13 -08:00
Igor Canadi
a486352e07 Merge pull request #423 from zerebubuth/c_iterate_upper_bound
C iterate upper bound
2014-11-26 10:14:16 -05:00
Matt Amos
26109d487a Store upper bound Slice with the same lifetime as the ReadOptions so that we can provide a pointer to it. 2014-11-26 11:29:13 +00:00
Yueh-Hsuan Chiang
a97314219e Fix compile error in ROCKSDB_LITE 2014-11-25 23:39:52 -08:00
Yueh-Hsuan Chiang
9d5019327b Replace log2 by implementing Log2 in options_builder
Summary:
log2 function is only used in options_builder, and this function
is not available under certain platform such as android.
This patch implements Log2 by log(n) / log(2).

Test Plan:
make
2014-11-25 23:28:36 -08:00
Matt Amos
805bac6d25 Add test for upper bounds on iterators using C interface. 2014-11-25 23:07:40 +00:00
fyrz
f193deea31 [RocksJava] Addressed comments in D28971 2014-11-25 23:33:42 +01:00
fyrz
94f70a86b9 [RocksJava] Incoroporated changes for D29013 2014-11-25 22:26:03 +01:00
fyrz
a280af2a57 [RocksJava] Sigsegv fix for MergerOperatorByName 2014-11-25 22:26:02 +01:00
fyrz
fcc2dfd9f9 [RocksJava] Support for stored snapshots
Summary:
RocksDB supports two ways of saving snapshots. In
memory and on disk. The later was added with this
pull request to RocksJava.

Test Plan:

Reviewers:

Subscribers:
2014-11-25 22:25:22 +01:00
Yueh-Hsuan Chiang
274ba62707 Block internal_stats in ROCKSDB_LITE
Summary: Block internal_stats in ROCKSDB_LITE.

Test Plan: make OPT=-DROCKSDB_LITE shared_lib

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29541
2014-11-25 12:01:27 -08:00
Igor Canadi
4f2e8bab5f Merge pull request #421 from fyrz/RocksJava-PrecisionFix
[RocksJava] Fix precision problem in rocksjni
2014-11-25 14:32:44 -05:00
fyrz
c4765dc10b [RocksJava] Fix precision problem in rocksjni 2014-11-25 20:29:52 +01:00
Igor Canadi
14788e1811 Merge pull request #420 from rdallman/add-wal
c api: add max wal total to opts
2014-11-25 09:34:15 -05:00
Reed Allman
88dd8d889b c api: add max wal total to opts 2014-11-24 22:00:29 -08:00
Yueh-Hsuan Chiang
7e608e2fe3 Block plain_table_index.cc in ROCKSDB_LITE
Summary: Block plain_table_index.cc in ROCKSDB_LITE

Test Plan:
make clean
make OPT=-DROCKSDB_LITE shared_lib -j32
make clean
make shared_lib -j32

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29535
2014-11-24 20:47:27 -08:00
Yueh-Hsuan Chiang
13de000f07 Add rocksdb::ToString() to address cases where std::to_string is not available.
Summary:
In some environment such as android, the c++ library does not have
std::to_string.  This path adds rocksdb::ToString(), which wraps std::to_string
when std::to_string is not available, and implements std::to_string
in the other case.

Test Plan:
make dbg -j32
./db_test
make clean
make dbg OPT=-DOS_ANDROID -j32
./db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29181
2014-11-24 20:44:49 -08:00
Yueh-Hsuan Chiang
90ee85f8e1 Improve listener_test to avoid possible false alarm
Summary:
Improve listener_test to avoid possible false alarm

Test Plan:
./listener_test
2014-11-24 18:28:06 -08:00
Lei Jin
2946e37a08 remove unreliable test in db/cuckoo_table_db_test.cc
Summary:
This compaction trigger does not seem to test any thing specific to
cuckoo table. Remove it.

Test Plan: make all check

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29523
2014-11-24 15:18:09 -08:00
Lei Jin
9c7ca65d21 free builders in VersionSet::DumpManifest
Summary:
Reported by bootcamper
This causes ldb tool to fail the assertion in ~ColumnFamilyData()

Test Plan:
./ldb --db=/tmp/test_db1 --create_if_missing put a1 b1
./ldb manifest_dump --path=/tmp/test_db1/MANIFEST-000001

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29517
2014-11-24 15:03:08 -08:00
Igor Canadi
7530c75ab0 Merge pull request #413 from saghmrossi/master
first rdb commit
2014-11-24 16:54:44 -05:00
Igor Canadi
d699d70343 Make RocksDB compile without gflags
Summary: We want to make sure people without gflags can compile RocksDB.

Test Plan: remove gflags, make all

Reviewers: sdong, rven, yhchiang, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29469
2014-11-24 15:53:23 -05:00
Venkatesh Radhakrishnan
3257221499 Fixes valgrind error in GetSnapshotLink. Free checkpoint now.
Summary: Free checkpoint after its directory is removed.

Test Plan: Run valgrind with GetSnapshotLink.

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29493
2014-11-24 10:20:50 -08:00