Commit Graph

1570 Commits

Author SHA1 Message Date
Igor Canadi
161d9e586b Don't overflow size_t in mac 2014-04-16 15:15:22 -07:00
Igor Canadi
5c12f27791 Remove tautological assert 2014-04-16 09:09:28 -07:00
Igor Canadi
faf7691358 Close DB at the end of DontRollEmptyLogs test 2014-04-15 17:20:56 -07:00
Igor Canadi
1803ed2ccb Fix Mac OS compile 2014-04-15 16:31:49 -07:00
Igor Canadi
7d838856cf Fix compile issues when doing make release 2014-04-15 16:00:10 -07:00
sdong
0f40fe4bc7 When creating a new DB, fail it when wal_dir contains existing log files
Summary: Current behavior of creating new DB is, if there is existing log files, we will go ahead and replay them on top of empty DB. This is a behavior that no user would expect. With this patch, we will fail the creation if a user creates a DB with existing log files.

Test Plan: make all check

Reviewers: haobo, igor, ljin

Reviewed By: haobo

CC: nkg-, yhchiang, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D17817
2014-04-15 14:01:57 -07:00
Igor Canadi
c166615850 Fix compile issues introduced by RocksDBLite 2014-04-15 13:51:07 -07:00
Igor Canadi
588bca2020 RocksDBLite
Summary:
Introducing RocksDBLite! Removes all the non-essential features and reduces the binary size. This effort should help our adoption on mobile.

Binary size when compiling for IOS (`TARGET_OS=IOS m static_lib`) is down to 9MB from 15MB (without stripping)

Test Plan: compiles :)

Reviewers: dhruba, haobo, ljin, sdong, yhchiang

Reviewed By: yhchiang

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17835
2014-04-15 13:39:26 -07:00
Igor Canadi
23c8f89b57 Revert "Don't compile ldb tool into static library"
This reverts commit e296577ef6.
2014-04-15 11:29:02 -07:00
Igor Canadi
a347ffe92f Revert "Fix sst_dump and reduce_levels_test compile errors"
This reverts commit d8f00b4109.
2014-04-15 11:28:52 -07:00
Igor Canadi
d8f00b4109 Fix sst_dump and reduce_levels_test compile errors 2014-04-15 11:13:12 -07:00
Igor Canadi
e296577ef6 Don't compile ldb tool into static library
Summary:
This is first step of my effort to reduce size of librocksdb.a for use in mobile.

ldb object files are huge and are ment to be used as a command line tool. I moved them to `tools/` directory and include them only when compiling `ldb`

This diff reduced librocksdb.a from 42MB to 39MB on my mac (not stripped).

Test Plan: ran ldb

Reviewers: dhruba, haobo, sdong, ljin, yhchiang

Reviewed By: yhchiang

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17823
2014-04-15 10:52:39 -07:00
Igor Canadi
dbe0f327ca Set log_empty to false even when options.sync is off [fix tests] 2014-04-15 10:28:34 -07:00
Igor Canadi
e6acb874cd Don't roll empty logs
Summary:
With multiple column families, especially when manual Flush is executed, we might roll the log file, although the current log file is empty (no data has been written to the log).

After the diff, we won't create new log file if current is empty.

Next, I will write an algorithm that will flush column families that reference old log files (i.e., that weren't flushed in a while)

Test Plan: Added an unit test. Confirmed that unit test failes in master

Reviewers: dhruba, haobo, ljin, sdong

Reviewed By: ljin

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17631
2014-04-15 09:57:25 -07:00
sdong
c87ed0942c Fix db_bench's multireadrandom
Summary: multireadrandom is broken. Fix it

Test Plan: run it and see segfault has gone.

Reviewers: ljin

Reviewed By: ljin

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17781
2014-04-14 15:43:34 -07:00
Yueh-Hsuan Chiang
118f88d25d Fix compile error in tailing_iter.h
Summary:
Fix the following compile error

  ./db/tailing_iter.h:17:1: error: class 'SuperVersion' was previously declared as a struct [-Werror,-Wmismatched-tags]
  class SuperVersion;
  ^
  ./db/column_family.h:77:8: note: previous use is here
  struct SuperVersion {
         ^
         ./db/tailing_iter.h:17:1: note: did you mean struct here?
         class SuperVersion;
         ^~~~~
         struct
         1 error generated.

Test Plan: make

Reviewers: ljin, igor, haobo, sdong

Reviewed By: ljin

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17799
2014-04-14 14:05:15 -07:00
Yueh-Hsuan Chiang
51f4b5090d [Java] Add SizeUnit in org.rocksdb.util to store const like KB, GB.
Summary:
* Add a class SizeUnit to store frequently used consts.  Currently
  it has KB, MB, GB, TB, and PB.

* Change the parameter type of Options.writeBufferSize and Options.blockSize
  from int to long.

Test Plan:
make rocksdbjava
make jtest

Reviewers: haobo, ankgup87, sdong, dhruba

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17703
2014-04-14 14:03:43 -07:00
Yueh-Hsuan Chiang
31e7e7fe84 [JAVA] Add java binding for Options.block_cache.
Summary:
Add java bindings for Options.block_cache and allow DbBenchmark to
set cache_size.

Test Plan:
make rocksdbjava
make jtest
make jdb_Bench

Reviewers: haobo, sdong, ankgup87

Reviewed By: ankgup87

CC: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D17481
2014-04-14 13:42:36 -07:00
Yueh-Hsuan Chiang
2885ad9b77 [JNI] Each set function of Options / WriteOptions now returns its option instance.
Summary:
Make each set function of Options / WriteOptions return its option instance.
Java developers can now easier specify each option like the following:

    options.setCreateIfMissing(true)
        .setWriteBufferSize(8 * 1024)
        .setMaxWriteBufferNumber(3)
        .setDisableSeekCompaction(true)
        .setBlockSize(64 * 1024)
        .setMaxBackgroundCompactions(10);

Test Plan:
make rocksdbjava
make jtest

Reviewers: haobo, ankgup87, sdong, dhruba

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17661
2014-04-14 12:48:50 -07:00
Igor Canadi
be016613c2 Expose in memory Env to the world
Summary: That will help with some iOS testing I'm doing.

Test Plan: compiles

Reviewers: dhruba, haobo, ljin, yhchiang, sdong

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17787
2014-04-14 12:28:15 -07:00
Yueh-Hsuan Chiang
327102efa5 Fix merge_test failure due to incorrect assert behavior in the release mode. 2014-04-14 12:06:49 -07:00
Lei Jin
82b37a18bd thread local for tailing iterator
Summary:
replace the super version acquisision in tailing itrator with thread
local

Test Plan: will post results

Reviewers: igor, haobo, sdong, yhchiang, dhruba

Reviewed By: igor

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17757
2014-04-14 10:48:01 -07:00
Lei Jin
539dd207df using thread local SuperVersion for NewIterator
Summary:
Similar to GetImp(), use SuperVersion from thread local instead of acquriing mutex.
I don't expect this change will make a dent on NewIterator() performance
because the bottleneck seems to be on the rest part of the API

Test Plan:
make asan_check
will post perf numbers

Reviewers: haobo, igor, sdong, dhruba, yhchiang

Reviewed By: sdong

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17643
2014-04-14 09:34:59 -07:00
sdong
d5e087b6df db_bench: add a mode to operate multiple DBs
Summary: This patch introduces a new parameter num_multi_db in db_bench. When this parameter is larger than 1, multiple DBs will be created. In all benchmarks, any operation applies to a random DB among them. This is to benchmark the performance of similar applications.

Test Plan: run db_bench on both of num_multi_db=0 and more.

Reviewers: haobo, ljin, igor

Reviewed By: igor

CC: igor, yhchiang, dhruba, nkg-, leveldb

Differential Revision: https://reviews.facebook.net/D17769
2014-04-11 16:59:08 -07:00
Igor Canadi
30aff72f77 Don't shadow in ColumnFamilyDescriptor 2014-04-11 14:48:20 -07:00
Lei Jin
eba3fc644a make corruption_test:CompactionInputErrorParanoid deterministic
Summary:
it writes ~10M data, default L0 compaction trigger is 4, plus 2 writer
buffer, so that can accommodate ~6M data before compaction happens for
sure. I guess encoding is doing a good job to shrink the data so that
sometime, compaction does not get triggered. I get test failure quite
often.

Test Plan: ran it multiple times and all got pass

Reviewers: igor, sdong

Reviewed By: sdong

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17775
2014-04-11 12:48:38 -07:00
Igor Canadi
9433e359d1 Dont build version if compiling for IOS 2014-04-11 10:54:47 -07:00
Igor Canadi
de41357a18 Don't dump rocksdb version on IOS 2014-04-11 10:19:58 -07:00
Lei Jin
0af36d6aa6 SeekRandomWhileWriting
Summary: as title

Test Plan: ran it

Reviewers: igor, haobo, yhchiang

Reviewed By: yhchiang

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17751
2014-04-11 09:47:20 -07:00
Kai Liu
e23e73e67c Use shorten index key for hash-index
Summary:
I was wrong about the "index builder", right now since we create index
by scanning both whole table and index, there is not need to preserve
the whole key as the index key.

I switch back to original way index which is both space efficient and
able to supprot in-fly construction of hash index.

IN this patch, I made minimal change since I'm not sure if we still need
the "pluggable index builder", under current circumstance it is of no use
and kind of over-engineered. But I'm not sure if we can still exploit its
usefulness in the future; otherwise I think I can just burn them with great
vengeance.

Test Plan: unit tests

Reviewers: sdong, haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17745
2014-04-10 22:45:25 -07:00
Igor Canadi
b3d7435b4e No shadow in public headers 2014-04-10 17:19:03 -07:00
Kai Liu
1405232b6d Temporarily disable a test case in db_test
Summary:

Root cause is still under investigation. Just Disable the troubling use case for now.
2014-04-10 17:17:39 -07:00
Igor Canadi
6c0fc5d3ba Fix some InfoLogLevels 2014-04-10 16:49:49 -07:00
Yueh-Hsuan Chiang
dfe2d2f3a2 [JNI] Add readwhilewriting to Java db_bench
Summary: Add readwhilewriting to Java db_bench

Test Plan:
make jni
make jdb_bench

Reviewers: haobo, ankgup87, sdong, dhruba

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17619
2014-04-10 16:12:04 -07:00
Igor Canadi
ddef6841b3 Renamed InfoLogLevel::DEBUG to InfoLogLevel::DEBUG_LEVEL
Summary: XCode for some reason injects `#define DEBUG 1` into our code, which makes compile fail because we use `DEBUG` keyword for other stuff. This diff fixes the issue by renaming `DEBUG` to `DEBUG_LEVEL`.

Test Plan: compiles

Reviewers: dhruba, haobo, sdong, yhchiang, ljin

Reviewed By: yhchiang

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17709
2014-04-10 15:27:42 -07:00
Kai Liu
75b59d5146 Enable hash index for block-based table
Summary: Based on previous patches, this diff eventually provides the end-to-end mechanism for users to specify the hash-index.

Test Plan: Wrote several new unit tests.

Reviewers: sdong, haobo, dhruba

Reviewed By: sdong

CC: leveldb

Differential Revision: https://reviews.facebook.net/D16539
2014-04-10 14:19:43 -07:00
Lei Jin
7a92537fc4 db_bench: add IteratorCreationWhileWriting mode and allow prefix_seek
Summary: as title

Test Plan: ran it

Reviewers: igor, haobo, yhchiang

Reviewed By: igor

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17655
2014-04-10 10:15:59 -07:00
Yueh-Hsuan Chiang
ca4fa2047e [Java] rename 'make jni' to 'make rocksdbjava' 2014-04-10 10:04:48 -07:00
Igor Canadi
4daea66343 Turn on -Wmissing-prototypes
Summary: Compiling for iOS has by default turned on -Wmissing-prototypes, which causes rocksdb to fail compiling. This diff turns on -Wmissing-prototypes in our compile options and cleans up all functions with missing prototypes.

Test Plan: compiles

Reviewers: dhruba, haobo, ljin, sdong

Reviewed By: ljin

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17649
2014-04-09 21:17:14 -07:00
sdong
df2a8b6a1a Polish IterKey and use it in DBImpl::ProcessKeyValueCompaction()
Summary:
1. Polish IterKey a little bit.
2. Turn to use it in local parameter of current_user_key in DBImpl::ProcessKeyValueCompaction(). Our profile showing that DBImpl::ProcessKeyValueCompaction() has about 14% costs in std::string (the base including reading and writing data but excluding compaction filtering), which is higher than it should be. There are two std::string used in DBImpl::ProcessKeyValueCompaction(), compaction_filter_value and current_user_key and it's hard to distinguish the two.

Test Plan: make all check

Reviewers: haobo, ljin

Reviewed By: haobo

CC: igor, yhchiang, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D17613
2014-04-09 20:50:58 -07:00
Igor Canadi
dc55903293 Improved CompressedCache
Summary:
This is testing behavior that was reported in https://github.com/facebook/rocksdb/issues/111

No issue was found, but it still good to commit this and make CompressedCache more robust.

Test Plan: this is a plan

Reviewers: ljin, dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17625
2014-04-09 11:43:14 -07:00
Lei Jin
4824014e3b speed up db_bench filluniquerandom mode
Summary:
filluniquerandom is painfully slow due to the naive bitmap check to find
out if a key has been seen before. Majority of time is spent on searching
the last few keys. Split a giant BitSet to smaller ones so that we can
quickly check if a BitSet is full and thus can skip quickly.

It used to take over one hour to filluniquerandom for 100M keys, now it
takes about 10 mins.

Test Plan:
unit test
also verified correctness in db_bench and make sure all keys are
generated

Reviewers: igor, haobo, yhchiang

Reviewed By: igor

CC: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D17607
2014-04-09 11:25:21 -07:00
Igor Canadi
2014915d32 Fix ASAN issue 2014-04-09 10:38:05 -07:00
sdong
e9ed28f9c8 PlainTableBuilder::Add() to use local char array instead of reused std::string as tmp buffer
Summary: Our profile shows that in one of the applications, 5% of the CPU costs of PlainTableBuilder::Add() are spent on std::string stacks. By this simple change, we avoid this global reusable string. Also, we avoid another call of file appending, which probably gives another 2%.

Test Plan: make all check

Reviewers: haobo, ljin

Reviewed By: haobo

CC: igor, yhchiang, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D17601
2014-04-09 10:17:32 -07:00
Igor Canadi
b947fdc89d Column family support for DB::OpenForReadOnly()
Summary: When opening DB in read-only mode, client can choose to only specify a subset of column families ("default" column family can't be omitted, though)

Test Plan: added a unit test in column_family_test

Reviewers: haobo, sdong, ljin, dhruba

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17565
2014-04-09 09:56:17 -07:00
Yueh-Hsuan Chiang
0f5cbcd798 [JNI] Add an initial benchmark for java binding for rocksdb.
Summary:
* Add a benchmark for java binding for rocksdb.  The java benchmark
  is a complete rewrite based on the c++ db/db_bench.cc and the
  DbBenchmark in dain's java leveldb.
* Support multithreading.
* 'readseq' is currently not supported as it requires RocksDB Iterator.

* usage:

  --benchmarks
    Comma-separated list of operations to run in the specified order
        Actual benchmarks:
                fillseq    -- write N values in sequential key order in async mode
                fillrandom -- write N values in random key order in async mode
                fillbatch  -- write N/1000 batch where each batch has 1000 values
                              in random key order in sync mode
                fillsync   -- write N/100 values in random key order in sync mode
                fill100K   -- write N/1000 100K values in random order in async mode
                readseq    -- read N times sequentially
                readrandom -- read N times in random order
                readhot    -- read N times in random order from 1% section of DB
        Meta Operations:
                delete     -- delete DB
    DEFAULT: [fillseq, readrandom, fillrandom]

  --compression_ratio
    Arrange to generate values that shrink to this fraction of
        their original size after compression
    DEFAULT: 0.5

  --use_existing_db
    If true, do not destroy the existing database.  If you set this
        flag and also specify a benchmark that wants a fresh database,  that benchmark will fail.
    DEFAULT: false

  --num
    Number of key/values to place in database.
    DEFAULT: 1000000

  --threads
    Number of concurrent threads to run.
    DEFAULT: 1

  --reads
    Number of read operations to do.  If negative, do --nums reads.

  --key_size
    The size of each key in bytes.
    DEFAULT: 16

  --value_size
    The size of each value in bytes.
    DEFAULT: 100

  --write_buffer_size
    Number of bytes to buffer in memtable before compacting
        (initialized to default value by 'main'.)
    DEFAULT: 4194304

  --cache_size
    Number of bytes to use as a cache of uncompressed data.
        Negative means use default settings.
    DEFAULT: -1

  --seed
    Seed base for random number generators.
    DEFAULT: 0

  --db
    Use the db with the following name.
    DEFAULT: /tmp/rocksdbjni-bench

* Add RocksDB.write().

Test Plan: make jbench

Reviewers: haobo, sdong, dhruba, ankgup87

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17433
2014-04-09 00:48:20 -07:00
Yueh-Hsuan Chiang
c65448f95a Merge branch 'master' of github.com:facebook/rocksdb into HEAD 2014-04-08 22:21:39 -07:00
Igor Canadi
731e55c01c Fix GetProperty() test
Summary:
GetProperty test is flakey.
Before this diff: P8635927
After: P8635945

We need to make sure the thread is done before we destruct sleeping tasks. Otherwise, bad things happen.

Test Plan: See summary

Reviewers: ljin, sdong, haobo, dhruba

Reviewed By: ljin

CC: leveldb

Differential Revision: https://reviews.facebook.net/D17595
2014-04-08 14:57:00 -07:00
Yueh-Hsuan Chiang
2e0d432157 Merge pull request #110 from ankgup87/jni
[JNI] Add JNI bindings for rocksdb Options
2014-04-08 14:15:55 -07:00
Igor Canadi
34455deb06 Fix Mac OS compile issues 2014-04-08 14:05:53 -07:00