Commit Graph

13 Commits

Author SHA1 Message Date
mrambacher
12f1137355 Add a SystemClock class to capture the time functions of an Env (#7858)
Summary:
Introduces and uses a SystemClock class to RocksDB.  This class contains the time-related functions of an Env and these functions can be redirected from the Env to the SystemClock.

Many of the places that used an Env (Timer, PerfStepTimer, RepeatableThread, RateLimiter, WriteController) for time-related functions have been changed to use SystemClock instead.  There are likely more places that can be changed, but this is a start to show what can/should be done.  Over time it would be nice to migrate most (if not all) of the uses of the time functions from the Env to the SystemClock.

There are several Env classes that implement these functions.  Most of these have not been converted yet to SystemClock implementations; that will come in a subsequent PR.  It would be good to unify many of the Mock Timer implementations, so that they behave similarly and be tested similarly (some override Sleep, some use a MockSleep, etc).

Additionally, this change will allow new methods to be introduced to the SystemClock (like https://github.com/facebook/rocksdb/issues/7101 WaitFor) in a consistent manner across a smaller number of classes.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7858

Reviewed By: pdillinger

Differential Revision: D26006406

Pulled By: mrambacher

fbshipit-source-id: ed10a8abbdab7ff2e23d69d85bd25b3e7e899e90
2021-01-25 22:09:11 -08:00
mrambacher
7d472accdc Bring the Configurable options together (#5753)
Summary:
This PR merges the functionality of making the ColumnFamilyOptions, TableFactory, and DBOptions into Configurable into a single PR, resolving any merge conflicts

Pull Request resolved: https://github.com/facebook/rocksdb/pull/5753

Reviewed By: ajkr

Differential Revision: D23385030

Pulled By: zhichao-cao

fbshipit-source-id: 8b977a7731556230b9b8c5a081b98e49ee4f160a
2020-09-14 17:01:01 -07:00
Jay Zhuang
e500c730cf Shutdown timer in destructor (#7292)
Summary:
Make sure deleting a running timer works fine.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7292

Test Plan: unittest and an invalid benchmark command: `./db_bench --db=/tmp --use_existing_db=false --benchmarks=fred --compression_type=none`

Reviewed By: riversand963

Differential Revision: D23248500

Pulled By: jay-zhuang

fbshipit-source-id: 04111681b389a9aa23a439db4568d5ca351f1144
2020-08-21 15:48:52 -07:00
Jay Zhuang
187964a039 Add test function MockTimeEnv.SleepForMicroseconds() (#7293)
Summary:
And change the internal time value from seconds to microseconds.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7293

Reviewed By: pdillinger

Differential Revision: D23253751

Pulled By: jay-zhuang

fbshipit-source-id: 36aa9376b8801b85bd10163173590a17cf4f3a3a
2020-08-21 11:34:37 -07:00
Jay Zhuang
3e422ce0ca Fix a timer_test deadlock (#7277)
Summary:
There's a potential deadlock caused by MockTimeEnv time value get to a large number, which causes TimedWait() wait forever. The test misuses the microseconds as seconds, making it more likely to happen.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7277

Reviewed By: pdillinger

Differential Revision: D23183873

Pulled By: jay-zhuang

fbshipit-source-id: 6fc38ebd40b4125a99551204b271f91a27e70086
2020-08-20 08:43:13 -07:00
Jay Zhuang
69760b4d05 Introduce a global StatsDumpScheduler for stats dumping (#7223)
Summary:
Have a global StatsDumpScheduler for all DB instance stats dumping, including `DumpStats()` and `PersistStats()`. Before this, there're 2 dedicate threads for every DB instance, one for DumpStats() one for PersistStats(), which could create lots of threads if there're hundreds DB instances.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7223

Reviewed By: riversand963

Differential Revision: D23056737

Pulled By: jay-zhuang

fbshipit-source-id: 0faa2311142a73433ebb3317361db7cbf43faeba
2020-08-14 20:12:44 -07:00
Yanqin Jin
76609cd38a Fix potential memory leak (#7245)
Summary:
```
int* value = new int;
ASSERT_NE(nullptr, value);
```
`ASSERT_NE` can expand the expression such that a memory leak is
reported by clang analyzer.
We can remove this ASSERT_NE since we can assume the memory allocation
must succeed. Otherwise a bad alloc exception will be thrown and the
process will be killed anyway.

Test plan (dev server):
```
USE_CLANG=1 make analyze
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7245

Reviewed By: jay-zhuang

Differential Revision: D23079641

Pulled By: riversand963

fbshipit-source-id: a6739a903f90f8715f6f1ef3e5c8a329245b8e78
2020-08-12 12:03:22 -07:00
Yanqin Jin
f15414b656 Timer should run scheduled function without mutex (#7228)
Summary:
Timer (defined in timer.h) schedules and runs user-specified fuctions
regularly. Current implementation holds the mutex while running user
function, which will lead to contention and waiting.
To fix, Timer::Run releases mutex before running user function, and
re-acquires it afterwards.
This fix will impact how we can cancel a task. If the task is running,
it is not holding the mutex. The thread calling Cancel() should wait
until the current task finishes.

Test Plan (devserver):
make check
COMPILE_WITH_ASAN=1 make check

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7228

Reviewed By: jay-zhuang

Differential Revision: D23065487

Pulled By: riversand963

fbshipit-source-id: 07cb59741f506d3eb875c8ab90f73437568d3724
2020-08-11 22:37:39 -07:00
Peter Dillinger
6ac1d25fd0 Fix+clean up handling of mock sleeps (#7101)
Summary:
We have a number of tests hanging on MacOS and windows due to
mishandling of code for mock sleeps. In addition, the code was in
terrible shape because the same variable (addon_time_) would sometimes
refer to microseconds and sometimes to seconds. One test even assumed it
was nanoseconds but was written to pass anyway.

This has been cleaned up so that DB tests generally use a SpecialEnv
function to mock sleep, for either some number of microseconds or seconds
depending on the function called. But to call one of these, the test must first
call SetMockSleep (precondition enforced with assertion), which also turns
sleeps in RocksDB into mock sleeps. To also removes accounting for actual
clock time, call SetTimeElapseOnlySleepOnReopen, which implies
SetMockSleep (on DB re-open). This latter setting only works by applying
on DB re-open, otherwise havoc can ensue if Env goes back in time with
DB open.

More specifics:

Removed some unused test classes, and updated comments on the general
problem.

Fixed DBSSTTest.GetTotalSstFilesSize using a sync point callback instead
of mock time. For this we have the only modification to production code,
inserting a sync point callback in flush_job.cc, which is not a change to
production behavior.

Removed unnecessary resetting of mock times to 0 in many tests. RocksDB
deals in relative time. Any behaviors relying on absolute date/time are likely
a bug. (The above test DBSSTTest.GetTotalSstFilesSize was the only one
clearly injecting a specific absolute time for actual testing convenience.) Just
in case I misunderstood some test, I put this note in each replacement:
// NOTE: Presumed unnecessary and removed: resetting mock time in env

Strengthened some tests like MergeTestTime, MergeCompactionTimeTest, and
FilterCompactionTimeTest in db_test.cc

stats_history_test and blob_db_test are each their own beast, rather deeply
dependent on MockTimeEnv. Each gets its own variant of a work-around for
TimedWait in a mock time environment. (Reduces redundancy and
inconsistency in stats_history_test.)

Intended follow-up:

Remove TimedWait from the public API of InstrumentedCondVar, and only
make that accessible through Env by passing in an InstrumentedCondVar and
a deadline. Then the Env implementations mocking time can fix this problem
without using sync points. (Test infrastructure using sync points interferes
with individual tests' control over sync points.)

With that change, we can simplify/consolidate the scattered work-arounds.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7101

Test Plan: make check on Linux and MacOS

Reviewed By: zhichao-cao

Differential Revision: D23032815

Pulled By: pdillinger

fbshipit-source-id: 7f33967ada8b83011fb54e8279365c008bd6610b
2020-08-11 12:41:30 -07:00
Jay Zhuang
fea286d914 Fix Timer unable to schedule new added job (#7216)
Summary:
And added test to reproduce the problem.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7216

Reviewed By: riversand963

Differential Revision: D22905193

Pulled By: jay-zhuang

fbshipit-source-id: 8ca1435c91bf829f9076c743bdd66861364ff68c
2020-08-04 09:20:28 -07:00
Jay Zhuang
d941b89ddd Fix hang timer tests on macos (#7208)
Summary:
And re-enable disabled tests.
The issue is caused by `CondVar.TimedWait()` doesn't use `MockTimeEnv`.

Issue: https://github.com/facebook/rocksdb/issues/6698

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7208

Test Plan: `./timer_test --gtest_repeat=1000`

Reviewed By: riversand963

Differential Revision: D22857855

Pulled By: jay-zhuang

fbshipit-source-id: 6d15f65f6ae58b75b76cb132815c16ad81ffd12f
2020-08-03 10:17:01 -07:00
Sagar Vemuri
2e9324718a Disable a few timer tests (#6833)
Summary:
Disable `TimerTest.SingleScheduleRepeatedlyTest` and `TimerTest.MultipleScheduleRepeatedlyTest`. This is to help people to not hit any hangs (https://github.com/facebook/rocksdb/issues/6698) during their development process while I investigate further; I could not reproduce the issue on my dev machine yet. Note that timer is not being utilized anywhere yet.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6833

Test Plan:
```
svemuri@devbig187 ~/rocksdb (timer-disable-test) $ TEST_TMPDIR=/dev/shm ./timer_test
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from TimerTest
[ RUN      ] TimerTest.SingleScheduleOnceTest
[       OK ] TimerTest.SingleScheduleOnceTest (1 ms)
[ RUN      ] TimerTest.MultipleScheduleOnceTest
[       OK ] TimerTest.MultipleScheduleOnceTest (0 ms)
[----------] 2 tests from TimerTest (1 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (1 ms total)
[  PASSED  ] 2 tests.

  YOU HAVE 2 DISABLED TESTS
```

Reviewed By: pdillinger

Differential Revision: D21502474

Pulled By: sagar0

fbshipit-source-id: ac67caee2011fd14ffb2476a8914a6286a4f9abe
2020-05-11 13:30:00 -07:00
Sagar Vemuri
0355d14dd9 Add a simple timer support to schedule work at fixed times/intervals (#6543)
Summary:
Adding a simple timer support to schedule work at a fixed time.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6543

Test Plan: TODO: clean up the unit tests, and make them better.

Reviewed By: siying

Differential Revision: D20465390

Pulled By: sagar0

fbshipit-source-id: cba143f70b6339863e1d0f8b8bf92e51c2b3d678
2020-04-07 11:55:27 -07:00