andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
ed75dddc35
rocksdb/db_stress_tool
History
Andrew Kryczka ed75dddc35 Optimize db_stress setup phase (#9475) 
Summary:
It is too slow that our `db_crashtest.py` often kills `db_stress` before
the setup phase completes. Profiled it and found a few ways to optimize.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9475

Test Plan:
Measured setup phase time reduced 22% (36 -> 28 seconds) for first run, and
36% (38 -> 24 seconds) for non-first run on empty-ish DB.

- first run benchmark command: `rm -rf /dev/shm/dbstress*/ && mkdir -p /dev/shm/dbstress_expected/ && ./db_stress -max_key=100000000 -destroy_db_initially=1 -expected_values_dir=/dev/shm/dbstress_expected/ -db=/dev/shm/dbstress/ --clear_column_family_one_in=0 --reopen=0 --nooverwritepercent=1`

output before this PR:

```
2022/01/31-11:14:05  Initializing db_stress
...
2022/01/31-11:14:41  Starting database operations
```

output after this PR:

```
...
2022/01/31-11:12:23  Initializing db_stress
...
2022/01/31-11:12:51  Starting database operations
```

- non-first run benchmark command: `./db_stress -max_key=100000000 -destroy_db_initially=0 -expected_values_dir=/dev/shm/dbstress_expected/ -db=/dev/shm/dbstress/ --clear_column_family_one_in=0 --reopen=0 --nooverwritepercent=1`

output before this PR:

```
2022/01/31-11:20:45  Initializing db_stress
...
2022/01/31-11:21:23  Starting database operations
```

output after this PR:

```
2022/01/31-11:22:02  Initializing db_stress
...
2022/01/31-11:22:26  Starting database operations
```

- ran minified crash test a while: `DEBUG_LEVEL=0 TEST_TMPDIR=/dev/shm python3 tools/db_crashtest.py blackbox --simple --interval=10 --max_key=1000000 --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --value_size_mult=33`

Reviewed By: anand1976

Differential Revision: D33897793

Pulled By: ajkr

fbshipit-source-id: 0d7b2c93e1e2a9f8a878e87632c2455406313087
2022-02-01 11:47:28 -08:00
..
batched_ops_stress.cc db_stress print hex key for MultiGet() inconsistency (#9324) 2021-12-20 23:29:43 -08:00
cf_consistency_stress.cc db_stress support tracking historical values (#8960) 2021-12-07 13:41:48 -08:00
CMakeLists.txt Stress test for RocksDB transactions (#8936) 2021-12-14 13:34:43 -08:00
db_stress_common.cc db_stress: db_stress fails on custom filesystems. (#9352) 2022-01-25 16:22:58 -08:00
db_stress_common.h Optimize db_stress setup phase (#9475) 2022-02-01 11:47:28 -08:00
db_stress_compaction_filter.h Prevent deadlock in db_stress with DbStressCompactionFilter (#8956) 2021-09-24 16:54:02 -07:00
db_stress_driver.cc db_stress begin tracking expected state after verification (#9470) 2022-01-31 13:35:32 -08:00
db_stress_driver.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_stress_env_wrapper.h Make the Env class Customizable (#9293) 2022-01-04 16:45:49 -08:00
db_stress_gflags.cc Move HDFS support to separate repo (#9170) 2022-01-24 20:23:54 -08:00
db_stress_listener.cc db_stress: db_stress fails on custom filesystems. (#9352) 2022-01-25 16:22:58 -08:00
db_stress_listener.h db_stress: db_stress fails on custom filesystems. (#9352) 2022-01-25 16:22:58 -08:00
db_stress_shared_state.cc Silence false alarms in db_stress fault injection (#6741) 2020-04-24 13:06:12 -07:00
db_stress_shared_state.h Optimize db_stress setup phase (#9475) 2022-02-01 11:47:28 -08:00
db_stress_stat.cc Fix Statistics in db_stress (#9260) 2021-12-07 16:24:22 -08:00
db_stress_stat.h Fix Statistics in db_stress (#9260) 2021-12-07 16:24:22 -08:00
db_stress_table_properties_collector.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
db_stress_test_base.cc db_stress begin tracking expected state after verification (#9470) 2022-01-31 13:35:32 -08:00
db_stress_test_base.h db_stress begin tracking expected state after verification (#9470) 2022-01-31 13:35:32 -08:00
db_stress_tool.cc db_stress: db_stress fails on custom filesystems. (#9352) 2022-01-25 16:22:58 -08:00
db_stress.cc Add (& fix) some simple source code checks (#8821) 2021-09-07 21:19:27 -07:00
expected_state.cc Optimize db_stress setup phase (#9475) 2022-02-01 11:47:28 -08:00
expected_state.h db_stress verify with lost unsynced operations (#8966) 2021-12-15 12:54:44 -08:00
multi_ops_txns_stress.cc Make the Env class Customizable (#9293) 2022-01-04 16:45:49 -08:00
multi_ops_txns_stress.h Stress test for RocksDB transactions (#8936) 2021-12-14 13:34:43 -08:00
no_batched_ops_stress.cc db_stress support tracking historical values (#8960) 2021-12-07 13:41:48 -08:00