Commit Graph

25 Commits

Author SHA1 Message Date
Aaron Gao
7e5fac2c34 remove test dir before exit when current regression is running
Summary:
clean up the current test dir if the last regression test is still running.
Closes https://github.com/facebook/rocksdb/pull/2401

Differential Revision: D5177882

Pulled By: lightmark

fbshipit-source-id: 91d899fcc2bde841948eae71af8584d4bdb35468
2017-06-02 17:26:19 -07:00
Aaron Gao
8721996065 add checkpoint support for single db in regression test
Summary:
For level_compaction_style regression test.
Closes https://github.com/facebook/rocksdb/pull/2397

Differential Revision: D5168545

Pulled By: lightmark

fbshipit-source-id: 195e4d84917e7c261d9f4fbe9aee5d104c9cb9a2
2017-06-01 15:56:59 -07:00
Aaron Gao
9b3ed83506 fix regression test
Summary:
fix regression test by not reporting stats when building db
Closes https://github.com/facebook/rocksdb/pull/2390

Differential Revision: D5159909

Pulled By: lightmark

fbshipit-source-id: c3f4b9deb9c6799ff84207fd341c529144f8158d
2017-05-31 15:41:45 -07:00
Aaron Gao
cbc821c25b change regression rebuild to one level
Summary:
abandon fillseqdeterministic
test locally
Closes https://github.com/facebook/rocksdb/pull/2290

Differential Revision: D5151867

Pulled By: lightmark

fbshipit-source-id: 4c8a24cc937212ffb5ceb9bfaf7288eb8726d0c1
2017-05-30 16:41:21 -07:00
Maysam Yabandeh
8f61967881 Add cpu usage to regression benchmarks (4th attempt)
Summary:
Tested by running it on a remote machine.

I could not run it on the particular remote machine which has a different location for time command since it is busy and the script does not allow concurrent runs. So I tested it by hacking the script and replacing the command with "\$(hostname)" and confirmed that the scripts prints out the host name of the remote machine.
Closes https://github.com/facebook/rocksdb/pull/2181

Differential Revision: D4921654

Pulled By: maysamyabandeh

fbshipit-source-id: 8abb5ea9f7234f3c50a749576ccbb47ff605beb9
2017-04-20 09:31:09 -07:00
Maysam Yabandeh
927bbab25c Revert "Add cpu usage to regression benchmarks (3rd attempt)"
Summary:
This reverts commit 476e80be80.
Closes https://github.com/facebook/rocksdb/pull/2177

Differential Revision: D4914830

Pulled By: maysamyabandeh

fbshipit-source-id: 039299348ceb325aa721eb35e3a26e890f84ee74
2017-04-19 11:11:12 -07:00
Maysam Yabandeh
476e80be80 Add cpu usage to regression benchmarks (3rd attempt)
Summary:
Tested by running rocks/tools/debug_regression_test.sh and verifying the local output:
```
 cat local/rocks_regression_tests/OPTIONS-viewstate-66-1262-5000/2017-04-13-11-34-51/SUMMARY.csv
                               commit id,                benchmark,                     user@host,num-dbs,key-range,key-size,value-size,compress-rate,ops-per-thread,num-threads,  cache-size,flushes,compactions,ops-per-s,       p50,       p75,       p99,     p99.9,    p99.99,debug,real-sec,user-sec,sys-sec
d2dce5611a,               readrandom,                root@localhost,     12,     5000,      66,      1262,           50,           312,         16,  1073741824,      4,         16,   138458,      9380,     11530,     55200,  16803200,  32504000,    1,    0,    0,    0
d2dce5611a,         readwhilewriting,                root@localhost,     12,     5000,      66,      1262,           50,           312,         16,  1073741824,      4,         16,   104511,
Closes https://github.com/facebook/rocksdb/pull/2157

Differential Revision: D4909238

Pulled By: maysamyabandeh

fbshipit-source-id: dc7bb8569c3c33b9f7c4ba47a757b24d27bb3b31
2017-04-18 17:11:25 -07:00
Maysam Yabandeh
6a8d5c015b Revert "Report cpu usage using time command"
Summary:
This reverts commit 97ec8a1349.
Closes https://github.com/facebook/rocksdb/pull/2136

Differential Revision: D4870610

Pulled By: maysamyabandeh

fbshipit-source-id: cdbfba135b065562f38f704f350a9a4e63a9a122
2017-04-11 13:57:58 -07:00
Maysam Yabandeh
97ec8a1349 Report cpu usage using time command
Summary:
Run the time command before regression tests, parse the output, and add the numbers to the report.
Closes https://github.com/facebook/rocksdb/pull/2101

Differential Revision: D4862781

Pulled By: maysamyabandeh

fbshipit-source-id: 4a81caa5d14187d67093aad154c8f0ad56aba901
2017-04-10 14:59:31 -07:00
Aaron Gao
af256eb2b7 build db every monday
Summary:
Rebuilding regression db at the first run every monday.
Closes https://github.com/facebook/rocksdb/pull/2093

Differential Revision: D4836961

Pulled By: lightmark

fbshipit-source-id: 22d6c25
2017-04-05 12:24:21 -07:00
Aaron Gao
0537f515c2 fix run_remote with strong quoting
Summary:
add \' pair to avoid bash expanding before run remotely
Closes https://github.com/facebook/rocksdb/pull/2079

Differential Revision: D4821342

Pulled By: lightmark

fbshipit-source-id: 418ba41
2017-04-03 13:09:12 -07:00
Aaron Gao
bd7d13835e test remote instead run remote in regression test
Summary:
avoid exit when $? != 0
Closes https://github.com/facebook/rocksdb/pull/2076

Differential Revision: D4815655

Pulled By: lightmark

fbshipit-source-id: 829299e
2017-03-31 18:54:11 -07:00
Aaron Gao
c81a805fe6 test db existence on the remote host
Summary:
Fix the bug that previous test existence locally
Closes https://github.com/facebook/rocksdb/pull/2074

Differential Revision: D4813631

Pulled By: lightmark

fbshipit-source-id: eceb0d9
2017-03-31 14:54:17 -07:00
Aaron Gao
5fc1e6765a add -rf when remove db in regression test
Summary:
force remove db dir when rebuilding
Closes https://github.com/facebook/rocksdb/pull/2067

Differential Revision: D4811926

Pulled By: lightmark

fbshipit-source-id: ab068a2
2017-03-31 12:10:23 -07:00
Aaron Gao
da175f7ecc exit with code 2 when there is already a db_bench running in regression test
Summary: Closes https://github.com/facebook/rocksdb/pull/2063

Differential Revision: D4805507

Pulled By: lightmark

fbshipit-source-id: 59ccb18
2017-03-30 14:09:11 -07:00
Aaron Gao
f3607640a6 add ldb build to regression test
Summary:
add ldb to regression test in order to enable reuse db by creating checkpoint
Closes https://github.com/facebook/rocksdb/pull/2030

Differential Revision: D4800549

Pulled By: lightmark

fbshipit-source-id: d3a7325
2017-03-29 18:24:11 -07:00
Aaron Gao
93c68b642e change regression bash file with debug mode
Summary:
add debug mode for better debugging
refactor some regex
change micros/op to ops/sec
Closes https://github.com/facebook/rocksdb/pull/1999

Differential Revision: D4742806

Pulled By: lightmark

fbshipit-source-id: 0fe3ae6
2017-03-20 17:39:17 -07:00
Maysam Yabandeh
5dae019477 Revert "Report cpu usage using time command"
Summary:
This reverts commit d43adf21bb.

The patch has caused problems in regression tests. Will revert it for now until we figure how to debug the problems regression tests.
Closes https://github.com/facebook/rocksdb/pull/1975

Differential Revision: D4682880

Pulled By: maysamyabandeh

fbshipit-source-id: 84df83a
2017-03-09 11:09:13 -08:00
Maysam Yabandeh
d43adf21bb Report cpu usage using time command
Summary:
It augments the regression benchmarks with a time command, parses the output, and print them to the SUMMARY.csv file.

I tested a variation of the script locally. Any idea how to do run a test that also involves writing to scuba tables?
Closes https://github.com/facebook/rocksdb/pull/1967

Differential Revision: D4679470

Pulled By: maysamyabandeh

fbshipit-source-id: 44dac30
2017-03-08 17:54:11 -08:00
Yueh-Hsuan Chiang
f83cd64c0b Fix a bug that mistakenly disable regression_test.sh to update commit (#1415)
Summary:
Fix a bug that mistakenly disable regression_test.sh to update commit

Test Plan:
regression_test.sh
2016-10-21 17:26:24 -07:00
Aaron Gao
08616b4933 [db_bench] add filldeterministic (Universal+level compaction)
Summary:
in db_bench, we can dynamically create a rocksdb database that guarantees the shape of its LSM.
universal + level compaction
no fifo compaction
no multi db support

Test Plan:
./db_bench -benchmarks=fillseqdeterministic -compaction_style=1 -num_levels=3 --disable_auto_compactions -num=1000000 -value_size=1000
```
---------------------- LSM ---------------------
Level[0]: /000480.sst(size: 35060275 bytes)
Level[0]: /000479.sst(size: 70443197 bytes)
Level[0]: /000478.sst(size: 141600383 bytes)
Level[1]: /000341.sst - /000475.sst(total size: 284726629 bytes)
Level[2]: /000071.sst - /000340.sst(total size: 568649806 bytes)
fillseqdeterministic :      60.447 micros/op 16543 ops/sec;   16.0 MB/s
```

Reviewers: sdong, andrewkr, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D63111
2016-10-18 16:30:57 -07:00
Yueh-Hsuan Chiang
faa7eb3b99 Improve regression_test.sh
Summary:
This diff makes the following improvement in regression_test.sh:

1. Add NUM_OPS and DELETE_TEST_PATH to regression_test.sh:

  * NUM_OPS:  The number of operations that will be issued
    in EACH thread.
    Default: $NUM_KEYS / $NUM_THREADS

  * DELETE_TEST_PATH: If true, then the test directory
    will be deleted after the script ends.
    Default: 0
2. Add more information in SUMMARY.csv
3. Fix a bug in regression_test.sh where each thread in fillseq will all issue $NUM_KEYS writes.
4. Add --deletes in db_bench, which allows us to control the number of deletes instead of must using FLAGS_num.

Test Plan: run regression test with and without DELETE_TEST_PATH and NUM_OPS

Reviewers: yiwu, sdong, IslamAbdelRahman, gunnarku

Reviewed By: gunnarku

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D60039
2016-06-28 14:10:24 -07:00
Yueh-Hsuan Chiang
fda098461b Allow regression_test.sh to specify OPTIONS_FILE. Add header comments.
Summary:
This patch does the following improvement for regression_test.sh
* Allow regression_test.sh to specify OPTIONS_FILE.
* Add header comments that includes examples on how to run the script
  and introduce all configurable parameters.
* bug fix.

Test Plan: Run the example commands in the header comments of regression_test.sh

Reviewers: sdong, yiwu, gunnarku

Reviewed By: gunnarku

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59175
2016-06-06 22:57:46 -07:00
Yueh-Hsuan Chiang
8cf0f86d39 Allow regression test to run db_bench at a remost host
Summary:
This patch does the following improvement on the regression_test.sh
* allows db_bench being executed at a remost host while storing the
  benchmark results locally.
* kills all db_bench related processes before running db_bench
* better error handling.

Test Plan:
1. Run regression_test.sh both locally and remotely
2. Run multiple regression_test.sh at the same time and make sure
   i. Only one runs successfully.
   ii. The one that runs successfully will kill all other db_bench
       processes before it runs any benchmark.

Reviewers: sdong, yiwu, gunnarku

Reviewed By: gunnarku

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D58611
2016-06-02 02:53:39 -07:00
Yueh-Hsuan Chiang
fca5aa6fcc Initial script for the new regression test
Summary:
This diff includes an initial script running a set of benchmarks for
regression test.  The script does the following things:

  checkout the specified rocksdb commit (or origin/master as default)
  make clean && DEBUG_LEVEL=0 make db_bench
  setup test directories
  run set of benchmarks and store results

Currently, the script will run couple benchmarks, store all the benchmark
output, extract micros per op and percentile information for each benchmark
and store them in a single SUMMARY.csv file.  The SUMMARY.csv will make the
follow-up regression detection easier.

In addition, the current script only takes env arguments to set important
attributes of db_bench.  Will follow-up with a patch that allows db_bench
to construct options from an options file.

Test Plan:
NUM_KEYS=100 ./tools/regression_test.sh

  Sample SUMMARY.csv file:

                                     commit id,                      benchmark,  ms-per-op,        p50,        p75,        p99,      p99.9,     p99.99
      7e23ddf575890510e7d2fc7a79b31a1bbf317917,                        fillseq,      15.28,      54.66,      77.14,    5000.00,   17900.00,   18483.00
      7e23ddf575890510e7d2fc7a79b31a1bbf317917,                      overwrite,      13.54,      57.69,      86.39,    3000.00,   15600.00,   17013.00
      7e23ddf575890510e7d2fc7a79b31a1bbf317917,                     readrandom,       1.04,       0.80,       1.67,     293.33,     395.00,     504.00
      7e23ddf575890510e7d2fc7a79b31a1bbf317917,               readwhilewriting,       2.75,       1.01,       1.87,     200.00,     460.00,     485.00
      7e23ddf575890510e7d2fc7a79b31a1bbf317917,                   deleterandom,       3.64,      48.12,      70.09,     200.00,     336.67,     347.00
      7e23ddf575890510e7d2fc7a79b31a1bbf317917,                     seekrandom,      24.31,     391.87,     513.69,     872.73,     990.00,    1048.00
      7e23ddf575890510e7d2fc7a79b31a1bbf317917,         seekrandomwhilewriting,      14.02,     185.14,     294.15,     700.00,    1440.00,    1527.00

Reviewers: sdong, IslamAbdelRahman, kradhakrishnan, yiwu, andrewkr, gunnarku

Reviewed By: gunnarku

Subscribers: gunnarku, MarkCallaghan, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D57597
2016-05-09 13:32:57 -07:00