crash_test to trigger some less frequent crash point more frequently

Summary: crash_test still has a very low chance to hit some crash point. Have another mode for covering them more likely.

Test Plan: Run crash_test and see db_stress is called with expected prameters.

Reviewers: kradhakrishnan, igor, anthony, rven, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49473

tools/db_crashtest.py

@@ -245,20 +245,29 @@ def whitebox_crash_main(args):
                 # use large ops per thread since we will kill it anyway
                 "ops_per_thread": 100 * cmd_params['ops_per_thread'],
             }
-            # run with kill_random_test
+            # run with kill_random_test, with three modes.
+            # Mode 0 covers all kill points. Mode 1 covers less kill points but
+            # increases change of triggering them. Mode 2 covers even less
+            # frequent kill points and further increases triggering change.
             if kill_mode == 0:
                 additional_opts.update({
                     "kill_random_test": kill_random_test,
                 })
             elif kill_mode == 1:
                 additional_opts.update({
-                    "kill_random_test": (kill_random_test / 3 + 1),
+                    "kill_random_test": (kill_random_test / 2 + 1),
                     "kill_prefix_blacklist": "WritableFileWriter::Append,"
                     + "WritableFileWriter::WriteBuffered",
                 })
-
-            # Run kill mode 0 and 1 by turn.
-            kill_mode = (kill_mode + 1) % 2
+            elif kill_mode == 2:
+                additional_opts.update({
+                    "kill_random_test": (kill_random_test / 4 + 1),
+                    "kill_prefix_blacklist": "WritableFileWriter::Append,"
+                    "WritableFileWriter::WriteBuffered,"
+                    "PosixMmapFile::Allocate,WritableFileWriter::Flush",
+                })
+            # Run kill mode 0, 1 and 2 by turn.
+            kill_mode = (kill_mode + 1) % 3
         elif check_mode == 1:
             # normal run with universal compaction mode
             additional_opts = {

util/sync_point.cc

@@ -25,6 +25,11 @@ void TestKillRandom(std::string kill_point, int odds,
   Random r((uint32_t)curtime);
 
   assert(odds > 0);
+  if (odds % 7 == 0) {
+    // class Rarndom uses multiplier 16807, which is 7^5. If odds are
+    // multiplier of 7, the first random value might have limited values.
+    odds++;
+  }
   bool crash = r.OneIn(odds);
   if (crash) {
     port::Crash(srcfile, srcline);