First half of whitebox_crash_test to keep crashing the same DB

Summary: Currently, whitebox crash test is not really executed, because the DB is destroyed after each crash. With this fix, in the first half of the time, DB will keep opening the crashed DB and continue from there.

Test Plan: "make whitebox_crash_test" and see the same DB keeps crashing and being reopened.

Reviewers: IslamAbdelRahman, yhchiang, rven, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D43503

tools/db_crashtest2.py

@@ -60,7 +60,9 @@ def main(argv):
                   "-b <write_buffer_size>\n"
             sys.exit(2)
 
-    exit_time = time.time() + duration
+    cur_time = time.time()
+    exit_time = cur_time + duration
+    half_time = cur_time + duration / 2
 
     print "Running whitebox-crash-test with \ntotal-duration=" + str(duration) \
           + "\nthreads=" + str(threads) + "\nops_per_thread=" \
@@ -213,9 +215,13 @@ def main(argv):
         if (stdoutdata.find('fail') >= 0):
             print "TEST FAILED. Output has 'fail'!!!\n"
             sys.exit(2)
-        # we need to clean up after ourselves -- only do this on test success
-        shutil.rmtree(dbname, True)
 
+        # First half of the duration, keep doing kill test. For the next half,
+        # try different modes.
+        if time.time() > half_time:
+            # we need to clean up after ourselves -- only do this on test
+            # success
+            shutil.rmtree(dbname, True)
             check_mode = (check_mode + 1) % total_check_mode
 
         time.sleep(1)  # time to stabilize after a kill