Fix IOError on WAL write doesn't propagate to write group follower

Summary:
This is a simpler version of #3097 by removing all unrelated changes.

Fixing the bug where concurrent writes may get Status::OK while it actually gets IOError on WAL write. This happens when multiple writes form a write batch group, and the leader get an IOError while writing to WAL. The leader failed to pass the error to followers in the group, and the followers end up returning Status::OK() while actually writing nothing. The bug only affect writes in a batch group. Future writes after the batch group will correctly return immediately with the IOError.
Closes https://github.com/facebook/rocksdb/pull/3201

Differential Revision: D6421644

Pulled By: yiwu-arbug

fbshipit-source-id: 1c2a455c5b73f6842423785eb8a9dbfbb191dc0e

CMakeLists.txt

@@ -477,12 +477,12 @@ set(SOURCES
         utilities/backupable/backupable_db.cc
         utilities/blob_db/blob_db.cc
         utilities/blob_db/blob_db_impl.cc
-        utilities/blob_db/blob_db_options_impl.cc
         utilities/blob_db/blob_dump_tool.cc
         utilities/blob_db/blob_file.cc
         utilities/blob_db/blob_log_reader.cc
         utilities/blob_db/blob_log_writer.cc
         utilities/blob_db/blob_log_format.cc
+        utilities/blob_db/ttl_extractor.cc
         utilities/checkpoint/checkpoint_impl.cc
         utilities/col_buf_decoder.cc
         utilities/col_buf_encoder.cc

CONTRIBUTING.md

@@ -12,8 +12,3 @@ Complete your CLA here: <https://code.facebook.com/cla>
 
 If you prefer to sign a paper copy, we can send you a PDF.  Send us an 
 e-mail or create a new github issue to request the CLA in PDF format.
-
-## License
-
-By contributing to RocksDB, you agree that your contributions will be
-licensed under the [BSD License](LICENSE).

HISTORY.md

@@ -1,4 +1,27 @@
 # Rocksdb Change Log
+### New Features
+* Upon snapshot release, recompact bottommost files containing deleted/overwritten keys that previously could not be dropped due to the snapshot. This alleviates space-amp caused by long-held snapshots.
+* Support lower bound on iterators specified via `ReadOptions::iterate_lower_bound`.
+### Bug Fixes
+* Fix IOError on WAL write doesn't propagate to write group follower
+
+## 5.7.4 (08/31/2017)
+No significant changes.
+
+## 5.7.3 (08/29/2017)
+### Bug Fixes
+* Fix transient reappearance of keys covered by range deletions when memtable prefix bloom filter is enabled.
+* Fix potentially wrong file smallest key when range deletions separated by snapshot are written together.
+
+## 5.7.2 (08/15/2017)
+### Bug Fixes
+* Fix incorrect dropping of deletions issue with FIFO compaction.
+* Fix LITE build compiler error with missing abort().
+
+## 5.7.1 (08/13/2017)
+### Bug Fixes
+* Fix incorrect dropping of deletions during intra-L0 compaction.
+
 ## 5.7.0 (07/13/2017)
 ### Public API Change
 * DB property "rocksdb.sstables" now prints keys in hex form.
@@ -8,6 +31,8 @@
 * RateLimiter support for throttling background reads, or throttling the sum of background reads and writes. This can give more predictable I/O usage when compaction reads more data than it writes, e.g., due to lots of deletions.
 * [Experimental] FIFO compaction with TTL support. It can be enabled by setting CompactionOptionsFIFO.ttl > 0.
 * Introduce `EventListener::OnBackgroundError()` callback. Users can implement it to be notified of errors causing the DB to enter read-only mode, and optionally override them.
+* Partitioned Index/Filters exiting the experimental mode. To enable partitioned indexes set index_type to kTwoLevelIndexSearch and to further enable partitioned filters set partition_filters to true. To configure the partition size set metadata_block_size.
+
 
 ### Bug Fixes
 * Fix discarding empty compaction output files when `DeleteRange()` is used together with subcompactions.

LICENSE.Apache

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

LICENSE.leveldb

@@ -1,10 +1,4 @@
-BSD License
+This contains code that is from LevelDB, and that code is under the following license:
-
-For rocksdb software
-
-Copyright (c) 2011-present, Facebook, Inc.
-All rights reserved.
----------------------------------------------------------------------
 
 Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 

PATENTS

@@ -1,33 +0,0 @@
-Additional Grant of Patent Rights Version 2
-
-"Software" means the RocksDB software distributed by Facebook, Inc.
-
-Facebook, Inc. ("Facebook") hereby grants to each recipient of the Software
-("you") a perpetual, worldwide, royalty-free, non-exclusive, irrevocable
-(subject to the termination provision below) license under any Necessary
-Claims, to make, have made, use, sell, offer to sell, import, and otherwise
-transfer the Software. For avoidance of doubt, no license is granted under
-Facebook’s rights in any patent claims that are infringed by (i) modifications
-to the Software made by you or any third party or (ii) the Software in
-combination with any software or other technology.
-
-The license granted hereunder will terminate, automatically and without notice,
-if you (or any of your subsidiaries, corporate affiliates or agents) initiate
-directly or indirectly, or take a direct financial interest in, any Patent
-Assertion: (i) against Facebook or any of its subsidiaries or corporate
-affiliates, (ii) against any party if such Patent Assertion arises in whole or
-in part from any software, technology, product or service of Facebook or any of
-its subsidiaries or corporate affiliates, or (iii) against any party relating
-to the Software. Notwithstanding the foregoing, if Facebook or any of its
-subsidiaries or corporate affiliates files a lawsuit alleging patent
-infringement against you in the first instance, and you respond by filing a
-patent infringement counterclaim in that lawsuit against that party that is
-unrelated to the Software, the license granted hereunder will not terminate
-under section (i) of this paragraph due to such counterclaim.
-
-A "Necessary Claim" is a claim of a patent owned by Facebook that is
-necessarily infringed by the Software standing alone.
-
-A "Patent Assertion" is any lawsuit or other action alleging direct, indirect,
-or contributory infringement or inducement to infringe any patent, including a
-cross-claim or counterclaim.

TARGETS

@@ -207,11 +207,11 @@ cpp_library(
       "utilities/backupable/backupable_db.cc",
       "utilities/blob_db/blob_db.cc",
       "utilities/blob_db/blob_db_impl.cc",
-      "utilities/blob_db/blob_db_options_impl.cc",
       "utilities/blob_db/blob_file.cc",
       "utilities/blob_db/blob_log_reader.cc",
       "utilities/blob_db/blob_log_writer.cc",
       "utilities/blob_db/blob_log_format.cc",
+      "utilities/blob_db/ttl_extractor.cc",
       "utilities/checkpoint/checkpoint_impl.cc",
       "utilities/compaction_filters/remove_emptyvalue_compactionfilter.cc",
       "utilities/convenience/info_log_finder.cc",

build_tools/error_filter.py

@@ -1,7 +1,7 @@
-# Copyright (c) 2016-present, Facebook, Inc.  All rights reserved.
+#  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-# This source code is licensed under the BSD-style license found in the
+#  This source code is licensed under both the GPLv2 (found in the
-# LICENSE file in the root directory of this source tree. An additional grant
+#  COPYING file in the root directory) and Apache 2.0 License
-# of patent rights can be found in the PATENTS file in the same directory.
+#  (found in the LICENSE.Apache file in the root directory).
 
 '''Filter for error messages in test output:
     - Receives merged stdout/stderr from test on stdin

cache/cache_bench.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef __STDC_FORMAT_MACROS
 #define __STDC_FORMAT_MACROS

cache/cache_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

cache/clock_cache.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

cache/clock_cache.h

@@ -1,7 +1,7 @@
 // Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-// This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-// LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-// of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

cache/lru_cache.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

cache/lru_cache.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

cache/lru_cache_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "cache/lru_cache.h"
 

cache/sharded_cache.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

cache/sharded_cache.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/builder.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/builder.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

db/c.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/column_family.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/column_family.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/column_family_test.cc

@@ -1,8 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in
+//  This source code is licensed under both the GPLv2 (found in the
-//  the LICENSE file in the root directory of this source tree. An
+//  COPYING file in the root directory) and Apache 2.0 License
-//  additional grant of patent rights can be found in the PATENTS file
+//  (found in the LICENSE.Apache file in the root directory).
-//  in the same directory.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/compact_files_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef ROCKSDB_LITE
 

db/compacted_db_impl.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef ROCKSDB_LITE
 #include "db/compacted_db_impl.h"

db/compacted_db_impl.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #pragma once
 #ifndef ROCKSDB_LITE

db/compaction.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
@@ -88,35 +86,23 @@ void Compaction::GetBoundaryKeys(
 bool Compaction::IsBottommostLevel(
     int output_level, VersionStorageInfo* vstorage,
     const std::vector<CompactionInputFiles>& inputs) {
-  if (inputs[0].level == 0 &&
+  int output_l0_idx;
-      inputs[0].files.back() != vstorage->LevelFiles(0).back()) {
+  if (output_level == 0) {
-    return false;
+    output_l0_idx = 0;
+    for (const auto* file : vstorage->LevelFiles(0)) {
+      if (inputs[0].files.back() == file) {
+        break;
+      }
+      ++output_l0_idx;
+    }
+    assert(static_cast<size_t>(output_l0_idx) < vstorage->LevelFiles(0).size());
+  } else {
+    output_l0_idx = -1;
   }
-
   Slice smallest_key, largest_key;
   GetBoundaryKeys(vstorage, inputs, &smallest_key, &largest_key);
-
+  return !vstorage->RangeMightExistAfterSortedRun(smallest_key, largest_key,
-  // Checks whether there are files living beyond the output_level.
+                                                  output_level, output_l0_idx);
-  // If lower levels have files, it checks for overlap between files
-  // if the compaction process and those files.
-  // Bottomlevel optimizations can be made if there are no files in
-  // lower levels or if there is no overlap with the files in
-  // the lower levels.
-  for (int i = output_level + 1; i < vstorage->num_levels(); i++) {
-    // It is not the bottommost level if there are files in higher
-    // levels when the output level is 0 or if there are files in
-    // higher levels which overlap with files to be compacted.
-    // output_level == 0 means that we want it to be considered
-    // s the bottommost level only if the last file on the level
-    // is a part of the files to be compacted - this is verified by
-    // the first if condition in this function
-    if (vstorage->NumLevelFiles(i) > 0 &&
-        (output_level == 0 ||
-         vstorage->OverlapInLevel(i, &smallest_key, &largest_key))) {
-      return false;
-    }
-  }
-  return true;
 }
 
 // test function to validate the functionality of IsBottommostLevel()
@@ -285,28 +271,31 @@ bool Compaction::KeyNotExistsBeyondOutputLevel(
   assert(input_version_ != nullptr);
   assert(level_ptrs != nullptr);
   assert(level_ptrs->size() == static_cast<size_t>(number_levels_));
-  assert(cfd_->ioptions()->compaction_style != kCompactionStyleFIFO);
+  if (bottommost_level_) {
-  if (cfd_->ioptions()->compaction_style == kCompactionStyleUniversal) {
+    return true;
-    return bottommost_level_;
+  } else if (output_level_ != 0 &&
-  }
+             cfd_->ioptions()->compaction_style == kCompactionStyleLevel) {
-  // Maybe use binary search to find right entry instead of linear search?
+    // Maybe use binary search to find right entry instead of linear search?
-  const Comparator* user_cmp = cfd_->user_comparator();
+    const Comparator* user_cmp = cfd_->user_comparator();
-  for (int lvl = output_level_ + 1; lvl < number_levels_; lvl++) {
+    for (int lvl = output_level_ + 1; lvl < number_levels_; lvl++) {
-    const std::vector<FileMetaData*>& files = input_vstorage_->LevelFiles(lvl);
+      const std::vector<FileMetaData*>& files =
-    for (; level_ptrs->at(lvl) < files.size(); level_ptrs->at(lvl)++) {
+          input_vstorage_->LevelFiles(lvl);
-      auto* f = files[level_ptrs->at(lvl)];
+      for (; level_ptrs->at(lvl) < files.size(); level_ptrs->at(lvl)++) {
-      if (user_cmp->Compare(user_key, f->largest.user_key()) <= 0) {
+        auto* f = files[level_ptrs->at(lvl)];
-        // We've advanced far enough
+        if (user_cmp->Compare(user_key, f->largest.user_key()) <= 0) {
-        if (user_cmp->Compare(user_key, f->smallest.user_key()) >= 0) {
+          // We've advanced far enough
-          // Key falls in this file's range, so definitely
+          if (user_cmp->Compare(user_key, f->smallest.user_key()) >= 0) {
-          // exists beyond output level
+            // Key falls in this file's range, so definitely
-          return false;
+            // exists beyond output level
+            return false;
+          }
+          break;
         }
-        break;
       }
     }
+    return true;
   }
-  return true;
+  return false;
 }
 
 // Mark (or clear) each file that is being compacted

db/compaction.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/compaction_iteration_stats.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2016-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #pragma once
 

db/compaction_iterator.cc

@@ -1,11 +1,7 @@
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file. See the AUTHORS file for names of contributors.
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "db/compaction_iterator.h"
 #include "rocksdb/listener.h"

db/compaction_iterator.h

@@ -1,11 +1,7 @@
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file. See the AUTHORS file for names of contributors.
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 #pragma once
 
 #include <algorithm>

db/compaction_iterator_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "db/compaction_iterator.h"
 

db/compaction_job.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/compaction_job.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/compaction_job_stats_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/compaction_job_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef ROCKSDB_LITE
 

db/compaction_picker.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
@@ -403,7 +401,10 @@ bool CompactionPicker::SetupOtherInputs(
   assert(output_level_inputs->empty());
   const int input_level = inputs->level;
   const int output_level = output_level_inputs->level;
-  assert(input_level != output_level);
+  if (input_level == output_level) {
+    // no possibility of conflict
+    return true;
+  }
 
   // For now, we only support merging two levels, start level and output level.
   // We need to assert other levels are empty.
@@ -956,6 +957,9 @@ void CompactionPicker::UnregisterCompaction(Compaction* c) {
 
 bool LevelCompactionPicker::NeedsCompaction(
     const VersionStorageInfo* vstorage) const {
+  if (!vstorage->BottommostFilesMarkedForCompaction().empty()) {
+    return true;
+  }
   if (!vstorage->FilesMarkedForCompaction().empty()) {
     return true;
   }
@@ -1150,7 +1154,28 @@ void LevelCompactionBuilder::SetupInitialFiles() {
     is_manual_ = true;
     parent_index_ = base_index_ = -1;
     PickFilesMarkedForCompaction();
-    if (!start_level_inputs_.empty()) {
+    if (start_level_inputs_.empty()) {
+      size_t i;
+      for (i = 0; i < vstorage_->BottommostFilesMarkedForCompaction().size();
+           ++i) {
+        auto& level_and_file =
+            vstorage_->BottommostFilesMarkedForCompaction()[i];
+        assert(!level_and_file.second->being_compacted);
+        start_level_inputs_.level = output_level_ = start_level_ =
+            level_and_file.first;
+        start_level_inputs_.files = {level_and_file.second};
+        if (compaction_picker_->ExpandInputsToCleanCut(cf_name_, vstorage_,
+                                                       &start_level_inputs_)) {
+          break;
+        }
+      }
+      if (i == vstorage_->BottommostFilesMarkedForCompaction().size()) {
+        start_level_inputs_.clear();
+      } else {
+        assert(!start_level_inputs_.empty());
+        compaction_reason_ = CompactionReason::kBottommostFiles;
+      }
+    } else {
       compaction_reason_ = CompactionReason::kFilesMarkedForCompaction;
     }
   }

db/compaction_picker.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/compaction_picker_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "db/compaction_picker.h"
 #include <limits>

db/compaction_picker_universal.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/compaction_picker_universal.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/comparator_db_test.cc

@@ -1,11 +1,7 @@
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file. See the AUTHORS file for names of contributors.
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 #include <map>
 #include <string>
 

db/convenience.cc

@@ -1,13 +1,8 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
-// Copyright (c) 2012 Facebook.
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file.
 
 #ifndef ROCKSDB_LITE
 

db/corruption_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/cuckoo_table_db_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef ROCKSDB_LITE
 

db/db_basic_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_block_cache_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_bloom_filter_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_compaction_filter_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_compaction_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
@@ -2604,13 +2602,11 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
   // Files 6-9 are the longest span of available files for which
   // work-per-deleted-file decreases (see "score" row above).
   for (int i = 0; i < 10; ++i) {
-    for (int j = 0; j < 2; ++j) {
+    ASSERT_OK(Put(Key(0), ""));  // prevents trivial move
-      ASSERT_OK(Put(Key(0), ""));  // prevents trivial move
+    if (i == 5) {
-      if (i == 5) {
+      ASSERT_OK(Put(Key(i + 1), value + value));
-        ASSERT_OK(Put(Key(i + 1), value + value));
+    } else {
-      } else {
+      ASSERT_OK(Put(Key(i + 1), value));
-        ASSERT_OK(Put(Key(i + 1), value));
-      }
     }
     ASSERT_OK(Flush());
   }
@@ -2625,7 +2621,128 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
   ASSERT_EQ(2, level_to_files[0].size());
   ASSERT_GT(level_to_files[1].size(), 0);
   for (int i = 0; i < 2; ++i) {
-    ASSERT_GE(level_to_files[0][0].fd.file_size, 1 << 21);
+    ASSERT_GE(level_to_files[0][i].fd.file_size, 1 << 21);
+  }
+}
+
+TEST_P(DBCompactionTestWithParam, IntraL0CompactionDoesNotObsoleteDeletions) {
+  // regression test for issue #2722: L0->L0 compaction can resurrect deleted
+  // keys from older L0 files if L1+ files' key-ranges do not include the key.
+  Options options = CurrentOptions();
+  options.compression = kNoCompression;
+  options.level0_file_num_compaction_trigger = 5;
+  options.max_background_compactions = 2;
+  options.max_subcompactions = max_subcompactions_;
+  DestroyAndReopen(options);
+
+  const size_t kValueSize = 1 << 20;
+  Random rnd(301);
+  std::string value(RandomString(&rnd, kValueSize));
+
+  rocksdb::SyncPoint::GetInstance()->LoadDependency(
+      {{"LevelCompactionPicker::PickCompactionBySize:0",
+        "CompactionJob::Run():Start"}});
+  rocksdb::SyncPoint::GetInstance()->EnableProcessing();
+
+  // index:   0   1   2   3   4    5    6   7   8   9
+  // size:  1MB 1MB 1MB 1MB 1MB  1MB  1MB 1MB 1MB 1MB
+  // score:                     1.25 1.33 1.5 2.0 inf
+  //
+  // Files 0-4 will be included in an L0->L1 compaction.
+  //
+  // L0->L0 will be triggered since the sync points guarantee compaction to base
+  // level is still blocked when files 5-9 trigger another compaction. All files
+  // 5-9 are included in the L0->L0 due to work-per-deleted file decreasing.
+  //
+  // Put a key-value in files 0-4. Delete that key in files 5-9. Verify the
+  // L0->L0 preserves the deletion such that the key remains deleted.
+  for (int i = 0; i < 10; ++i) {
+    // key 0 serves both to prevent trivial move and as the key we want to
+    // verify is not resurrected by L0->L0 compaction.
+    if (i < 5) {
+      ASSERT_OK(Put(Key(0), ""));
+    } else {
+      ASSERT_OK(Delete(Key(0)));
+    }
+    ASSERT_OK(Put(Key(i + 1), value));
+    ASSERT_OK(Flush());
+  }
+  dbfull()->TEST_WaitForCompact();
+  rocksdb::SyncPoint::GetInstance()->DisableProcessing();
+
+  std::vector<std::vector<FileMetaData>> level_to_files;
+  dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
+                                  &level_to_files);
+  ASSERT_GE(level_to_files.size(), 2);  // at least L0 and L1
+  // L0 has a single output file from L0->L0
+  ASSERT_EQ(1, level_to_files[0].size());
+  ASSERT_GT(level_to_files[1].size(), 0);
+  ASSERT_GE(level_to_files[0][0].fd.file_size, 1 << 22);
+
+  ReadOptions roptions;
+  std::string result;
+  ASSERT_TRUE(db_->Get(roptions, Key(0), &result).IsNotFound());
+}
+
+TEST_F(DBCompactionTest, CompactBottomLevelFilesWithDeletions) {
+  // bottom-level files may contain deletions due to snapshots protecting the
+  // deleted keys. Once the snapshot is released, we should see files with many
+  // such deletions undergo single-file compactions.
+  const int kNumKeysPerFile = 1024;
+  const int kNumLevelFiles = 4;
+  const int kValueSize = 128;
+  Options options = CurrentOptions();
+  options.compression = kNoCompression;
+  options.level0_file_num_compaction_trigger = kNumLevelFiles;
+  // inflate it a bit to account for key/metadata overhead
+  options.target_file_size_base = 120 * kNumKeysPerFile * kValueSize / 100;
+  Reopen(options);
+
+  Random rnd(301);
+  const Snapshot* snapshot = nullptr;
+  for (int i = 0; i < kNumLevelFiles; ++i) {
+    for (int j = 0; j < kNumKeysPerFile; ++j) {
+      ASSERT_OK(
+          Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
+    }
+    if (i == kNumLevelFiles - 1) {
+      snapshot = db_->GetSnapshot();
+      // delete every other key after grabbing a snapshot, so these deletions
+      // and the keys they cover can't be dropped until after the snapshot is
+      // released.
+      for (int j = 0; j < kNumLevelFiles * kNumKeysPerFile; j += 2) {
+        ASSERT_OK(Delete(Key(j)));
+      }
+    }
+    Flush();
+    if (i < kNumLevelFiles - 1) {
+      ASSERT_EQ(i + 1, NumTableFilesAtLevel(0));
+    }
+  }
+  dbfull()->TEST_WaitForCompact();
+  ASSERT_EQ(kNumLevelFiles, NumTableFilesAtLevel(1));
+
+  std::vector<LiveFileMetaData> pre_release_metadata, post_release_metadata;
+  db_->GetLiveFilesMetaData(&pre_release_metadata);
+  // just need to bump seqnum so ReleaseSnapshot knows the newest key in the SST
+  // files does not need to be preserved in case of a future snapshot.
+  ASSERT_OK(Put(Key(0), "val"));
+  // release snapshot and wait for compactions to finish. Single-file
+  // compactions should be triggered, which reduce the size of each bottom-level
+  // file without changing file count.
+  db_->ReleaseSnapshot(snapshot);
+  dbfull()->TEST_WaitForCompact();
+  db_->GetLiveFilesMetaData(&post_release_metadata);
+  ASSERT_EQ(pre_release_metadata.size(), post_release_metadata.size());
+
+  for (size_t i = 0; i < pre_release_metadata.size(); ++i) {
+    const auto& pre_file = pre_release_metadata[i];
+    const auto& post_file = post_release_metadata[i];
+    ASSERT_EQ(1, pre_file.level);
+    ASSERT_EQ(1, post_file.level);
+    // each file is smaller than it was before as it was rewritten without
+    // deletion markers/deleted keys.
+    ASSERT_LT(post_file.size, pre_file.size);
   }
 }
 

db/db_dynamic_level_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_filesnapshot.cc

@@ -1,13 +1,8 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
-// Copyright (c) 2012 Facebook.
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file.
 
 #ifndef ROCKSDB_LITE
 

db/db_flush_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_impl.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
@@ -1574,6 +1572,22 @@ void DBImpl::ReleaseSnapshot(const Snapshot* s) {
   {
     InstrumentedMutexLock l(&mutex_);
     snapshots_.Delete(casted_s);
+    uint64_t oldest_snapshot;
+    if (snapshots_.empty()) {
+      oldest_snapshot = versions_->LastSequence();
+    } else {
+      oldest_snapshot = snapshots_.oldest()->number_;
+    }
+    for (auto* cfd : *versions_->GetColumnFamilySet()) {
+      cfd->current()->storage_info()->UpdateOldestSnapshot(oldest_snapshot);
+      if (!cfd->current()
+               ->storage_info()
+               ->BottommostFilesMarkedForCompaction()
+               .empty()) {
+        SchedulePendingCompaction(cfd);
+        MaybeScheduleFlushOrCompaction();
+      }
+    }
   }
   delete casted_s;
 }

db/db_impl.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_impl_compaction_flush.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_impl_debug.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_impl_experimental.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_impl_files.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_impl_open.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_impl_readonly.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "db/db_impl_readonly.h"
 

db/db_impl_readonly.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #pragma once
 

db/db_impl_write.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
@@ -232,7 +230,6 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
                             need_log_dir_sync, last_sequence + 1);
       }
     } else {
-      assert(!need_log_sync && !need_log_dir_sync);
       if (status.ok() && !write_options.disableWAL) {
         PERF_TIMER_GUARD(write_wal_time);
         // LastToBeWrittenSequence is increased inside WriteToWAL under
@@ -316,7 +313,7 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
       versions_->SetLastSequence(last_sequence);
     }
     MemTableInsertStatusCheck(w.status);
-    write_thread_.ExitAsBatchGroupLeader(write_group, w.status);
+    write_thread_.ExitAsBatchGroupLeader(write_group, status);
   }
 
   if (status.ok()) {
@@ -526,7 +523,7 @@ Status DBImpl::WriteImplWALOnly(const WriteOptions& write_options,
   if (!w.CallbackFailed()) {
     WriteCallbackStatusCheck(status);
   }
-  nonmem_write_thread_.ExitAsBatchGroupLeader(write_group, w.status);
+  nonmem_write_thread_.ExitAsBatchGroupLeader(write_group, status);
   if (status.ok()) {
     status = w.FinalStatus();
   }

db/db_info_dumper.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef __STDC_FORMAT_MACROS
 #define __STDC_FORMAT_MACROS

db/db_info_dumper.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 #pragma once
 
 #include <string>

db/db_inplace_update_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_io_failure_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_iter.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
@@ -119,6 +117,7 @@ class DBIter: public Iterator {
         current_entry_is_merged_(false),
         statistics_(cf_options.statistics),
         version_number_(version_number),
+        iterate_lower_bound_(read_options.iterate_lower_bound),
         iterate_upper_bound_(read_options.iterate_upper_bound),
         prefix_same_as_start_(read_options.prefix_same_as_start),
         pin_thru_lifetime_(read_options.pin_data),
@@ -277,6 +276,7 @@ class DBIter: public Iterator {
   uint64_t max_skippable_internal_keys_;
   uint64_t num_internal_keys_skipped_;
   uint64_t version_number_;
+  const Slice* iterate_lower_bound_;
   const Slice* iterate_upper_bound_;
   IterKey prefix_start_buf_;
   Slice prefix_start_key_;
@@ -680,6 +680,22 @@ void DBIter::PrevInternal() {
         ExtractUserKey(iter_->key()),
         !iter_->IsKeyPinned() || !pin_thru_lifetime_ /* copy */);
 
+    if (prefix_extractor_ && prefix_same_as_start_ &&
+        prefix_extractor_->Transform(saved_key_.GetUserKey())
+                .compare(prefix_start_key_) != 0) {
+      // Current key does not have the same prefix as start
+      valid_ = false;
+      return;
+    }
+
+    if (iterate_lower_bound_ != nullptr &&
+        user_comparator_->Compare(saved_key_.GetUserKey(),
+                                  *iterate_lower_bound_) < 0) {
+      // We've iterated earlier than the user-specified lower bound.
+      valid_ = false;
+      return;
+    }
+
     if (FindValueForCurrentKey()) {
       valid_ = true;
       if (!iter_->Valid()) {
@@ -689,11 +705,6 @@ void DBIter::PrevInternal() {
       if (user_comparator_->Equal(ikey.user_key, saved_key_.GetUserKey())) {
         FindPrevUserKey();
       }
-      if (valid_ && prefix_extractor_ && prefix_same_as_start_ &&
-          prefix_extractor_->Transform(saved_key_.GetUserKey())
-                  .compare(prefix_start_key_) != 0) {
-        valid_ = false;
-      }
       return;
     }
 
@@ -1077,6 +1088,10 @@ void DBIter::SeekToFirst() {
   if (prefix_extractor_ != nullptr) {
     max_skip_ = std::numeric_limits<uint64_t>::max();
   }
+  if (iterate_lower_bound_ != nullptr) {
+    Seek(*iterate_lower_bound_);
+    return;
+  }
   direction_ = kForward;
   ReleaseTempPinnedData();
   ResetInternalKeysSkippedCounter();

db/db_iter.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_iter_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include <string>
 #include <vector>
@@ -2798,6 +2796,110 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace8) {
 
   rocksdb::SyncPoint::GetInstance()->DisableProcessing();
 }
+
+
+TEST_F(DBIteratorTest, SeekPrefixTombstones) {
+  ReadOptions ro;
+  Options options;
+  options.prefix_extractor.reset(NewNoopTransform());
+  TestIterator* internal_iter = new TestIterator(BytewiseComparator());
+  internal_iter->AddDeletion("b");
+  internal_iter->AddDeletion("c");
+  internal_iter->AddDeletion("d");
+  internal_iter->AddDeletion("e");
+  internal_iter->AddDeletion("f");
+  internal_iter->AddDeletion("g");
+  internal_iter->Finish();
+
+  ro.prefix_same_as_start = true;
+  std::unique_ptr<Iterator> db_iter(NewDBIterator(
+      env_, ro, ImmutableCFOptions(options), BytewiseComparator(),
+      internal_iter, 10, options.max_sequential_skip_in_iterations,
+      0));
+
+  int skipped_keys = 0;
+
+  get_perf_context()->Reset();
+  db_iter->SeekForPrev("z");
+  skipped_keys =
+      static_cast<int>(get_perf_context()->internal_key_skipped_count);
+  ASSERT_EQ(skipped_keys, 0);
+
+  get_perf_context()->Reset();
+  db_iter->Seek("a");
+  skipped_keys =
+      static_cast<int>(get_perf_context()->internal_key_skipped_count);
+  ASSERT_EQ(skipped_keys, 0);
+}
+
+TEST_F(DBIteratorTest, SeekToFirstLowerBound) {
+  const int kNumKeys = 3;
+  for (int i = 0; i < kNumKeys + 2; ++i) {
+    // + 2 for two special cases: lower bound before and lower bound after the
+    // internal iterator's keys
+    TestIterator* internal_iter = new TestIterator(BytewiseComparator());
+    for (int j = 1; j <= kNumKeys; ++j) {
+      internal_iter->AddPut(std::to_string(j), "val");
+    }
+    internal_iter->Finish();
+
+    ReadOptions ro;
+    auto lower_bound_str = std::to_string(i);
+    Slice lower_bound(lower_bound_str);
+    ro.iterate_lower_bound = &lower_bound;
+    Options options;
+    std::unique_ptr<Iterator> db_iter(NewDBIterator(
+        env_, ro, ImmutableCFOptions(options), BytewiseComparator(),
+        internal_iter, 10 /* sequence */,
+        options.max_sequential_skip_in_iterations, 0));
+
+    db_iter->SeekToFirst();
+    if (i == kNumKeys + 1) {
+      // lower bound was beyond the last key
+      ASSERT_FALSE(db_iter->Valid());
+    } else {
+      ASSERT_TRUE(db_iter->Valid());
+      int expected;
+      if (i == 0) {
+        // lower bound was before the first key
+        expected = 1;
+      } else {
+        // lower bound was at the ith key
+        expected = i;
+      }
+      ASSERT_EQ(std::to_string(expected), db_iter->key().ToString());
+    }
+  }
+}
+
+TEST_F(DBIteratorTest, PrevLowerBound) {
+  const int kNumKeys = 3;
+  const int kLowerBound = 2;
+  TestIterator* internal_iter = new TestIterator(BytewiseComparator());
+  for (int j = 1; j <= kNumKeys; ++j) {
+    internal_iter->AddPut(std::to_string(j), "val");
+  }
+  internal_iter->Finish();
+
+  ReadOptions ro;
+  auto lower_bound_str = std::to_string(kLowerBound);
+  Slice lower_bound(lower_bound_str);
+  ro.iterate_lower_bound = &lower_bound;
+  Options options;
+  std::unique_ptr<Iterator> db_iter(NewDBIterator(
+      env_, ro, ImmutableCFOptions(options), BytewiseComparator(),
+      internal_iter, 10 /* sequence */,
+      options.max_sequential_skip_in_iterations, 0));
+
+  db_iter->SeekToLast();
+  for (int i = kNumKeys; i >= kLowerBound; --i) {
+    ASSERT_TRUE(db_iter->Valid());
+    ASSERT_EQ(std::to_string(i), db_iter->key().ToString());
+    db_iter->Prev();
+  }
+  ASSERT_FALSE(db_iter->Valid());
+}
+
 }  // namespace rocksdb
 
 int main(int argc, char** argv) {

db/db_iterator_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_log_iter_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_memtable_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include <memory>
 #include <string>

db/db_merge_operator_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 #include <string>
 #include <vector>
 

db/db_options_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_properties_test.cc

@@ -1,7 +1,7 @@
 // Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-// This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-// LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-// of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_range_del_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2016-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "db/db_test_util.h"
 #include "port/stack_trace.h"
@@ -870,6 +868,66 @@ TEST_F(DBRangeDelTest, SubcompactionHasEmptyDedicatedRangeDelFile) {
   db_->ReleaseSnapshot(snapshot);
 }
 
+TEST_F(DBRangeDelTest, MemtableBloomFilter) {
+  // regression test for #2743. the range delete tombstones in memtable should
+  // be added even when Get() skips searching due to its prefix bloom filter
+  const int kMemtableSize = 1 << 20;              // 1MB
+  const int kMemtablePrefixFilterSize = 1 << 13;  // 8KB
+  const int kNumKeys = 1000;
+  const int kPrefixLen = 8;
+  Options options = CurrentOptions();
+  options.memtable_prefix_bloom_size_ratio =
+      static_cast<double>(kMemtablePrefixFilterSize) / kMemtableSize;
+  options.prefix_extractor.reset(rocksdb::NewFixedPrefixTransform(kPrefixLen));
+  options.write_buffer_size = kMemtableSize;
+  Reopen(options);
+
+  for (int i = 0; i < kNumKeys; ++i) {
+    ASSERT_OK(Put(Key(i), "val"));
+  }
+  Flush();
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0),
+                             Key(kNumKeys)));
+  for (int i = 0; i < kNumKeys; ++i) {
+    std::string value;
+    ASSERT_TRUE(db_->Get(ReadOptions(), Key(i), &value).IsNotFound());
+  }
+}
+
+TEST_F(DBRangeDelTest, UnorderedTombstones) {
+  // Regression test for #2752. Range delete tombstones between
+  // different snapshot stripes are not stored in order, so the first
+  // tombstone of each snapshot stripe should be checked as a smallest
+  // candidate.
+  Options options = CurrentOptions();
+  DestroyAndReopen(options);
+
+  auto cf = db_->DefaultColumnFamily();
+
+  ASSERT_OK(db_->Put(WriteOptions(), cf, "a", "a"));
+  ASSERT_OK(db_->Flush(FlushOptions(), cf));
+  ASSERT_EQ(1, NumTableFilesAtLevel(0));
+  ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
+  ASSERT_EQ(1, NumTableFilesAtLevel(1));
+
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), cf, "b", "c"));
+  // Hold a snapshot to separate these two delete ranges.
+  auto snapshot = db_->GetSnapshot();
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), cf, "a", "b"));
+  ASSERT_OK(db_->Flush(FlushOptions(), cf));
+  db_->ReleaseSnapshot(snapshot);
+
+  std::vector<std::vector<FileMetaData>> files;
+  dbfull()->TEST_GetFilesMetaData(cf, &files);
+  ASSERT_EQ(1, files[0].size());
+  ASSERT_EQ("a", files[0][0].smallest.user_key());
+  ASSERT_EQ("c", files[0][0].largest.user_key());
+
+  std::string v;
+  auto s = db_->Get(ReadOptions(), "a", &v);
+  ASSERT_TRUE(s.IsNotFound());
+}
+
 #endif  // ROCKSDB_LITE
 
 }  // namespace rocksdb

db/db_sst_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_statistics_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include <string>
 

db/db_table_properties_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_tailing_iter_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
@@ -2808,6 +2806,46 @@ TEST_F(DBTest, FIFOCompactionTestWithCompaction) {
             options.compaction_options_fifo.max_table_files_size);
 }
 
+TEST_F(DBTest, FIFOCompactionStyleWithCompactionAndDelete) {
+  Options options;
+  options.compaction_style = kCompactionStyleFIFO;
+  options.write_buffer_size = 20 << 10;  // 20K
+  options.arena_block_size = 4096;
+  options.compaction_options_fifo.max_table_files_size = 1500 << 10;  // 1MB
+  options.compaction_options_fifo.allow_compaction = true;
+  options.level0_file_num_compaction_trigger = 3;
+  options.compression = kNoCompression;
+  options.create_if_missing = true;
+  options = CurrentOptions(options);
+  DestroyAndReopen(options);
+
+  Random rnd(301);
+  for (int i = 0; i < 3; i++) {
+    // Each file contains a different key which will be dropped later.
+    ASSERT_OK(Put("a" + ToString(i), RandomString(&rnd, 500)));
+    ASSERT_OK(Put("key" + ToString(i), ""));
+    ASSERT_OK(Put("z" + ToString(i), RandomString(&rnd, 500)));
+    Flush();
+    ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  }
+  ASSERT_EQ(NumTableFilesAtLevel(0), 1);
+  for (int i = 0; i < 3; i++) {
+    ASSERT_EQ("", Get("key" + ToString(i)));
+  }
+  for (int i = 0; i < 3; i++) {
+    // Each file contains a different key which will be dropped later.
+    ASSERT_OK(Put("a" + ToString(i), RandomString(&rnd, 500)));
+    ASSERT_OK(Delete("key" + ToString(i)));
+    ASSERT_OK(Put("z" + ToString(i), RandomString(&rnd, 500)));
+    Flush();
+    ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  }
+  ASSERT_EQ(NumTableFilesAtLevel(0), 2);
+  for (int i = 0; i < 3; i++) {
+    ASSERT_EQ("NOT_FOUND", Get("key" + ToString(i)));
+  }
+}
+
 // Check that FIFO-with-TTL is not supported with max_open_files != -1.
 TEST_F(DBTest, FIFOCompactionWithTTLAndMaxOpenFilesTest) {
   Options options;

db/db_test2.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_test_util.cc

@@ -1,7 +1,7 @@
 // Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-// This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-// LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-// of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_test_util.h

@@ -1,7 +1,7 @@
 // Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-// This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-// LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-// of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_universal_compaction_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_wal_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/db_write_test.cc

@@ -1,16 +1,19 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
+#include <atomic>
 #include <memory>
 #include <thread>
 #include <vector>
 #include "db/db_test_util.h"
 #include "db/write_batch_internal.h"
+#include "db/write_thread.h"
+#include "port/port.h"
 #include "port/stack_trace.h"
+#include "util/fault_injection_test_env.h"
+#include "util/string_util.h"
 #include "util/sync_point.h"
 
 namespace rocksdb {
@@ -20,7 +23,9 @@ class DBWriteTest : public DBTestBase, public testing::WithParamInterface<int> {
  public:
   DBWriteTest() : DBTestBase("/db_write_test") {}
 
-  void Open() { DBTestBase::Reopen(GetOptions(GetParam())); }
+  Options GetOptions() { return DBTestBase::GetOptions(GetParam()); }
+
+  void Open() { DBTestBase::Reopen(GetOptions()); }
 };
 
 // Sequence number should be return through input write batch.
@@ -69,6 +74,47 @@ TEST_P(DBWriteTest, ReturnSeuqneceNumberMultiThreaded) {
   }
 }
 
+TEST_P(DBWriteTest, IOErrorOnWALWritePropagateToWriteThreadFollower) {
+  constexpr int kNumThreads = 5;
+  std::unique_ptr<FaultInjectionTestEnv> mock_env(
+      new FaultInjectionTestEnv(Env::Default()));
+  Options options = GetOptions();
+  options.env = mock_env.get();
+  Reopen(options);
+  std::atomic<int> ready_count{0};
+  std::atomic<int> leader_count{0};
+  std::vector<port::Thread> threads;
+  mock_env->SetFilesystemActive(false);
+  // Wait until all threads linked to write threads, to make sure
+  // all threads join the same batch group.
+  SyncPoint::GetInstance()->SetCallBack(
+      "WriteThread::JoinBatchGroup:Wait", [&](void* arg) {
+        ready_count++;
+        auto* w = reinterpret_cast<WriteThread::Writer*>(arg);
+        if (w->state == WriteThread::STATE_GROUP_LEADER) {
+          leader_count++;
+          while (ready_count < kNumThreads) {
+            // busy waiting
+          }
+        }
+      });
+  SyncPoint::GetInstance()->EnableProcessing();
+  for (int i = 0; i < kNumThreads; i++) {
+    threads.push_back(port::Thread(
+        [&](int index) {
+          // All threads should fail.
+          ASSERT_FALSE(Put("key" + ToString(index), "value").ok());
+        },
+        i));
+  }
+  for (int i = 0; i < kNumThreads; i++) {
+    threads[i].join();
+  }
+  ASSERT_EQ(1, leader_count);
+  // Close before mock_env destruct.
+  Close();
+}
+
 INSTANTIATE_TEST_CASE_P(DBWriteTestInstance, DBWriteTest,
                         testing::Values(DBTestBase::kDefault,
                                         DBTestBase::kConcurrentWALWrites,

db/dbformat.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/dbformat.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/dbformat_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/deletefile_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be

db/event_helpers.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "db/event_helpers.h"
 

db/event_helpers.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 #pragma once
 
 #include <memory>

db/experimental.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include "rocksdb/experimental.h"
 

db/external_sst_file_basic_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #include <functional>
 

db/external_sst_file_ingestion_job.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef ROCKSDB_LITE
 

db/external_sst_file_ingestion_job.h

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #pragma once
 #include <string>

db/external_sst_file_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 
 #ifndef ROCKSDB_LITE
 

db/fault_injection_test.cc

@@ -1,9 +1,7 @@
 //  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
+//  This source code is licensed under both the GPLv2 (found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
+//  COPYING file in the root directory) and Apache 2.0 License
-//  of patent rights can be found in the PATENTS file in the same directory.
+//  (found in the LICENSE.Apache file in the root directory).
-//  This source code is also licensed under the GPLv2 license found in the
-//  COPYING file in the root directory of this source tree.
 //
 // Copyright 2014 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be