Makefile support to statically link external plugin code (#7918)

Summary:
Added support for detecting plugins linked in the "plugin/" directory and building them from our Makefile in a standardized way. See "plugin/README.md" for details. An example of a plugin that can be built in this way can be found in https://github.com/ajkr/dedupfs.

There will be more to do in terms of making this process more convenient and adding support for CMake.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7918

Test Plan: my own plugin (https://github.com/ajkr/dedupfs) and also heard this patch worked with ZenFS.

Reviewed By: pdillinger

Differential Revision: D26189969

Pulled By: ajkr

fbshipit-source-id: 6624d4357d0ffbaedb42f0d12a3fcb737c78f758
This commit is contained in:
Andrew Kryczka 2021-02-10 08:34:00 -08:00 committed by Facebook GitHub Bot
parent 65487da1f5
commit c16d5a4fda
4 changed files with 41 additions and 1 deletions

View File

@ -6,6 +6,7 @@
### New Features
* Add support for key-value integrity protection in live updates from the user buffers provided to `WriteBatch` through the write to RocksDB's in-memory update buffer (memtable). This is intended to detect some cases of in-memory data corruption, due to either software or hardware errors. Users can enable protection by constructing their `WriteBatch` with `protection_bytes_per_key == 8`.
* Add support for updating `full_history_ts_low` option in manual compaction, which is for old timestamp data GC.
* Add a mechanism for using Makefile to build external plugin code into the RocksDB libraries/binaries. This intends to simplify compatibility and distribution for plugins (e.g., special-purpose `FileSystem`s) whose source code resides outside the RocksDB repo. See "plugin/README.md" for developer details, and "PLUGINS.md" for a listing of available plugins.
### Bug Fixes
* Since 6.15.0, `TransactionDB` returns error `Status`es from calls to `DeleteRange()` and calls to `Write()` where the `WriteBatch` contains a range deletion. Previously such operations may have succeeded while not providing the expected transactional guarantees. There are certain cases where range deletion can still be used on such DBs; see the API doc on `TransactionDB::DeleteRange()` for details.

View File

@ -269,6 +269,12 @@ dummy := $(shell (export ROCKSDB_ROOT="$(CURDIR)"; \
# this file is generated by the previous line to set build flags and sources
include make_config.mk
ROCKSDB_PLUGIN_MKS = $(foreach plugin, $(ROCKSDB_PLUGINS), plugin/$(plugin)/*.mk)
include $(ROCKSDB_PLUGIN_MKS)
ROCKSDB_PLUGIN_SOURCES = $(foreach plugin, $(ROCKSDB_PLUGINS), $(foreach source, $($(plugin)_SOURCES), plugin/$(plugin)/$(source)))
ROCKSDB_PLUGIN_HEADERS = $(foreach plugin, $(ROCKSDB_PLUGINS), $(foreach header, $($(plugin)_HEADERS), plugin/$(plugin)/$(header)))
PLATFORM_LDFLAGS += $(foreach plugin, $(ROCKSDB_PLUGINS), $($(plugin)_LDFLAGS))
export JAVAC_ARGS
CLEAN_FILES += make_config.mk rocksdb.pc
@ -481,6 +487,7 @@ LDFLAGS += $(PLATFORM_LDFLAGS)
OBJ_DIR?=.
LIB_OBJECTS = $(patsubst %.cc, $(OBJ_DIR)/%.o, $(LIB_SOURCES))
LIB_OBJECTS += $(patsubst %.cc, $(OBJ_DIR)/%.o, $(ROCKSDB_PLUGIN_SOURCES))
ifeq ($(HAVE_POWER8),1)
LIB_OBJECTS += $(patsubst %.c, $(OBJ_DIR)/%.o, $(LIB_SOURCES_C))
LIB_OBJECTS += $(patsubst %.S, $(OBJ_DIR)/%.o, $(LIB_SOURCES_ASM))
@ -1412,7 +1419,7 @@ $(STATIC_TEST_LIBRARY): $(TEST_OBJECTS)
$(AM_V_AR)rm -f $@ $(SHARED_TEST_LIBRARY)
$(AM_V_at)$(AR) $(ARFLAGS) $@ $^
$(STATIC_TOOLS_LIBRARY): $(BENCH_OBJECTS) $(TOOL_OBJECTS)
$(STATIC_TOOLS_LIBRARY): $(TOOL_OBJECTS)
$(AM_V_AR)rm -f $@ $(SHARED_TOOLS_LIBRARY)
$(AM_V_at)$(AR) $(ARFLAGS) $@ $^
@ -2076,6 +2083,10 @@ install-headers: gen-pc
for header in `$(FIND) "include/rocksdb" -type f -name *.h`; do \
install -C -m 644 $$header $(DESTDIR)/$(PREFIX)/$$header; \
done
for header in $(ROCKSDB_PLUGIN_HEADERS); do \
install -d $(DESTDIR)/$(PREFIX)/include/rocksdb/`dirname $$header`; \
install -C -m 644 $$header $(DESTDIR)/$(PREFIX)/include/rocksdb/$$header; \
done
install -C -m 644 rocksdb.pc $(INSTALL_LIBDIR)/pkgconfig/rocksdb.pc
install-static: install-headers $(LIBRARY)

3
PLUGINS.md Normal file
View File

@ -0,0 +1,3 @@
This is the list of all known third-party plugins for RocksDB. If something is missing, please open a pull request to add it.
* [Dedupfs](https://github.com/ajkr/dedupfs): an example for plugin developers to reference

25
plugin/README.md Normal file
View File

@ -0,0 +1,25 @@
## Building external plugins together with RocksDB
RocksDB offers several plugin interfaces for developers to customize its behavior. One difficulty developers face is how to make their plugin available to end users. The approach discussed here involves building the external code together with the RocksDB code into a single binary. Note another approach we plan to support involves loading plugins dynamically from shared libraries.
### Discovery
We hope developers will mention their work in "PLUGINS.md" so users can easily discover and reuse solutions for customizing RocksDB.
### Directory organization
External plugins will be linked according to their name into a subdirectory of "plugin/". For example, a plugin called "dedupfs" would be linked into "plugin/dedupfs/".
### Build standard
Currently the only supported build system is make. In the plugin directory, files ending in the .mk extension can define the following variables.
* `$(PLUGIN_NAME)_SOURCES`: these files will be compiled and linked with RocksDB. They can access RocksDB public header files.
* `$(PLUGIN_NAME)_HEADERS`: these files will be installed in the RocksDB header directory. Their paths will be prefixed by "rocksdb/plugin/$(PLUGIN_NAME)/".
* `$(PLUGIN_NAME)_LDFLAGS`: these flags will be passed to the final link step. For example, library dependencies can be propagated here, or symbols can be forcibly included, e.g., for static registration.
Users will run the usual make commands from the RocksDB directory, specifying the plugins to include in a space-separated list in the variable `ROCKSDB_PLUGINS`.
### Example
For a working example, see [Dedupfs](https://github.com/ajkr/dedupfs).