rocksdb/hdfs
Zhen Li d63f86e506 fix build with 'USE_HDFS' on windows (#6950)
Summary:
Build with "USE_HDFS" failed with below errors on Windows. This PR is trying to fix them
Severity	Code	Description	Project	File	Line	Suppression State
Error (active)	E0020	identifier "ssize_t" is undefined	rocksdb	D:\Git\rocksdb\rocksdb\env\env_hdfs.cc	127
Error (active)	E1696	cannot open source file "sys/time.h"	rocksdb	D:\Git\rocksdb\rocksdb\env\env_hdfs.cc	15
Error	C2065	'pthread_t': undeclared identifier	rocksdb	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	166
Error	C3861	'pthread_self': identifier not found	rocksdb	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	167
Error	C1083	Cannot open include file: 'sys/time.h': No such file or directory	rocksdb	d:\git\rocksdb\rocksdb\env\env_hdfs.cc	15
Error	C2065	'pthread_t': undeclared identifier	db_bench	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	166
Error	C3861	'pthread_self': identifier not found	db_bench	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	167
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6950

Test Plan:
1. manually test build with "USE_HDFS" on Windows, verified HDFS Env related function by db_bench.exe.
D:\Git\rocksdb\build\Debug>db_bench.exe --hdfs="abfs://test@rdbtest2.dfs.core.windows.net" --num=100 --benchmarks="fillseq,readseq,fillseekseq" --db="abfs://test@rdbtest2.dfs.core.windows.net/test"
2020-06-05 20:42:21,102 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-05 20:42:22,646 WARN utils.SSLSocketFactoryEx: Failed to load OpenSSL. Falling back to the JSSE default.
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 6.10
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    100
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    0.0 MB (estimated)
FileSize:   0.0 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test]
fillseq      :    1138.350 micros/op 877 ops/sec;    0.1 MB/s
DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test]
readseq      :      63.580 micros/op 15627 ops/sec;    1.7 MB/s
DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test]
fillseekseq  :      45.615 micros/op 21762 ops/sec;

Reviewed By: cheng-chang

Differential Revision: D21964806

Pulled By: riversand963

fbshipit-source-id: 9d7413178ece0113d11bc4398583f7d0590d5dbd
2020-06-12 16:21:50 -07:00
..
env_hdfs.h fix build with 'USE_HDFS' on windows (#6950) 2020-06-12 16:21:50 -07:00
README using java7 in runtime for hdfs env (#1072) 2016-04-12 00:08:26 -04:00
setup.sh Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00

This directory contains the hdfs extensions needed to make rocksdb store
files in HDFS.

It has been compiled and testing against CDH 4.4 (2.0.0+1475-1.cdh4.4.0.p0.23~precise-cdh4.4.0).

The configuration assumes that packages libhdfs0, libhdfs0-dev are 
installed which basically means that hdfs.h is in /usr/include and libhdfs in /usr/lib

The env_hdfs.h file defines the rocksdb objects that are needed to talk to an
underlying filesystem. 

If you want to compile rocksdb with hdfs support, please set the following
environment variables appropriately (also defined in setup.sh for convenience)
   USE_HDFS=1
   JAVA_HOME=/usr/local/jdk-7u79-64
   LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/jdk-7u79-64/jre/lib/amd64/server:/usr/local/jdk-7u79-64/jre/lib/amd64/:./snappy/libs
   make clean all db_bench

To run dbbench,
  set CLASSPATH to include your hadoop distribution
  db_bench --hdfs="hdfs://hbaseudbperf001.snc1.facebook.com:9000"