andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

fix build with 'USE_HDFS' on windows (#6950)

Browse Source 
Summary:
Build with "USE_HDFS" failed with below errors on Windows. This PR is trying to fix them
Severity	Code	Description	Project	File	Line	Suppression State
Error (active)	E0020	identifier "ssize_t" is undefined	rocksdb	D:\Git\rocksdb\rocksdb\env\env_hdfs.cc	127
Error (active)	E1696	cannot open source file "sys/time.h"	rocksdb	D:\Git\rocksdb\rocksdb\env\env_hdfs.cc	15
Error	C2065	'pthread_t': undeclared identifier	rocksdb	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	166
Error	C3861	'pthread_self': identifier not found	rocksdb	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	167
Error	C1083	Cannot open include file: 'sys/time.h': No such file or directory	rocksdb	d:\git\rocksdb\rocksdb\env\env_hdfs.cc	15
Error	C2065	'pthread_t': undeclared identifier	db_bench	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	166
Error	C3861	'pthread_self': identifier not found	db_bench	d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h	167
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6950

Test Plan:
1. manually test build with "USE_HDFS" on Windows, verified HDFS Env related function by db_bench.exe.
D:\Git\rocksdb\build\Debug>db_bench.exe --hdfs="abfs://test@rdbtest2.dfs.core.windows.net" --num=100 --benchmarks="fillseq,readseq,fillseekseq" --db="abfs://test@rdbtest2.dfs.core.windows.net/test"
2020-06-05 20:42:21,102 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-05 20:42:22,646 WARN utils.SSLSocketFactoryEx: Failed to load OpenSSL. Falling back to the JSSE default.
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 6.10
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    100
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    0.0 MB (estimated)
FileSize:   0.0 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test]
fillseq      :    1138.350 micros/op 877 ops/sec;    0.1 MB/s
DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test]
readseq      :      63.580 micros/op 15627 ops/sec;    1.7 MB/s
DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test]
fillseekseq  :      45.615 micros/op 21762 ops/sec;

Reviewed By: cheng-chang

Differential Revision: D21964806

Pulled By: riversand963

fbshipit-source-id: 9d7413178ece0113d11bc4398583f7d0590d5dbd
This commit is contained in:
Zhen Li 2020-06-12 16:19:56 -07:00 committed by Facebook GitHub Bot
parent 9810f40075
commit d63f86e506
2 changed files with 5 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
env/env_hdfs.cc vendored
View File

@ -12,7 +12,6 @@
#define ROCKSDB_HDFS_FILE_C #define ROCKSDB_HDFS_FILE_C
#include <stdio.h> #include <stdio.h>
#include <sys/time.h>
#include <time.h> #include <time.h>
#include <algorithm> #include <algorithm>
#include <iostream> #include <iostream>
@ -124,8 +123,9 @@ class HdfsReadableFile : virtual public SequentialFile,
Status s; Status s;
ROCKS_LOG_DEBUG(mylog, "[hdfs] HdfsReadableFile preading %s\n", ROCKS_LOG_DEBUG(mylog, "[hdfs] HdfsReadableFile preading %s\n",
filename_.c_str()); filename_.c_str());
ssize_t bytes_read = hdfsPread(fileSys_, hfile_, offset, tSize bytes_read =
(void*)scratch, (tSize)n); hdfsPread(fileSys_, hfile_, offset, static_cast<void*>(scratch),
static_cast<tSize>(n));
ROCKS_LOG_DEBUG(mylog, "[hdfs] HdfsReadableFile pread %s\n", ROCKS_LOG_DEBUG(mylog, "[hdfs] HdfsReadableFile pread %s\n",
filename_.c_str()); filename_.c_str());
*result = Slice(scratch, (bytes_read < 0) ? 0 : bytes_read); *result = Slice(scratch, (bytes_read < 0) ? 0 : bytes_read);

8
hdfs/env_hdfs.h
View File

@ -162,10 +162,7 @@ class HdfsEnv : public Env {
return posixEnv->TimeToString(number); return posixEnv->TimeToString(number);
} }
static uint64_t gettid() { static uint64_t gettid() { return Env::Default()->GetThreadID(); }
assert(sizeof(pthread_t) <= sizeof(uint64_t));
return (uint64_t)pthread_self();
}
uint64_t GetThreadID() const override { return HdfsEnv::gettid(); } uint64_t GetThreadID() const override { return HdfsEnv::gettid(); }
@ -209,8 +206,7 @@ class HdfsEnv : public Env {
std::string portStr = (rem == 0 ? remaining : std::string portStr = (rem == 0 ? remaining :
remaining.substr(0, rem)); remaining.substr(0, rem));
tPort port; tPort port = static_cast<tPort>(atoi(portStr.c_str()));
port = atoi(portStr.c_str());
if (port == 0) { if (port == 0) {
throw HdfsFatalException("Bad host-port for hdfs " + uri); throw HdfsFatalException("Bad host-port for hdfs " + uri);
} }