andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Ensure the destruction order of PosixEnv and ThreadLocalPtr

Browse Source 
Summary:
By default, RocksDB initializes the singletons of ThreadLocalPtr first, then initializes PosixEnv
via static initializer.  Destructor terminates objects in reverse order, so terminating PosixEnv
(calling pthread_mutex_lock), then ThreadLocal (calling pthread_mutex_destroy).

However, in certain case, application might initialize PosixEnv first, then ThreadLocalPtr.
This will cause core dump at the end of the program (eg. https://github.com/facebook/mysql-5.6/issues/122)

This patch fix this issue by ensuring the destruction order by moving the global static singletons
to function static singletons.  Since function static singletons are initialized when the function is first
called, this property allows us invoke to enforce the construction of the static PosixEnv and the
singletons of ThreadLocalPtr by calling the function where the ThreadLocalPtr singletons belongs
right before we initialize the static PosixEnv.

Test Plan: Verified in the MyRocks.

Reviewers: yoshinorim, IslamAbdelRahman, rven, kradhakrishnan, anthony, sdong, MarkCallaghan

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D51789
This commit is contained in:
Yueh-Hsuan Chiang 2015-12-11 00:21:58 -08:00
parent 64fa43843b
commit 00d6edf6a0
3 changed files with 82 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
util/env_posix.cc
View File

@ -48,6 +48,7 @@
#include "util/posix_logger.h"
#include "util/string_util.h"
#include "util/sync_point.h"
#include "util/thread_local.h"
#include "util/thread_status_updater.h"
#if !defined(TMPFS_MAGIC)
@ -761,6 +762,17 @@ std::string Env::GenerateUniqueId() {
}
Env* Env::Default() {
// The following function call initializes the singletons of ThreadLocalPtr
// right before the static default_env. This guarantees default_env will
// always being destructed before the ThreadLocalPtr singletons get
// destructed as C++ guarantees that the destructions of static variables
// is in the reverse order of their constructions.
//
// Since static members are destructed in the reverse order
// of their construction, having this call here guarantees that
// the destructor of static PosixEnv will go first, then the
// the singletons of ThreadLocalPtr.
ThreadLocalPtr::InitSingletons();
static PosixEnv default_env;
return &default_env;
}

51
util/thread_local.cc
View File

@ -14,7 +14,6 @@
namespace rocksdb {
port::Mutex ThreadLocalPtr::StaticMeta::mutex_;
#if ROCKSDB_SUPPORT_THREAD_LOCAL
__thread ThreadLocalPtr::ThreadData* ThreadLocalPtr::StaticMeta::tls_ = nullptr;
#endif
@ -103,11 +102,33 @@ PIMAGE_TLS_CALLBACK p_thread_callback_on_exit = wintlscleanup::WinOnThreadExit;
#endif // OS_WIN
void ThreadLocalPtr::InitSingletons() {
ThreadLocalPtr::StaticMeta::InitSingletons();
ThreadLocalPtr::Instance();
}
ThreadLocalPtr::StaticMeta* ThreadLocalPtr::Instance() {
// Here we prefer function static variable instead of global
// static variable as function static variable is initialized
// when the function is first call. As a result, we can properly
// control their construction order by properly preparing their
// first function call.
static ThreadLocalPtr::StaticMeta inst;
return &inst;
}
void ThreadLocalPtr::StaticMeta::InitSingletons() { Mutex(); }
port::Mutex* ThreadLocalPtr::StaticMeta::Mutex() {
// Here we prefer function static variable instead of global
// static variable as function static variable is initialized
// when the function is first call. As a result, we can properly
// control their construction order by properly preparing their
// first function call.
static port::Mutex mutex;
return &mutex;
}
void ThreadLocalPtr::StaticMeta::OnThreadExit(void* ptr) {
auto* tls = static_cast<ThreadData*>(ptr);
assert(tls != nullptr);
@ -115,7 +136,7 @@ void ThreadLocalPtr::StaticMeta::OnThreadExit(void* ptr) {
auto* inst = Instance();
pthread_setspecific(inst->pthread_key_, nullptr);
MutexLock l(&mutex_);
MutexLock l(Mutex());
inst->RemoveThreadData(tls);
// Unref stored pointers of current thread from all instances
uint32_t id = 0;
@ -175,7 +196,7 @@ ThreadLocalPtr::StaticMeta::StaticMeta() : next_instance_id_(0) {
}
void ThreadLocalPtr::StaticMeta::AddThreadData(ThreadLocalPtr::ThreadData* d) {
mutex_.AssertHeld();
Mutex()->AssertHeld();
d->next = &head_;
d->prev = head_.prev;
head_.prev->next = d;
@ -184,7 +205,7 @@ void ThreadLocalPtr::StaticMeta::AddThreadData(ThreadLocalPtr::ThreadData* d) {
void ThreadLocalPtr::StaticMeta::RemoveThreadData(
ThreadLocalPtr::ThreadData* d) {
mutex_.AssertHeld();
Mutex()->AssertHeld();
d->next->prev = d->prev;
d->prev->next = d->next;
d->next = d->prev = d;
@ -204,14 +225,14 @@ ThreadLocalPtr::ThreadData* ThreadLocalPtr::StaticMeta::GetThreadLocal() {
{
// Register it in the global chain, needs to be done before thread exit
// handler registration
MutexLock l(&mutex_);
MutexLock l(Mutex());
inst->AddThreadData(tls_);
}
// Even it is not OS_MACOSX, need to register value for pthread_key_ so that
// its exit handler will be triggered.
if (pthread_setspecific(inst->pthread_key_, tls_) != 0) {
{
MutexLock l(&mutex_);
MutexLock l(Mutex());
inst->RemoveThreadData(tls_);
}
delete tls_;
@ -233,7 +254,7 @@ void ThreadLocalPtr::StaticMeta::Reset(uint32_t id, void* ptr) {
auto* tls = GetThreadLocal();
if (UNLIKELY(id >= tls->entries.size())) {
// Need mutex to protect entries access within ReclaimId
MutexLock l(&mutex_);
MutexLock l(Mutex());
tls->entries.resize(id + 1);
}
tls->entries[id].ptr.store(ptr, std::memory_order_release);
@ -243,7 +264,7 @@ void* ThreadLocalPtr::StaticMeta::Swap(uint32_t id, void* ptr) {
auto* tls = GetThreadLocal();
if (UNLIKELY(id >= tls->entries.size())) {
// Need mutex to protect entries access within ReclaimId
MutexLock l(&mutex_);
MutexLock l(Mutex());
tls->entries.resize(id + 1);
}
return tls->entries[id].ptr.exchange(ptr, std::memory_order_acquire);
@ -254,7 +275,7 @@ bool ThreadLocalPtr::StaticMeta::CompareAndSwap(uint32_t id, void* ptr,
auto* tls = GetThreadLocal();
if (UNLIKELY(id >= tls->entries.size())) {
// Need mutex to protect entries access within ReclaimId
MutexLock l(&mutex_);
MutexLock l(Mutex());
tls->entries.resize(id + 1);
}
return tls->entries[id].ptr.compare_exchange_strong(
@ -263,7 +284,7 @@ bool ThreadLocalPtr::StaticMeta::CompareAndSwap(uint32_t id, void* ptr,
void ThreadLocalPtr::StaticMeta::Scrape(uint32_t id, autovector<void*>* ptrs,
void* const replacement) {
MutexLock l(&mutex_);
MutexLock l(Mutex());
for (ThreadData* t = head_.next; t != &head_; t = t->next) {
if (id < t->entries.size()) {
void* ptr =
@ -276,12 +297,12 @@ void ThreadLocalPtr::StaticMeta::Scrape(uint32_t id, autovector<void*>* ptrs,
}
void ThreadLocalPtr::StaticMeta::SetHandler(uint32_t id, UnrefHandler handler) {
MutexLock l(&mutex_);
MutexLock l(Mutex());
handler_map_[id] = handler;
}
UnrefHandler ThreadLocalPtr::StaticMeta::GetHandler(uint32_t id) {
mutex_.AssertHeld();
Mutex()->AssertHeld();
auto iter = handler_map_.find(id);
if (iter == handler_map_.end()) {
return nullptr;
@ -290,7 +311,7 @@ UnrefHandler ThreadLocalPtr::StaticMeta::GetHandler(uint32_t id) {
}
uint32_t ThreadLocalPtr::StaticMeta::GetId() {
MutexLock l(&mutex_);
MutexLock l(Mutex());
if (free_instance_ids_.empty()) {
return next_instance_id_++;
}
@ -301,7 +322,7 @@ uint32_t ThreadLocalPtr::StaticMeta::GetId() {
}
uint32_t ThreadLocalPtr::StaticMeta::PeekId() const {
MutexLock l(&mutex_);
MutexLock l(Mutex());
if (!free_instance_ids_.empty()) {
return free_instance_ids_.back();
}
@ -311,7 +332,7 @@ uint32_t ThreadLocalPtr::StaticMeta::PeekId() const {
void ThreadLocalPtr::StaticMeta::ReclaimId(uint32_t id) {
// This id is not used, go through all thread local data and release
// corresponding value
MutexLock l(&mutex_);
MutexLock l(Mutex());
auto unref = GetHandler(id);
for (ThreadData* t = head_.next; t != &head_; t = t->next) {
if (id < t->entries.size()) {

35
util/thread_local.h
View File

@ -63,6 +63,15 @@ class ThreadLocalPtr {
// data for all existing threads
void Scrape(autovector<void*>* ptrs, void* const replacement);
// Initialize the static singletons of the ThreadLocalPtr.
//
// If this function is not called, then the singletons will be
// automatically initialized when they are used.
//
// Calling this function twice or after the singletons have been
// initialized will be no-op.
static void InitSingletons();
protected:
struct Entry {
Entry() : ptr(nullptr) {}
@ -121,6 +130,15 @@ class ThreadLocalPtr {
// Register the UnrefHandler for id
void SetHandler(uint32_t id, UnrefHandler handler);
// Initialize all the singletons associated with StaticMeta.
//
// If this function is not called, then the singletons will be
// automatically initialized when they are used.
//
// Calling this function twice or after the singletons have been
// initialized will be no-op.
static void InitSingletons();
private:
// Get UnrefHandler for id with acquiring mutex
// REQUIRES: mutex locked
@ -153,7 +171,22 @@ class ThreadLocalPtr {
// protect inst, next_instance_id_, free_instance_ids_, head_,
// ThreadData.entries
static port::Mutex mutex_;
//
// Note that here we prefer function static variable instead of the usual
// global static variable. The reason is that c++ destruction order of
// static variables in the reverse order of their construction order.
// However, C++ does not guarantee any construction order when global
// static variables are defined in different files, while the function
// static variables are initialized when their function are first called.
// As a result, the construction order of the function static variables
// can be controlled by properly invoke their first function calls in
// the right order.
//
// For instance, the following function contains a function static
// variable. We place a dummy function call of this inside
// Env::Default() to ensure the construction order of the construction
// order.
static port::Mutex* Mutex();
#if ROCKSDB_SUPPORT_THREAD_LOCAL
// Thread local storage
static __thread ThreadData* tls_;