2016-02-10 00:12:00 +01:00
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
2017-07-16 01:03:42 +02:00
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
|
2014-08-01 17:55:37 +02:00
|
|
|
#include "rocksdb/utilities/spatial_db.h"
|
|
|
|
|
2014-09-04 16:04:37 +02:00
|
|
|
#ifndef __STDC_FORMAT_MACROS
|
2014-08-01 17:55:37 +02:00
|
|
|
#define __STDC_FORMAT_MACROS
|
2014-09-04 16:04:37 +02:00
|
|
|
#endif
|
|
|
|
|
2014-11-13 22:34:29 +01:00
|
|
|
#include <algorithm>
|
|
|
|
#include <condition_variable>
|
2014-08-01 17:55:37 +02:00
|
|
|
#include <inttypes.h>
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
#include <string>
|
|
|
|
#include <vector>
|
2014-11-13 22:34:29 +01:00
|
|
|
#include <mutex>
|
|
|
|
#include <thread>
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
#include <set>
|
2014-08-01 22:50:40 +02:00
|
|
|
#include <unordered_set>
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
|
|
|
#include "rocksdb/cache.h"
|
2014-08-26 02:50:18 +02:00
|
|
|
#include "rocksdb/options.h"
|
2014-08-27 04:01:36 +02:00
|
|
|
#include "rocksdb/memtablerep.h"
|
2014-08-26 02:50:18 +02:00
|
|
|
#include "rocksdb/slice_transform.h"
|
2014-09-09 02:00:43 +02:00
|
|
|
#include "rocksdb/statistics.h"
|
2014-08-25 23:22:05 +02:00
|
|
|
#include "rocksdb/table.h"
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
#include "rocksdb/db.h"
|
|
|
|
#include "rocksdb/utilities/stackable_db.h"
|
|
|
|
#include "util/coding.h"
|
2014-07-24 22:39:33 +02:00
|
|
|
#include "utilities/spatialdb/utils.h"
|
2017-02-06 23:43:55 +01:00
|
|
|
#include "port/port.h"
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
|
|
|
namespace rocksdb {
|
|
|
|
namespace spatial {
|
|
|
|
|
2014-07-24 22:39:33 +02:00
|
|
|
// Column families are used to store element's data and spatial indexes. We use
|
|
|
|
// [default] column family to store the element data. This is the format of
|
|
|
|
// [default] column family:
|
|
|
|
// * id (fixed 64 big endian) -> blob (length prefixed slice) feature_set
|
|
|
|
// (serialized)
|
|
|
|
// We have one additional column family for each spatial index. The name of the
|
|
|
|
// column family is [spatial$<spatial_index_name>]. The format is:
|
|
|
|
// * quad_key (fixed 64 bit big endian) id (fixed 64 bit big endian) -> ""
|
|
|
|
// We store information about indexes in [metadata] column family. Format is:
|
|
|
|
// * spatial$<spatial_index_name> -> bbox (4 double encodings) tile_bits
|
|
|
|
// (varint32)
|
|
|
|
|
|
|
|
namespace {
|
|
|
|
const std::string kMetadataColumnFamilyName("metadata");
|
|
|
|
inline std::string GetSpatialIndexColumnFamilyName(
|
|
|
|
const std::string& spatial_index_name) {
|
|
|
|
return "spatial$" + spatial_index_name;
|
|
|
|
}
|
|
|
|
inline bool GetSpatialIndexName(const std::string& column_family_name,
|
|
|
|
Slice* dst) {
|
|
|
|
*dst = Slice(column_family_name);
|
|
|
|
if (dst->starts_with("spatial$")) {
|
|
|
|
dst->remove_prefix(8); // strlen("spatial$")
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
} // namespace
|
|
|
|
|
2015-07-02 01:13:49 +02:00
|
|
|
void Variant::Init(const Variant& v, Data& d) {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
switch (v.type_) {
|
2015-07-13 21:11:05 +02:00
|
|
|
case kNull:
|
|
|
|
break;
|
|
|
|
case kBool:
|
|
|
|
d.b = v.data_.b;
|
|
|
|
break;
|
|
|
|
case kInt:
|
|
|
|
d.i = v.data_.i;
|
|
|
|
break;
|
|
|
|
case kDouble:
|
|
|
|
d.d = v.data_.d;
|
|
|
|
break;
|
|
|
|
case kString:
|
|
|
|
new (d.s) std::string(*GetStringPtr(v.data_));
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
assert(false);
|
2015-07-02 01:13:49 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
Variant& Variant::operator=(const Variant& v) {
|
|
|
|
// Construct first a temp so exception from a string ctor
|
|
|
|
// does not change this object
|
|
|
|
Data tmp;
|
|
|
|
Init(v, tmp);
|
|
|
|
|
|
|
|
Type thisType = type_;
|
|
|
|
// Boils down to copying bits so safe
|
|
|
|
std::swap(tmp, data_);
|
|
|
|
type_ = v.type_;
|
|
|
|
|
|
|
|
Destroy(thisType, tmp);
|
|
|
|
|
|
|
|
return *this;
|
|
|
|
}
|
|
|
|
|
|
|
|
Variant& Variant::operator=(Variant&& rhs) {
|
|
|
|
Destroy(type_, data_);
|
|
|
|
if (rhs.type_ == kString) {
|
2015-07-02 02:03:56 +02:00
|
|
|
new (data_.s) std::string(std::move(*GetStringPtr(rhs.data_)));
|
2015-07-02 01:13:49 +02:00
|
|
|
} else {
|
|
|
|
data_ = rhs.data_;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2015-07-02 01:13:49 +02:00
|
|
|
type_ = rhs.type_;
|
|
|
|
rhs.type_ = kNull;
|
|
|
|
return *this;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
|
2015-07-02 01:13:49 +02:00
|
|
|
bool Variant::operator==(const Variant& rhs) const {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
if (type_ != rhs.type_) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
switch (type_) {
|
|
|
|
case kNull:
|
|
|
|
return true;
|
|
|
|
case kBool:
|
|
|
|
return data_.b == rhs.data_.b;
|
|
|
|
case kInt:
|
|
|
|
return data_.i == rhs.data_.i;
|
|
|
|
case kDouble:
|
|
|
|
return data_.d == rhs.data_.d;
|
|
|
|
case kString:
|
2015-07-02 02:03:56 +02:00
|
|
|
return *GetStringPtr(data_) == *GetStringPtr(rhs.data_);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
default:
|
|
|
|
assert(false);
|
|
|
|
}
|
2014-07-23 22:49:58 +02:00
|
|
|
// it will never reach here, but otherwise the compiler complains
|
|
|
|
return false;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
FeatureSet* FeatureSet::Set(const std::string& key, const Variant& value) {
|
|
|
|
map_.insert({key, value});
|
|
|
|
return this;
|
|
|
|
}
|
|
|
|
|
|
|
|
bool FeatureSet::Contains(const std::string& key) const {
|
|
|
|
return map_.find(key) != map_.end();
|
|
|
|
}
|
|
|
|
|
|
|
|
const Variant& FeatureSet::Get(const std::string& key) const {
|
|
|
|
auto itr = map_.find(key);
|
|
|
|
assert(itr != map_.end());
|
|
|
|
return itr->second;
|
|
|
|
}
|
|
|
|
|
|
|
|
FeatureSet::iterator FeatureSet::Find(const std::string& key) const {
|
|
|
|
return iterator(map_.find(key));
|
|
|
|
}
|
|
|
|
|
|
|
|
void FeatureSet::Clear() { map_.clear(); }
|
|
|
|
|
|
|
|
void FeatureSet::Serialize(std::string* output) const {
|
|
|
|
for (const auto& iter : map_) {
|
|
|
|
PutLengthPrefixedSlice(output, iter.first);
|
|
|
|
output->push_back(static_cast<char>(iter.second.type()));
|
|
|
|
switch (iter.second.type()) {
|
|
|
|
case Variant::kNull:
|
|
|
|
break;
|
|
|
|
case Variant::kBool:
|
|
|
|
output->push_back(static_cast<char>(iter.second.get_bool()));
|
|
|
|
break;
|
|
|
|
case Variant::kInt:
|
|
|
|
PutVarint64(output, iter.second.get_int());
|
|
|
|
break;
|
|
|
|
case Variant::kDouble: {
|
2014-07-24 22:39:33 +02:00
|
|
|
PutDouble(output, iter.second.get_double());
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
case Variant::kString:
|
|
|
|
PutLengthPrefixedSlice(output, iter.second.get_string());
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
assert(false);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
bool FeatureSet::Deserialize(const Slice& input) {
|
|
|
|
assert(map_.empty());
|
|
|
|
Slice s(input);
|
|
|
|
while (s.size()) {
|
|
|
|
Slice key;
|
|
|
|
if (!GetLengthPrefixedSlice(&s, &key) || s.size() == 0) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
char type = s[0];
|
|
|
|
s.remove_prefix(1);
|
|
|
|
switch (type) {
|
|
|
|
case Variant::kNull: {
|
|
|
|
map_.insert({key.ToString(), Variant()});
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case Variant::kBool: {
|
|
|
|
if (s.size() == 0) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
map_.insert({key.ToString(), Variant(static_cast<bool>(s[0]))});
|
|
|
|
s.remove_prefix(1);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case Variant::kInt: {
|
|
|
|
uint64_t v;
|
|
|
|
if (!GetVarint64(&s, &v)) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
map_.insert({key.ToString(), Variant(v)});
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case Variant::kDouble: {
|
2014-07-24 22:39:33 +02:00
|
|
|
double d;
|
|
|
|
if (!GetDouble(&s, &d)) {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
map_.insert({key.ToString(), Variant(d)});
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case Variant::kString: {
|
|
|
|
Slice str;
|
|
|
|
if (!GetLengthPrefixedSlice(&s, &str)) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
map_.insert({key.ToString(), str.ToString()});
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
default:
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2014-08-01 17:55:37 +02:00
|
|
|
std::string FeatureSet::DebugString() const {
|
|
|
|
std::string out = "{";
|
|
|
|
bool comma = false;
|
|
|
|
for (const auto& iter : map_) {
|
|
|
|
if (comma) {
|
|
|
|
out.append(", ");
|
|
|
|
} else {
|
|
|
|
comma = true;
|
|
|
|
}
|
|
|
|
out.append("\"" + iter.first + "\": ");
|
|
|
|
switch (iter.second.type()) {
|
|
|
|
case Variant::kNull:
|
|
|
|
out.append("null");
|
2014-09-05 23:14:30 +02:00
|
|
|
break;
|
2014-08-01 17:55:37 +02:00
|
|
|
case Variant::kBool:
|
|
|
|
if (iter.second.get_bool()) {
|
|
|
|
out.append("true");
|
|
|
|
} else {
|
|
|
|
out.append("false");
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case Variant::kInt: {
|
|
|
|
char buf[32];
|
|
|
|
snprintf(buf, sizeof(buf), "%" PRIu64, iter.second.get_int());
|
|
|
|
out.append(buf);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case Variant::kDouble: {
|
|
|
|
char buf[32];
|
|
|
|
snprintf(buf, sizeof(buf), "%lf", iter.second.get_double());
|
|
|
|
out.append(buf);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case Variant::kString:
|
|
|
|
out.append("\"" + iter.second.get_string() + "\"");
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
assert(false);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return out + "}";
|
|
|
|
}
|
|
|
|
|
2014-08-26 02:50:18 +02:00
|
|
|
class ValueGetter {
|
|
|
|
public:
|
|
|
|
ValueGetter() {}
|
|
|
|
virtual ~ValueGetter() {}
|
|
|
|
|
|
|
|
virtual bool Get(uint64_t id) = 0;
|
|
|
|
virtual const Slice value() const = 0;
|
|
|
|
|
|
|
|
virtual Status status() const = 0;
|
|
|
|
};
|
|
|
|
|
|
|
|
class ValueGetterFromDB : public ValueGetter {
|
|
|
|
public:
|
|
|
|
ValueGetterFromDB(DB* db, ColumnFamilyHandle* cf) : db_(db), cf_(cf) {}
|
|
|
|
|
|
|
|
virtual bool Get(uint64_t id) override {
|
|
|
|
std::string encoded_id;
|
|
|
|
PutFixed64BigEndian(&encoded_id, id);
|
|
|
|
status_ = db_->Get(ReadOptions(), cf_, encoded_id, &value_);
|
|
|
|
if (status_.IsNotFound()) {
|
|
|
|
status_ = Status::Corruption("Index inconsistency");
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
virtual const Slice value() const override { return value_; }
|
|
|
|
|
|
|
|
virtual Status status() const override { return status_; }
|
|
|
|
|
|
|
|
private:
|
|
|
|
std::string value_;
|
|
|
|
DB* db_;
|
|
|
|
ColumnFamilyHandle* cf_;
|
|
|
|
Status status_;
|
|
|
|
};
|
|
|
|
|
|
|
|
class ValueGetterFromIterator : public ValueGetter {
|
|
|
|
public:
|
|
|
|
explicit ValueGetterFromIterator(Iterator* iterator) : iterator_(iterator) {}
|
|
|
|
|
|
|
|
virtual bool Get(uint64_t id) override {
|
|
|
|
std::string encoded_id;
|
|
|
|
PutFixed64BigEndian(&encoded_id, id);
|
|
|
|
iterator_->Seek(encoded_id);
|
|
|
|
|
|
|
|
if (!iterator_->Valid() || iterator_->key() != Slice(encoded_id)) {
|
|
|
|
status_ = Status::Corruption("Index inconsistency");
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
virtual const Slice value() const override { return iterator_->value(); }
|
|
|
|
|
|
|
|
virtual Status status() const override { return status_; }
|
|
|
|
|
|
|
|
private:
|
|
|
|
std::unique_ptr<Iterator> iterator_;
|
|
|
|
Status status_;
|
|
|
|
};
|
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
class SpatialIndexCursor : public Cursor {
|
|
|
|
public:
|
2014-08-01 22:50:40 +02:00
|
|
|
// tile_box is inclusive
|
2014-08-26 02:50:18 +02:00
|
|
|
SpatialIndexCursor(Iterator* spatial_iterator, ValueGetter* value_getter,
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
const BoundingBox<uint64_t>& tile_bbox, uint32_t tile_bits)
|
2014-08-26 02:50:18 +02:00
|
|
|
: value_getter_(value_getter), valid_(true) {
|
2014-08-01 22:50:40 +02:00
|
|
|
// calculate quad keys we'll need to query
|
|
|
|
std::vector<uint64_t> quad_keys;
|
|
|
|
quad_keys.reserve((tile_bbox.max_x - tile_bbox.min_x + 1) *
|
|
|
|
(tile_bbox.max_y - tile_bbox.min_y + 1));
|
|
|
|
for (uint64_t x = tile_bbox.min_x; x <= tile_bbox.max_x; ++x) {
|
|
|
|
for (uint64_t y = tile_bbox.min_y; y <= tile_bbox.max_y; ++y) {
|
|
|
|
quad_keys.push_back(GetQuadKeyFromTile(x, y, tile_bits));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
std::sort(quad_keys.begin(), quad_keys.end());
|
|
|
|
|
|
|
|
// load primary key ids for all quad keys
|
|
|
|
for (auto quad_key : quad_keys) {
|
|
|
|
std::string encoded_quad_key;
|
|
|
|
PutFixed64BigEndian(&encoded_quad_key, quad_key);
|
|
|
|
Slice slice_quad_key(encoded_quad_key);
|
|
|
|
|
|
|
|
// If CheckQuadKey is true, there is no need to reseek, since
|
|
|
|
// spatial_iterator is already pointing at the correct quad key. This is
|
|
|
|
// an optimization.
|
|
|
|
if (!CheckQuadKey(spatial_iterator, slice_quad_key)) {
|
|
|
|
spatial_iterator->Seek(slice_quad_key);
|
|
|
|
}
|
|
|
|
|
|
|
|
while (CheckQuadKey(spatial_iterator, slice_quad_key)) {
|
|
|
|
// extract ID from spatial_iterator
|
|
|
|
uint64_t id;
|
|
|
|
bool ok = GetFixed64BigEndian(
|
|
|
|
Slice(spatial_iterator->key().data() + sizeof(uint64_t),
|
|
|
|
sizeof(uint64_t)),
|
|
|
|
&id);
|
|
|
|
if (!ok) {
|
|
|
|
valid_ = false;
|
|
|
|
status_ = Status::Corruption("Spatial index corruption");
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
primary_key_ids_.insert(id);
|
|
|
|
spatial_iterator->Next();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!spatial_iterator->status().ok()) {
|
|
|
|
status_ = spatial_iterator->status();
|
|
|
|
valid_ = false;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2014-08-01 22:50:40 +02:00
|
|
|
delete spatial_iterator;
|
|
|
|
|
2014-09-29 23:11:00 +02:00
|
|
|
valid_ = valid_ && !primary_key_ids_.empty();
|
2014-08-01 22:50:40 +02:00
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
if (valid_) {
|
2014-08-01 22:50:40 +02:00
|
|
|
primary_keys_iterator_ = primary_key_ids_.begin();
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
ExtractData();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
virtual bool Valid() const override { return valid_; }
|
|
|
|
|
|
|
|
virtual void Next() override {
|
|
|
|
assert(valid_);
|
|
|
|
|
2014-08-01 22:50:40 +02:00
|
|
|
++primary_keys_iterator_;
|
|
|
|
if (primary_keys_iterator_ == primary_key_ids_.end()) {
|
|
|
|
valid_ = false;
|
|
|
|
return;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2014-08-01 22:50:40 +02:00
|
|
|
|
|
|
|
ExtractData();
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
virtual const Slice blob() override { return current_blob_; }
|
|
|
|
virtual const FeatureSet& feature_set() override {
|
|
|
|
return current_feature_set_;
|
|
|
|
}
|
|
|
|
|
|
|
|
virtual Status status() const override {
|
|
|
|
if (!status_.ok()) {
|
|
|
|
return status_;
|
|
|
|
}
|
2014-08-26 02:50:18 +02:00
|
|
|
return value_getter_->status();
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
private:
|
|
|
|
// * returns true if spatial iterator is on the current quad key and all is
|
2014-08-01 22:50:40 +02:00
|
|
|
// well
|
|
|
|
// * returns false if spatial iterator is not on current, or iterator is
|
|
|
|
// invalid or corruption
|
|
|
|
bool CheckQuadKey(Iterator* spatial_iterator, const Slice& quad_key) {
|
|
|
|
if (!spatial_iterator->Valid()) {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
return false;
|
|
|
|
}
|
2014-08-01 22:50:40 +02:00
|
|
|
if (spatial_iterator->key().size() != 2 * sizeof(uint64_t)) {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
status_ = Status::Corruption("Invalid spatial index key");
|
|
|
|
valid_ = false;
|
|
|
|
return false;
|
|
|
|
}
|
2014-08-01 22:50:40 +02:00
|
|
|
Slice spatial_iterator_quad_key(spatial_iterator->key().data(),
|
|
|
|
sizeof(uint64_t));
|
|
|
|
if (spatial_iterator_quad_key != quad_key) {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
// caller needs to reseek
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
// if we come to here, we have found the quad key
|
|
|
|
return true;
|
|
|
|
}
|
2014-08-01 22:50:40 +02:00
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
void ExtractData() {
|
|
|
|
assert(valid_);
|
2014-08-26 02:50:18 +02:00
|
|
|
valid_ = value_getter_->Get(*primary_keys_iterator_);
|
2014-08-01 22:50:40 +02:00
|
|
|
|
2014-08-26 02:50:18 +02:00
|
|
|
if (valid_) {
|
|
|
|
Slice data = value_getter_->value();
|
|
|
|
current_feature_set_.Clear();
|
|
|
|
if (!GetLengthPrefixedSlice(&data, ¤t_blob_) ||
|
|
|
|
!current_feature_set_.Deserialize(data)) {
|
|
|
|
status_ = Status::Corruption("Primary key column family corruption");
|
|
|
|
valid_ = false;
|
|
|
|
}
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
}
|
|
|
|
|
2014-08-26 02:50:18 +02:00
|
|
|
unique_ptr<ValueGetter> value_getter_;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
bool valid_;
|
|
|
|
Status status_;
|
|
|
|
|
|
|
|
FeatureSet current_feature_set_;
|
|
|
|
Slice current_blob_;
|
|
|
|
|
2014-08-01 22:50:40 +02:00
|
|
|
// This is loaded from spatial iterator.
|
|
|
|
std::unordered_set<uint64_t> primary_key_ids_;
|
|
|
|
std::unordered_set<uint64_t>::iterator primary_keys_iterator_;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
};
|
|
|
|
|
|
|
|
class ErrorCursor : public Cursor {
|
|
|
|
public:
|
|
|
|
explicit ErrorCursor(Status s) : s_(s) { assert(!s.ok()); }
|
|
|
|
virtual Status status() const override { return s_; }
|
|
|
|
virtual bool Valid() const override { return false; }
|
|
|
|
virtual void Next() override { assert(false); }
|
|
|
|
|
|
|
|
virtual const Slice blob() override {
|
|
|
|
assert(false);
|
|
|
|
return Slice();
|
|
|
|
}
|
|
|
|
virtual const FeatureSet& feature_set() override {
|
|
|
|
assert(false);
|
|
|
|
// compiler complains otherwise
|
|
|
|
return trash_;
|
|
|
|
}
|
|
|
|
|
|
|
|
private:
|
|
|
|
Status s_;
|
|
|
|
FeatureSet trash_;
|
|
|
|
};
|
|
|
|
|
|
|
|
class SpatialDBImpl : public SpatialDB {
|
|
|
|
public:
|
|
|
|
// * db -- base DB that needs to be forwarded to StackableDB
|
|
|
|
// * data_column_family -- column family used to store the data
|
|
|
|
// * spatial_indexes -- a list of spatial indexes together with column
|
|
|
|
// families that correspond to those spatial indexes
|
|
|
|
// * next_id -- next ID in auto-incrementing ID. This is usually
|
|
|
|
// `max_id_currenty_in_db + 1`
|
2014-07-25 02:26:48 +02:00
|
|
|
SpatialDBImpl(
|
|
|
|
DB* db, ColumnFamilyHandle* data_column_family,
|
|
|
|
const std::vector<std::pair<SpatialIndexOptions, ColumnFamilyHandle*>>&
|
|
|
|
spatial_indexes,
|
2014-08-26 02:50:18 +02:00
|
|
|
uint64_t next_id, bool read_only)
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
: SpatialDB(db),
|
|
|
|
data_column_family_(data_column_family),
|
2014-08-26 02:50:18 +02:00
|
|
|
next_id_(next_id),
|
|
|
|
read_only_(read_only) {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
for (const auto& index : spatial_indexes) {
|
|
|
|
name_to_index_.insert(
|
|
|
|
{index.first.name, IndexColumnFamily(index.first, index.second)});
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
~SpatialDBImpl() {
|
|
|
|
for (auto& iter : name_to_index_) {
|
|
|
|
delete iter.second.column_family;
|
|
|
|
}
|
|
|
|
delete data_column_family_;
|
|
|
|
}
|
|
|
|
|
|
|
|
virtual Status Insert(
|
|
|
|
const WriteOptions& write_options, const BoundingBox<double>& bbox,
|
|
|
|
const Slice& blob, const FeatureSet& feature_set,
|
|
|
|
const std::vector<std::string>& spatial_indexes) override {
|
|
|
|
WriteBatch batch;
|
|
|
|
|
|
|
|
if (spatial_indexes.size() == 0) {
|
|
|
|
return Status::InvalidArgument("Spatial indexes can't be empty");
|
|
|
|
}
|
|
|
|
|
2014-09-18 19:57:20 +02:00
|
|
|
const size_t kWriteOutEveryBytes = 1024 * 1024; // 1MB
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
uint64_t id = next_id_.fetch_add(1);
|
|
|
|
|
|
|
|
for (const auto& si : spatial_indexes) {
|
|
|
|
auto itr = name_to_index_.find(si);
|
|
|
|
if (itr == name_to_index_.end()) {
|
|
|
|
return Status::InvalidArgument("Can't find index " + si);
|
|
|
|
}
|
|
|
|
const auto& spatial_index = itr->second.index;
|
|
|
|
if (!spatial_index.bbox.Intersects(bbox)) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
BoundingBox<uint64_t> tile_bbox = GetTileBoundingBox(spatial_index, bbox);
|
|
|
|
|
|
|
|
for (uint64_t x = tile_bbox.min_x; x <= tile_bbox.max_x; ++x) {
|
|
|
|
for (uint64_t y = tile_bbox.min_y; y <= tile_bbox.max_y; ++y) {
|
|
|
|
// see above for format
|
|
|
|
std::string key;
|
|
|
|
PutFixed64BigEndian(
|
|
|
|
&key, GetQuadKeyFromTile(x, y, spatial_index.tile_bits));
|
|
|
|
PutFixed64BigEndian(&key, id);
|
|
|
|
batch.Put(itr->second.column_family, key, Slice());
|
2014-09-09 02:00:43 +02:00
|
|
|
if (batch.GetDataSize() >= kWriteOutEveryBytes) {
|
|
|
|
Status s = Write(write_options, &batch);
|
|
|
|
batch.Clear();
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
}
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// see above for format
|
|
|
|
std::string data_key;
|
|
|
|
PutFixed64BigEndian(&data_key, id);
|
|
|
|
std::string data_value;
|
|
|
|
PutLengthPrefixedSlice(&data_value, blob);
|
|
|
|
feature_set.Serialize(&data_value);
|
|
|
|
batch.Put(data_column_family_, data_key, data_value);
|
|
|
|
|
|
|
|
return Write(write_options, &batch);
|
|
|
|
}
|
|
|
|
|
2014-11-13 22:34:29 +01:00
|
|
|
virtual Status Compact(int num_threads) override {
|
|
|
|
std::vector<ColumnFamilyHandle*> column_families;
|
|
|
|
column_families.push_back(data_column_family_);
|
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
for (auto& iter : name_to_index_) {
|
2014-11-13 22:34:29 +01:00
|
|
|
column_families.push_back(iter.second.column_family);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2014-11-13 22:34:29 +01:00
|
|
|
|
|
|
|
std::mutex state_mutex;
|
|
|
|
std::condition_variable cv;
|
|
|
|
Status s;
|
|
|
|
int threads_running = 0;
|
|
|
|
|
2017-02-06 23:43:55 +01:00
|
|
|
std::vector<port::Thread> threads;
|
2014-11-13 22:34:29 +01:00
|
|
|
|
|
|
|
for (auto cfh : column_families) {
|
|
|
|
threads.emplace_back([&, cfh] {
|
|
|
|
{
|
|
|
|
std::unique_lock<std::mutex> lk(state_mutex);
|
|
|
|
cv.wait(lk, [&] { return threads_running < num_threads; });
|
|
|
|
threads_running++;
|
|
|
|
}
|
|
|
|
|
|
|
|
Status t = Flush(FlushOptions(), cfh);
|
|
|
|
if (t.ok()) {
|
2015-06-17 23:36:14 +02:00
|
|
|
t = CompactRange(CompactRangeOptions(), cfh, nullptr, nullptr);
|
2014-11-13 22:34:29 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
{
|
|
|
|
std::unique_lock<std::mutex> lk(state_mutex);
|
|
|
|
threads_running--;
|
|
|
|
if (s.ok() && !t.ok()) {
|
|
|
|
s = t;
|
|
|
|
}
|
|
|
|
cv.notify_one();
|
|
|
|
}
|
|
|
|
});
|
2014-07-24 22:39:33 +02:00
|
|
|
}
|
2014-11-13 22:34:29 +01:00
|
|
|
|
|
|
|
for (auto& t : threads) {
|
|
|
|
t.join();
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2014-11-13 22:34:29 +01:00
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
return s;
|
|
|
|
}
|
|
|
|
|
|
|
|
virtual Cursor* Query(const ReadOptions& read_options,
|
|
|
|
const BoundingBox<double>& bbox,
|
|
|
|
const std::string& spatial_index) override {
|
|
|
|
auto itr = name_to_index_.find(spatial_index);
|
|
|
|
if (itr == name_to_index_.end()) {
|
|
|
|
return new ErrorCursor(Status::InvalidArgument(
|
|
|
|
"Spatial index " + spatial_index + " not found"));
|
|
|
|
}
|
2014-08-26 02:50:18 +02:00
|
|
|
const auto& si = itr->second.index;
|
|
|
|
Iterator* spatial_iterator;
|
|
|
|
ValueGetter* value_getter;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
2014-08-26 02:50:18 +02:00
|
|
|
if (read_only_) {
|
|
|
|
spatial_iterator = NewIterator(read_options, itr->second.column_family);
|
|
|
|
value_getter = new ValueGetterFromDB(this, data_column_family_);
|
|
|
|
} else {
|
|
|
|
std::vector<Iterator*> iterators;
|
|
|
|
Status s = NewIterators(read_options,
|
|
|
|
{data_column_family_, itr->second.column_family},
|
|
|
|
&iterators);
|
|
|
|
if (!s.ok()) {
|
|
|
|
return new ErrorCursor(s);
|
|
|
|
}
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
2014-08-26 02:50:18 +02:00
|
|
|
spatial_iterator = iterators[1];
|
|
|
|
value_getter = new ValueGetterFromIterator(iterators[0]);
|
|
|
|
}
|
|
|
|
return new SpatialIndexCursor(spatial_iterator, value_getter,
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
GetTileBoundingBox(si, bbox), si.tile_bits);
|
|
|
|
}
|
|
|
|
|
|
|
|
private:
|
|
|
|
ColumnFamilyHandle* data_column_family_;
|
|
|
|
struct IndexColumnFamily {
|
|
|
|
SpatialIndexOptions index;
|
|
|
|
ColumnFamilyHandle* column_family;
|
|
|
|
IndexColumnFamily(const SpatialIndexOptions& _index,
|
|
|
|
ColumnFamilyHandle* _cf)
|
|
|
|
: index(_index), column_family(_cf) {}
|
|
|
|
};
|
|
|
|
// constant after construction!
|
|
|
|
std::unordered_map<std::string, IndexColumnFamily> name_to_index_;
|
|
|
|
|
|
|
|
std::atomic<uint64_t> next_id_;
|
2014-08-26 02:50:18 +02:00
|
|
|
bool read_only_;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
};
|
|
|
|
|
|
|
|
namespace {
|
2015-05-11 23:51:51 +02:00
|
|
|
DBOptions GetDBOptionsFromSpatialDBOptions(const SpatialDBOptions& options) {
|
2014-08-26 02:50:18 +02:00
|
|
|
DBOptions db_options;
|
2014-09-09 02:00:43 +02:00
|
|
|
db_options.max_open_files = 50000;
|
2014-08-28 16:27:26 +02:00
|
|
|
db_options.max_background_compactions = 3 * options.num_threads / 4;
|
|
|
|
db_options.max_background_flushes =
|
|
|
|
options.num_threads - db_options.max_background_compactions;
|
|
|
|
db_options.env->SetBackgroundThreads(db_options.max_background_compactions,
|
|
|
|
Env::LOW);
|
|
|
|
db_options.env->SetBackgroundThreads(db_options.max_background_flushes,
|
|
|
|
Env::HIGH);
|
2014-09-09 02:00:43 +02:00
|
|
|
db_options.statistics = CreateDBStatistics();
|
2014-08-26 02:50:18 +02:00
|
|
|
if (options.bulk_load) {
|
2014-09-09 02:00:43 +02:00
|
|
|
db_options.stats_dump_period_sec = 600;
|
|
|
|
} else {
|
|
|
|
db_options.stats_dump_period_sec = 1800; // 30min
|
2014-08-26 02:50:18 +02:00
|
|
|
}
|
|
|
|
return db_options;
|
|
|
|
}
|
|
|
|
|
2018-02-22 18:37:17 +01:00
|
|
|
ColumnFamilyOptions GetColumnFamilyOptions(const SpatialDBOptions& /*options*/,
|
2014-08-26 02:50:18 +02:00
|
|
|
std::shared_ptr<Cache> block_cache) {
|
|
|
|
ColumnFamilyOptions column_family_options;
|
2014-08-28 16:27:26 +02:00
|
|
|
column_family_options.write_buffer_size = 128 * 1024 * 1024; // 128MB
|
2014-08-27 16:56:10 +02:00
|
|
|
column_family_options.max_write_buffer_number = 4;
|
2014-09-09 02:00:43 +02:00
|
|
|
column_family_options.max_bytes_for_level_base = 256 * 1024 * 1024; // 256MB
|
|
|
|
column_family_options.target_file_size_base = 64 * 1024 * 1024; // 64MB
|
2014-08-28 16:27:26 +02:00
|
|
|
column_family_options.level0_file_num_compaction_trigger = 2;
|
|
|
|
column_family_options.level0_slowdown_writes_trigger = 16;
|
2016-04-07 18:02:28 +02:00
|
|
|
column_family_options.level0_stop_writes_trigger = 32;
|
2014-08-28 16:27:26 +02:00
|
|
|
// only compress levels >= 2
|
2014-08-26 02:50:18 +02:00
|
|
|
column_family_options.compression_per_level.resize(
|
|
|
|
column_family_options.num_levels);
|
|
|
|
for (int i = 0; i < column_family_options.num_levels; ++i) {
|
2014-08-28 16:27:26 +02:00
|
|
|
if (i < 2) {
|
2014-08-26 02:50:18 +02:00
|
|
|
column_family_options.compression_per_level[i] = kNoCompression;
|
2014-08-20 20:14:01 +02:00
|
|
|
} else {
|
2014-08-26 02:50:18 +02:00
|
|
|
column_family_options.compression_per_level[i] = kLZ4Compression;
|
2014-08-20 20:14:01 +02:00
|
|
|
}
|
|
|
|
}
|
2014-08-25 23:22:05 +02:00
|
|
|
BlockBasedTableOptions table_options;
|
2014-08-26 02:50:18 +02:00
|
|
|
table_options.block_cache = block_cache;
|
|
|
|
column_family_options.table_factory.reset(
|
|
|
|
NewBlockBasedTableFactory(table_options));
|
|
|
|
return column_family_options;
|
|
|
|
}
|
|
|
|
|
|
|
|
ColumnFamilyOptions OptimizeOptionsForDataColumnFamily(
|
|
|
|
ColumnFamilyOptions options, std::shared_ptr<Cache> block_cache) {
|
|
|
|
options.prefix_extractor.reset(NewNoopTransform());
|
|
|
|
BlockBasedTableOptions block_based_options;
|
|
|
|
block_based_options.index_type = BlockBasedTableOptions::kHashSearch;
|
|
|
|
block_based_options.block_cache = block_cache;
|
|
|
|
options.table_factory.reset(NewBlockBasedTableFactory(block_based_options));
|
|
|
|
return options;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2014-08-26 02:50:18 +02:00
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
} // namespace
|
|
|
|
|
2014-07-24 22:39:33 +02:00
|
|
|
class MetadataStorage {
|
|
|
|
public:
|
|
|
|
MetadataStorage(DB* db, ColumnFamilyHandle* cf) : db_(db), cf_(cf) {}
|
|
|
|
~MetadataStorage() {}
|
|
|
|
|
|
|
|
// format: <min_x double> <min_y double> <max_x double> <max_y double>
|
|
|
|
// <tile_bits varint32>
|
|
|
|
Status AddIndex(const SpatialIndexOptions& index) {
|
|
|
|
std::string encoded_index;
|
|
|
|
PutDouble(&encoded_index, index.bbox.min_x);
|
|
|
|
PutDouble(&encoded_index, index.bbox.min_y);
|
|
|
|
PutDouble(&encoded_index, index.bbox.max_x);
|
|
|
|
PutDouble(&encoded_index, index.bbox.max_y);
|
|
|
|
PutVarint32(&encoded_index, index.tile_bits);
|
|
|
|
return db_->Put(WriteOptions(), cf_,
|
|
|
|
GetSpatialIndexColumnFamilyName(index.name), encoded_index);
|
|
|
|
}
|
|
|
|
|
|
|
|
Status GetIndex(const std::string& name, SpatialIndexOptions* dst) {
|
|
|
|
std::string value;
|
|
|
|
Status s = db_->Get(ReadOptions(), cf_,
|
|
|
|
GetSpatialIndexColumnFamilyName(name), &value);
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
dst->name = name;
|
|
|
|
Slice encoded_index(value);
|
|
|
|
bool ok = GetDouble(&encoded_index, &(dst->bbox.min_x));
|
|
|
|
ok = ok && GetDouble(&encoded_index, &(dst->bbox.min_y));
|
|
|
|
ok = ok && GetDouble(&encoded_index, &(dst->bbox.max_x));
|
|
|
|
ok = ok && GetDouble(&encoded_index, &(dst->bbox.max_y));
|
|
|
|
ok = ok && GetVarint32(&encoded_index, &(dst->tile_bits));
|
|
|
|
return ok ? Status::OK() : Status::Corruption("Index encoding corrupted");
|
|
|
|
}
|
|
|
|
|
|
|
|
private:
|
|
|
|
DB* db_;
|
|
|
|
ColumnFamilyHandle* cf_;
|
|
|
|
};
|
|
|
|
|
|
|
|
Status SpatialDB::Create(
|
|
|
|
const SpatialDBOptions& options, const std::string& name,
|
|
|
|
const std::vector<SpatialIndexOptions>& spatial_indexes) {
|
2015-05-11 23:51:51 +02:00
|
|
|
DBOptions db_options = GetDBOptionsFromSpatialDBOptions(options);
|
2014-08-26 02:50:18 +02:00
|
|
|
db_options.create_if_missing = true;
|
|
|
|
db_options.create_missing_column_families = true;
|
|
|
|
db_options.error_if_exists = true;
|
|
|
|
|
|
|
|
auto block_cache = NewLRUCache(options.cache_size);
|
|
|
|
ColumnFamilyOptions column_family_options =
|
|
|
|
GetColumnFamilyOptions(options, block_cache);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
|
|
|
std::vector<ColumnFamilyDescriptor> column_families;
|
|
|
|
column_families.push_back(ColumnFamilyDescriptor(
|
2014-08-26 02:50:18 +02:00
|
|
|
kDefaultColumnFamilyName,
|
|
|
|
OptimizeOptionsForDataColumnFamily(column_family_options, block_cache)));
|
|
|
|
column_families.push_back(
|
|
|
|
ColumnFamilyDescriptor(kMetadataColumnFamilyName, column_family_options));
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
|
|
|
for (const auto& index : spatial_indexes) {
|
2014-07-24 22:39:33 +02:00
|
|
|
column_families.emplace_back(GetSpatialIndexColumnFamilyName(index.name),
|
2014-08-26 02:50:18 +02:00
|
|
|
column_family_options);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2014-07-24 22:39:33 +02:00
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
std::vector<ColumnFamilyHandle*> handles;
|
|
|
|
DB* base_db;
|
2014-08-26 02:50:18 +02:00
|
|
|
Status s = DB::Open(db_options, name, column_families, &handles, &base_db);
|
2014-07-24 22:39:33 +02:00
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
MetadataStorage metadata(base_db, handles[1]);
|
|
|
|
for (const auto& index : spatial_indexes) {
|
|
|
|
s = metadata.AddIndex(index);
|
|
|
|
if (!s.ok()) {
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
for (auto h : handles) {
|
|
|
|
delete h;
|
|
|
|
}
|
|
|
|
delete base_db;
|
|
|
|
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
|
|
|
|
Status SpatialDB::Open(const SpatialDBOptions& options, const std::string& name,
|
|
|
|
SpatialDB** db, bool read_only) {
|
2015-05-11 23:51:51 +02:00
|
|
|
DBOptions db_options = GetDBOptionsFromSpatialDBOptions(options);
|
2014-08-26 02:50:18 +02:00
|
|
|
auto block_cache = NewLRUCache(options.cache_size);
|
|
|
|
ColumnFamilyOptions column_family_options =
|
|
|
|
GetColumnFamilyOptions(options, block_cache);
|
2014-07-24 22:39:33 +02:00
|
|
|
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
Status s;
|
2014-07-24 22:39:33 +02:00
|
|
|
std::vector<std::string> existing_column_families;
|
|
|
|
std::vector<std::string> spatial_indexes;
|
2014-08-26 02:50:18 +02:00
|
|
|
s = DB::ListColumnFamilies(db_options, name, &existing_column_families);
|
2014-07-24 22:39:33 +02:00
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
for (const auto& cf_name : existing_column_families) {
|
|
|
|
Slice spatial_index;
|
|
|
|
if (GetSpatialIndexName(cf_name, &spatial_index)) {
|
|
|
|
spatial_indexes.emplace_back(spatial_index.data(), spatial_index.size());
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
std::vector<ColumnFamilyDescriptor> column_families;
|
|
|
|
column_families.push_back(ColumnFamilyDescriptor(
|
2014-08-26 02:50:18 +02:00
|
|
|
kDefaultColumnFamilyName,
|
|
|
|
OptimizeOptionsForDataColumnFamily(column_family_options, block_cache)));
|
|
|
|
column_families.push_back(
|
|
|
|
ColumnFamilyDescriptor(kMetadataColumnFamilyName, column_family_options));
|
2014-07-24 22:39:33 +02:00
|
|
|
|
|
|
|
for (const auto& index : spatial_indexes) {
|
|
|
|
column_families.emplace_back(GetSpatialIndexColumnFamilyName(index),
|
2014-08-26 02:50:18 +02:00
|
|
|
column_family_options);
|
2014-07-24 22:39:33 +02:00
|
|
|
}
|
|
|
|
std::vector<ColumnFamilyHandle*> handles;
|
|
|
|
DB* base_db;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
if (read_only) {
|
2014-08-26 02:50:18 +02:00
|
|
|
s = DB::OpenForReadOnly(db_options, name, column_families, &handles,
|
|
|
|
&base_db);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
} else {
|
2014-08-26 02:50:18 +02:00
|
|
|
s = DB::Open(db_options, name, column_families, &handles, &base_db);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
|
2014-07-24 22:39:33 +02:00
|
|
|
MetadataStorage metadata(base_db, handles[1]);
|
|
|
|
|
2014-07-25 02:26:48 +02:00
|
|
|
std::vector<std::pair<SpatialIndexOptions, ColumnFamilyHandle*>> index_cf;
|
2014-07-24 22:39:33 +02:00
|
|
|
assert(handles.size() == spatial_indexes.size() + 2);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
for (size_t i = 0; i < spatial_indexes.size(); ++i) {
|
2014-07-24 22:39:33 +02:00
|
|
|
SpatialIndexOptions index_options;
|
|
|
|
s = metadata.GetIndex(spatial_indexes[i], &index_options);
|
|
|
|
if (!s.ok()) {
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
index_cf.emplace_back(index_options, handles[i + 2]);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
2014-07-25 19:29:50 +02:00
|
|
|
uint64_t next_id = 1;
|
2014-07-24 22:39:33 +02:00
|
|
|
if (s.ok()) {
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
// find next_id
|
|
|
|
Iterator* iter = base_db->NewIterator(ReadOptions(), handles[0]);
|
|
|
|
iter->SeekToLast();
|
|
|
|
if (iter->Valid()) {
|
2014-07-24 22:45:26 +02:00
|
|
|
uint64_t last_id = 0;
|
|
|
|
if (!GetFixed64BigEndian(iter->key(), &last_id)) {
|
2014-07-24 22:39:33 +02:00
|
|
|
s = Status::Corruption("Invalid key in data column family");
|
2014-07-24 22:45:26 +02:00
|
|
|
} else {
|
|
|
|
next_id = last_id + 1;
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
delete iter;
|
|
|
|
}
|
2014-07-24 22:39:33 +02:00
|
|
|
if (!s.ok()) {
|
|
|
|
for (auto h : handles) {
|
|
|
|
delete h;
|
|
|
|
}
|
2014-08-26 02:50:18 +02:00
|
|
|
delete base_db;
|
2014-07-24 22:39:33 +02:00
|
|
|
return s;
|
|
|
|
}
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
|
2014-07-24 22:39:33 +02:00
|
|
|
// I don't need metadata column family any more, so delete it
|
|
|
|
delete handles[1];
|
2014-08-26 02:50:18 +02:00
|
|
|
*db = new SpatialDBImpl(base_db, handles[0], index_cf, next_id, read_only);
|
SpatialDB
Summary:
This diff is adding spatial index support to RocksDB.
When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels.
Each element inserted into SpatialDB has:
* a bounding box, which determines how will the element be indexed
* string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text)
* feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar)
* a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level).
Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements.
Test Plan: Added bunch of unit tests in spatial_db_test
Reviewers: dhruba, yinwang
Reviewed By: yinwang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
|
|
|
return Status::OK();
|
|
|
|
}
|
|
|
|
|
|
|
|
} // namespace spatial
|
|
|
|
} // namespace rocksdb
|
|
|
|
#endif // ROCKSDB_LITE
|