Commit Graph

483 Commits

Author SHA1 Message Date
Andrea Cavalli
28b4fdee50 Better errors logging, avoid zero-bytes bug in keyMayExist 2022-03-18 15:33:54 +01:00
Andrea Cavalli
ba3765eece Improve direct buffer support 2022-03-16 19:19:26 +01:00
Andrea Cavalli
0a6a0657a3 Use official netty 5 package 2022-03-16 13:47:56 +01:00
Andrea Cavalli
2b81006d56 Bypass groupBy bug and other small improvements 2022-03-13 11:01:51 +01:00
Andrea Cavalli
a4df72fe46 Improve performance 2022-03-12 02:55:18 +01:00
Andrea Cavalli
9b5071c45e Optional atomicity of addDocuments 2022-03-12 00:22:41 +01:00
Andrea Cavalli
4a2d143135 Various bugfixes 2022-03-11 17:59:46 +01:00
Andrea Cavalli
16f6025b30 Fix options 2022-03-10 02:38:57 +01:00
Andrea Cavalli
325457dd44 Bugfix 2022-03-10 01:43:37 +01:00
Andrea Cavalli
faa7118b8e Use buffer api 2022-03-09 02:29:38 +01:00
Andrea Cavalli
48f3a54e72 Partial RPC implementation 2022-03-05 15:46:40 +01:00
Andrea Cavalli
77af845a8a Remove unneeded parameter 2022-03-02 18:33:58 +01:00
Andrea Cavalli
ed37a769e2 Partial server implementation 2022-03-02 12:34:30 +01:00
Andrea Cavalli
53519fbc4e Use WriteBatch 2022-02-28 00:47:44 +01:00
Andrea Cavalli
86263af6f7 Create rocksdb directory 2022-02-26 22:51:22 +01:00
Andrea Cavalli
743919b831 Code cleanup 2022-02-26 03:28:20 +01:00
Andrea Cavalli
85642621df Use BytesRef when possible 2022-02-25 15:46:32 +01:00
Andrea Cavalli
cf61636141 Update rocksdb 2022-02-11 12:31:03 +01:00
Andrea Cavalli
f9335d890f Fix kExists 2022-02-11 12:19:32 +01:00
Andrea Cavalli
8e0d806d2b Update defaults 2022-02-04 01:55:36 +01:00
Andrea Cavalli
7b66259da6 Bugfix 2022-02-02 23:30:21 +01:00
Andrea Cavalli
dfe8361e19 Rename query rewrite class 2022-01-28 21:12:10 +01:00
Andrea Cavalli
58943b5e08 Simplify query transformations 2022-01-28 19:31:25 +01:00
Andrea Cavalli
5c0434c73f Fix compilation issue 2022-01-26 21:45:41 +01:00
Andrea Cavalli
5615c8019a Update containsKey 2022-01-26 21:30:08 +01:00
Andrea Cavalli
574a35907d Further optimizations 2022-01-26 21:18:43 +01:00
Andrea Cavalli
fb19a7a9f3 Optimize some methods 2022-01-26 19:56:51 +01:00
Andrea Cavalli
95afa6f9dd Clean database code 2022-01-26 19:03:51 +01:00
Andrea Cavalli
cdb65b31f3 Various local dict optimizations, customize fillCache in containsRange 2022-01-26 16:06:15 +01:00
Andrea Cavalli
cf53eb4f5a Use global hooks 2022-01-26 15:03:23 +01:00
Andrea Cavalli
4f19d2fff3 Add memory stats method 2022-01-15 20:00:10 +01:00
Andrea Cavalli
d2abe044cc Rethink rocksdb settings 2022-01-12 16:18:31 +01:00
Andrea Cavalli
aad377acb0 Change default values 2022-01-10 22:59:36 +01:00
Andrea Cavalli
ce97f33873 Fix infostream 2021-12-31 01:00:32 +01:00
Andrea Cavalli
57bcc92512 Fix statistics 2021-12-31 00:58:47 +01:00
Andrea Cavalli
d6244ab18d Rewind and clear the buffers 2021-12-30 22:29:17 +01:00
Andrea Cavalli
64f21409f1 Performance optimization 2021-12-30 22:29:06 +01:00
Andrea Cavalli
b493decd64 Fix metric names 2021-12-30 18:20:56 +01:00
Andrea Cavalli
68d8b5240c Implement more micrometer metrics 2021-12-30 17:28:06 +01:00
Andrea Cavalli
5769bc7076 Print errored key 2021-12-29 00:31:35 +01:00
Andrea Cavalli
24493eb4ff Change default timeout 2021-12-27 23:07:11 +01:00
Andrea Cavalli
7993a6210b Fix pessimistic db 2021-12-27 18:44:54 +01:00
Andrea Cavalli
dffb8eb3eb Set default compression types 2021-12-27 17:49:22 +01:00
Andrea Cavalli
9309692062 Configurable optimistic transactions 2021-12-27 17:45:52 +01:00
Andrea Cavalli
aa04a64c34 Fix logging, configurable compression 2021-12-27 17:34:44 +01:00
Andrea Cavalli
582813b6c7 Customizable volumes 2021-12-27 16:33:31 +01:00
Andrea Cavalli
3ae3bd7944 Fix bound 2021-12-26 20:24:02 +01:00
Andrea Cavalli
2b09f5a738 Update retry algorithm 2021-12-26 20:20:27 +01:00
Andrea Cavalli
83bc3ebb04 Update retry algorithm 2021-12-26 12:47:00 +01:00
Andrea Cavalli
a1eec93c64 Rewrite unsorted lucene queries 2021-12-23 02:13:51 +01:00
Andrea Cavalli
eaef75a304 Close LMDB databases after each full search 2021-12-18 21:01:14 +01:00
Andrea Cavalli
0f62362d21 Disable reader pooling 2021-12-17 23:51:10 +01:00
Andrea Cavalli
638595f518 Pool LMDB databases 2021-12-17 23:12:35 +01:00
Andrea Cavalli
6d92ba8a68 Bugfixes 2021-12-17 03:04:01 +01:00
Andrea Cavalli
1a35930909 use a reentrantlock to avoid multiple merges at the same time 2021-12-17 02:19:51 +01:00
Andrea Cavalli
1dffb55572 Use log4j for logging, rewrite some local dictionary parts 2021-12-17 01:48:49 +01:00
Andrea Cavalli
7a712722d7 Code cleanup 2021-12-16 16:34:20 +01:00
Andrea Cavalli
6e312fe102 Use uninterruptible scheduler for lucene searches 2021-12-16 16:14:44 +01:00
Andrea Cavalli
01099cc4d1 Fix unscored searcher 2021-12-16 02:38:56 +01:00
Andrea Cavalli
b7ca57a215 Schedule commits and merges 2021-12-15 16:47:59 +01:00
Andrea Cavalli
8ad622db0a Reduce the number of threads 2021-12-15 16:04:33 +01:00
Andrea Cavalli
d58d696ca4 Bugfixes 2021-12-13 01:57:37 +01:00
Andrea Cavalli
907561d93c Add requests timeout 2021-12-12 23:40:30 +01:00
Andrea Cavalli
3cdafd748e Optionally disable LMDB 2021-12-12 16:43:22 +01:00
Andrea Cavalli
297c249243 Code cleanup 2021-12-12 16:19:50 +01:00
Andrea Cavalli
2a5e90d667 Update rocksdb to 6.26.1, Update netty 2021-12-12 02:17:36 +01:00
Andrea Cavalli
6644e040dd Fix unsorted unscored streaming multi searcher 2021-12-08 11:58:06 +01:00
Andrea Cavalli
044d189600 Optimize single-shard indices 2021-11-21 12:31:23 +01:00
Andrea Cavalli
798b8a5288 Update policies 2021-11-20 01:30:06 +01:00
Andrea Cavalli
29d9aad8bf Update buckets 2021-11-19 19:03:31 +01:00
Andrea Cavalli
ed00d474d6 Implement numeric buckets collector 2021-11-18 17:13:53 +01:00
Andrea Cavalli
c1c57388e5 Add standard lucene queries 2021-11-16 23:19:23 +01:00
Andrea Cavalli
23d5f700fb Add update method to single values 2021-11-12 02:05:44 +01:00
Andrea Cavalli
42c4b6e651 Add a discard hook 2021-11-09 15:57:16 +01:00
Andrea Cavalli
d4dae4667d Remove stream executor 2021-11-09 02:14:21 +01:00
Andrea Cavalli
81b7df8702 Remove sends to improve performance 2021-11-08 16:33:41 +01:00
Andrea Cavalli
232e46bcea Add more discard handlers 2021-11-08 12:06:32 +01:00
Andrea Cavalli
59aa1ef5c6 Update database 2021-11-08 11:17:52 +01:00
Andrea Cavalli
63cd178988 Use merge operator when possible 2021-11-08 10:49:59 +01:00
Andrea Cavalli
b48ab0b095 Add soft update document 2021-11-07 18:34:34 +01:00
Andrea Cavalli
b34b620082 Allow lucene partial document updates 2021-11-07 17:46:40 +01:00
Andrea Cavalli
d8801b3471 Update loggers 2021-11-07 14:46:03 +01:00
Andrea Cavalli
eba6f92d7f Remove warning 2021-11-05 16:54:58 +01:00
Andrea Cavalli
c46320cdf7 Adjust database parameters 2021-11-02 13:02:49 +01:00
Andrea Cavalli
c506a7e71b Use netty direct memory if possible 2021-10-30 12:39:56 +02:00
Andrea Cavalli
98dcb39ce1 Reformat iterator 2021-10-30 12:21:28 +02:00
Andrea Cavalli
1ea004630e Use Micrometer registry, better optimistic commit failure logging 2021-10-30 11:13:46 +02:00
Andrea Cavalli
f3c63842c8 Don't close unreleasable buffers 2021-10-29 00:11:25 +02:00
Andrea Cavalli
37de47fd6c Refactor results 2021-10-28 23:57:23 +02:00
Andrea Cavalli
417c174761 Implement ResourceSupport 2021-10-28 11:44:20 +02:00
Andrea Cavalli
e24dc4bece Update dependencies 2021-10-26 23:59:55 +02:00
Andrea Cavalli
da2b302d1b Replace sinks with references when needed 2021-10-26 00:02:08 +02:00
Andrea Cavalli
db9b1bb7fb Rename constants 2021-10-25 00:42:48 +02:00
Andrea Cavalli
fb43fa78af Discard transaction data 2021-10-21 10:01:29 +02:00
Andrea Cavalli
1f04241ac6 Use optimistic db 2021-10-21 10:00:39 +02:00
Andrea Cavalli
1625a5c44b RocksDB database abstraction 2021-10-20 01:51:34 +02:00
Andrea Cavalli
80d0ced888 Optimistic transactions, inline codecs 2021-10-19 00:22:05 +02:00
Andrea Cavalli
aad5f8c96c Optimistic updates 2021-10-17 19:52:43 +02:00
Andrea Cavalli
df2b757fdd Remove unused class 2021-10-17 17:15:57 +02:00
Andrea Cavalli
3a591c38f2 Use new netty resource leak detector 2021-10-16 16:56:10 +02:00
Andrea Cavalli
e66efba834 Publish lucene 9 snapshots 2021-10-16 14:59:38 +02:00
Andrea Cavalli
f77784fc50 Improve temporary LMDB database lifecycle 2021-10-16 14:35:04 +02:00
Andrea Cavalli
6f8f70c23b Bugfix in sizeRange 2021-10-16 01:49:41 +02:00
Andrea Cavalli
584115c5cc Rename classes and improve readonly tests performance 2021-10-13 12:25:32 +02:00
Andrea Cavalli
09f60a3a99 Implement memory-mapped sorted searcher for streaming huge query results 2021-10-13 00:23:56 +02:00
Andrea Cavalli
33e4cb2f14 Add ensureOpen method to avoid writing to the index when closing 2021-10-07 00:53:38 +02:00
Andrea Cavalli
818bd20ed9 Update netty 2021-10-01 19:17:33 +02:00
Andrea Cavalli
6b8c1025d1 Fix ClassCastException 2021-09-29 11:47:17 +02:00
Andrea Cavalli
74d20204ab Avoid using BufferTimeoutPublisher (it kills lucene Async IO) 2021-09-29 11:38:53 +02:00
Andrea Cavalli
d96b5a168b Execute searches on its own executor 2021-09-25 13:06:24 +02:00
Andrea Cavalli
996165c3cc Do not release cached searcher 2021-09-24 02:47:52 +02:00
Andrea Cavalli
23fa46c775 Release cached searcher 2021-09-24 02:44:12 +02:00
Andrea Cavalli
20c19f662b Avoid multiple drops 2021-09-24 02:01:45 +02:00
Andrea Cavalli
3a544d4297 Partial migration to ResourceSupport 2021-09-23 20:57:28 +02:00
Andrea Cavalli
6c849fae78 Change makeInaccessible visibility 2021-09-23 15:34:56 +02:00
Andrea Cavalli
b77b441515 Fix all errors 2021-09-23 11:30:44 +02:00
Andrea Cavalli
29086b1939 Add some tests for low level dictionary, log tests, pass more tests 2021-09-23 02:15:58 +02:00
Andrea Cavalli
e034f3b778 Support memory segments 2021-09-22 18:33:28 +02:00
Andrea Cavalli
8b73a05177 Bugfixes and upgraded to java 17 2021-09-22 11:03:39 +02:00
Andrea Cavalli
0fee105f0b Bugfixes 2021-09-21 02:01:02 +02:00
Andrea Cavalli
bd97704a0b Better page limits 2021-09-20 18:20:59 +02:00
Andrea Cavalli
4a883ca8ea Done implementing Local and LocalMulti lucene indices 2021-09-20 11:52:21 +02:00
Andrea Cavalli
ca37d1fb68 Rewrite lucene transformers and implement MoreLikeThis sharding 2021-09-20 11:35:01 +02:00
Andrea Cavalli
65db1711b5 Missing: moreLikeThis 2021-09-19 19:59:37 +02:00
Andrea Cavalli
8bc0284f27 First safe search commit 2021-09-18 18:34:21 +02:00
Andrea Cavalli
6782abb66d Use Net5 2021-09-17 16:56:28 +02:00
Andrea Cavalli
7feeb4a9ce Use markers 2021-09-10 12:13:52 +02:00
Andrea Cavalli
0a378bc0f1 Fix negative value 2021-09-10 12:13:46 +02:00
Andrea Cavalli
cb5a5e36f2 Fix timeout subscription 2021-09-09 11:43:37 +02:00
Andrea Cavalli
48db79359b Use boundedelastic db scheduler 2021-09-08 23:51:05 +02:00
Andrea Cavalli
5293fbe22c Close leaked data 2021-09-08 23:41:18 +02:00
Andrea Cavalli
09c7e4f730 Performance bugfix 2021-09-08 21:34:52 +02:00
Andrea Cavalli
047a471bf7 Fix last leaks 2021-09-08 00:22:39 +02:00
Andrea Cavalli
57a0206cf1 Fully implement memory dictionary 2021-09-07 19:32:37 +02:00
Andrea Cavalli
a662033228 Less verbose exception 2021-09-07 11:39:13 +02:00
Andrea Cavalli
dd1ed1d18f Suppress warning 2021-09-07 11:29:28 +02:00
Andrea Cavalli
d4ed99cc24 Fix compilation issue 2021-09-07 11:28:03 +02:00
Andrea Cavalli
717d13ef1f Revert commit 1882e8b300 2021-09-07 11:26:10 +02:00
Andrea Cavalli
66fa853272 Avoid memory leaks 2021-09-07 02:36:11 +02:00
Andrea Cavalli
936c07406e Implement more phasers 2021-09-06 18:52:21 +02:00
Andrea Cavalli
51b60168f7 Implement phaser 2021-09-06 18:29:10 +02:00
Andrea Cavalli
d05994c8ff Rename manager 2021-09-06 18:24:36 +02:00
Andrea Cavalli
5a79d2d7cb Close the CachedIndexSearcher automatically 2021-09-06 18:23:47 +02:00
Andrea Cavalli
7230e96343 Bugfix 2021-09-06 17:42:12 +02:00
Andrea Cavalli
e59705bd0b Better manage cached index searchers 2021-09-06 17:35:02 +02:00
Andrea Cavalli
b7d975ccfb MultiGet window set to 16 to avoid memory problems 2021-09-06 15:54:44 +02:00
Andrea Cavalli
16bd3b263b Limit cached indexwriters 2021-09-06 15:08:07 +02:00
Andrea Cavalli
d5964a7bed Pool searchers 2021-09-06 15:06:51 +02:00
Andrea Cavalli
4c348a6b2f Throw exception if running in a nonblocking scope 2021-09-05 14:23:46 +02:00
Andrea Cavalli
1882e8b300 Optimize performance by removing all superfluous subscribeOn calls 2021-09-04 16:42:47 +02:00
Andrea Cavalli
f5f46edbec Use netty5 package 2021-09-04 02:19:10 +02:00
Andrea Cavalli
154086dee2 Fix compilation issues and buffer expansions 2021-09-03 16:11:14 +02:00
Andrea Cavalli
47de651c94 Bugfixes 2021-09-03 02:22:55 +02:00
Andrea Cavalli
a8028024c8 Finish refactoring 2021-09-02 21:14:26 +02:00
Andrea Cavalli
3091b81d34 Fix map tests 2021-09-02 17:15:40 +02:00
Andrea Cavalli
ff7823656e y 2021-09-01 00:01:56 +02:00
Andrea Cavalli
013d26387d (unfinished) Netty 5 refactoring 2021-08-31 15:50:11 +02:00
Andrea Cavalli
0faef5316e (unfinished) Netty 5 refactoring 2021-08-31 09:14:46 +02:00
Andrea Cavalli
3b55e8bd24 (unfinished) Netty 5 refactoring 2021-08-29 23:18:03 +02:00
Andrea Cavalli
9984cfff73 Bugfixes 2021-08-29 01:15:51 +02:00
Andrea Cavalli
03b5876001 Bugfixes 2021-08-28 22:42:51 +02:00
Andrea Cavalli
0c17af2ae5 Bugfixes 2021-08-27 02:49:51 +02:00
Andrea Cavalli
c5552d2827 Bugfixes 2021-08-22 23:50:50 +02:00
Andrea Cavalli
2a24570512 Add checked serialization exception 2021-08-22 21:23:22 +02:00
bc759c344d Use a new approach to completely avoid memory leaks 2021-08-22 18:20:05 +02:00
6ed31ab2a6 Code cleanup 2021-08-16 10:36:54 +02:00
435e7d4886 Use a new approach to completely avoid memory leaks 2021-08-16 10:27:47 +02:00
Andrea Cavalli
9734d78839 Total hits with precision 2021-08-04 01:12:39 +02:00
Andrea Cavalli
bcd99f4727 Avoid errors 2021-08-04 00:44:06 +02:00
Andrea Cavalli
586be8751c Fix compilation issue 2021-07-31 19:18:20 +02:00
Andrea Cavalli
d411675c2b Use flatMapIterable when possible 2021-07-31 18:00:53 +02:00
Andrea Cavalli
7597e54bac Remove unnecessary schedulers 2021-07-31 12:28:53 +02:00
Andrea Cavalli
b816e1f9e5 Use buffertimeoutpublisher 2021-07-27 01:31:18 +02:00
Andrea Cavalli
32b514c866 Add max open files option 2021-07-24 23:47:51 +02:00
Andrea Cavalli
1df38022fb Fix max open files with direct I/O 2021-07-23 22:43:24 +02:00
Andrea Cavalli
abde1d1aab Fix getMulti 2021-07-23 15:20:33 +02:00
Andrea Cavalli
aa1aa7a6fb Test more numbers 2021-07-18 19:37:24 +02:00
Andrea Cavalli
43439c6f10 Optimize performance 2021-07-17 11:52:08 +02:00
Andrea Cavalli
e4bbeeca3a Fix messages query not releasing 2021-07-15 15:54:34 +02:00
Andrea Cavalli
58a9121978 Add heap kv database implementation 2021-07-10 20:52:01 +02:00
Andrea Cavalli
c85dcfb54e Clean rocksdb code 2021-07-06 22:27:03 +02:00
Andrea Cavalli
2a47e6f3fd Add lucene options 2021-07-06 14:33:47 +02:00
Andrea Cavalli
e7e25e646d Fix scheduling 2021-07-06 02:23:06 +02:00
Andrea Cavalli
6c97b0f068 Bugfixes 2021-07-06 01:30:37 +02:00
Andrea Cavalli
b5ccf315be Implement adaptive multi searcher for shard indexes 2021-07-06 00:30:14 +02:00
Andrea Cavalli
00ff36836e Improve reactive searchers performance 2021-07-05 15:43:19 +02:00
Andrea Cavalli
a5d4584a11 Adaptive reactive lucene search engine, lazy results 2021-07-05 12:05:45 +02:00
Andrea Cavalli
7929f0dc8c Fully reactive lucene queries 2021-07-04 01:34:17 +02:00
Andrea Cavalli
8a1e4028f7 Advanced lucene options 2021-07-01 21:19:52 +02:00
Andrea Cavalli
b80feabb97 Use NIOFSDirectory 2021-06-30 23:43:46 +02:00
Andrea Cavalli
3758d06853 Configurable direct buffers 2021-06-29 23:31:02 +02:00
Andrea Cavalli
2df2c00e36 Fix unmodifiable read options 2021-06-27 16:52:45 +02:00
Andrea Cavalli
cb8dbf2fa2 Fix direct I/O parameters 2021-06-27 16:47:59 +02:00
Andrea Cavalli
c9e9cb8a01 Bugfix 2021-06-27 16:33:23 +02:00
Andrea Cavalli
c6d5beb33b Database options record 2021-06-27 15:40:56 +02:00
Andrea Cavalli
507101e453 Add verifyChecksum method 2021-06-27 15:06:48 +02:00
Andrea Cavalli
1d281d0305 Prevent reading corrupted database parts, reimplement badblocks 2021-06-27 11:58:12 +02:00
Andrea Cavalli
bd8755c180 Implement "badblocks" method 2021-06-26 02:35:33 +02:00
Andrea Cavalli
ee05614115 Use Direct I/O when available 2021-06-25 23:47:53 +02:00
Andrea Cavalli
bd62b83d57 Bugfix 2021-06-25 20:25:57 +02:00