Commit Graph

654 Commits

Author SHA1 Message Date
Andrea Cavalli
08ef14fb76 Typo 2022-01-17 18:12:12 +01:00
Andrea Cavalli
4f19d2fff3 Add memory stats method 2022-01-15 20:00:10 +01:00
Andrea Cavalli
d2abe044cc Rethink rocksdb settings 2022-01-12 16:18:31 +01:00
Andrea Cavalli
ce506f22dd Bugfix 2022-01-11 22:28:42 +01:00
Andrea Cavalli
4e5e4423ff Rewrite the custom analyzers 2022-01-11 22:23:07 +01:00
Andrea Cavalli
aad377acb0 Change default values 2022-01-10 22:59:36 +01:00
Andrea Cavalli
84dea46a35 Update logging 2022-01-04 12:55:33 +01:00
Andrea Cavalli
ce97f33873 Fix infostream 2021-12-31 01:00:32 +01:00
Andrea Cavalli
57bcc92512 Fix statistics 2021-12-31 00:58:47 +01:00
Andrea Cavalli
d6244ab18d Rewind and clear the buffers 2021-12-30 22:29:17 +01:00
Andrea Cavalli
64f21409f1 Performance optimization 2021-12-30 22:29:06 +01:00
Andrea Cavalli
b493decd64 Fix metric names 2021-12-30 18:20:56 +01:00
Andrea Cavalli
68d8b5240c Implement more micrometer metrics 2021-12-30 17:28:06 +01:00
Andrea Cavalli
5769bc7076 Print errored key 2021-12-29 00:31:35 +01:00
Andrea Cavalli
24493eb4ff Change default timeout 2021-12-27 23:07:11 +01:00
Andrea Cavalli
7993a6210b Fix pessimistic db 2021-12-27 18:44:54 +01:00
Andrea Cavalli
dffb8eb3eb Set default compression types 2021-12-27 17:49:22 +01:00
Andrea Cavalli
9309692062 Configurable optimistic transactions 2021-12-27 17:45:52 +01:00
Andrea Cavalli
aa04a64c34 Fix logging, configurable compression 2021-12-27 17:34:44 +01:00
Andrea Cavalli
582813b6c7 Customizable volumes 2021-12-27 16:33:31 +01:00
Andrea Cavalli
3ae3bd7944 Fix bound 2021-12-26 20:24:02 +01:00
Andrea Cavalli
2b09f5a738 Update retry algorithm 2021-12-26 20:20:27 +01:00
Andrea Cavalli
83bc3ebb04 Update retry algorithm 2021-12-26 12:47:00 +01:00
Andrea Cavalli
03bbf3baa9 Bugfix 2021-12-23 02:20:40 +01:00
Andrea Cavalli
a1eec93c64 Rewrite unsorted lucene queries 2021-12-23 02:13:51 +01:00
Andrea Cavalli
a5666dd5b4 Increase default timeout 2021-12-20 16:38:58 +01:00
Andrea Cavalli
eaef75a304 Close LMDB databases after each full search 2021-12-18 21:01:14 +01:00
Andrea Cavalli
480ab77db8 Use sorted maps 2021-12-18 18:16:56 +01:00
Andrea Cavalli
1de5e52ffd Use linkedhashmap 2021-12-18 15:15:19 +01:00
Andrea Cavalli
9952eaffc0 Using search instead of count is too heavy 2021-12-18 00:18:24 +01:00
Andrea Cavalli
0f62362d21 Disable reader pooling 2021-12-17 23:51:10 +01:00
Andrea Cavalli
638595f518 Pool LMDB databases 2021-12-17 23:12:35 +01:00
Andrea Cavalli
0e9c8c089e Bugfixes 2021-12-17 16:24:18 +01:00
Andrea Cavalli
6d92ba8a68 Bugfixes 2021-12-17 03:04:01 +01:00
Andrea Cavalli
1a35930909 use a reentrantlock to avoid multiple merges at the same time 2021-12-17 02:19:51 +01:00
Andrea Cavalli
1dffb55572 Use log4j for logging, rewrite some local dictionary parts 2021-12-17 01:48:49 +01:00
Andrea Cavalli
7a712722d7 Code cleanup 2021-12-16 16:34:20 +01:00
Andrea Cavalli
6e312fe102 Use uninterruptible scheduler for lucene searches 2021-12-16 16:14:44 +01:00
Andrea Cavalli
2e1678373c Bugfix 2021-12-16 03:00:51 +01:00
Andrea Cavalli
01099cc4d1 Fix unscored searcher 2021-12-16 02:38:56 +01:00
Andrea Cavalli
b7ca57a215 Schedule commits and merges 2021-12-15 16:47:59 +01:00
Andrea Cavalli
8ad622db0a Reduce the number of threads 2021-12-15 16:04:33 +01:00
Andrea Cavalli
d58d696ca4 Bugfixes 2021-12-13 01:57:37 +01:00
Andrea Cavalli
907561d93c Add requests timeout 2021-12-12 23:40:30 +01:00
Andrea Cavalli
5157656a2c Don't use official searcher for production queries 2021-12-12 18:52:50 +01:00
Andrea Cavalli
3cdafd748e Optionally disable LMDB 2021-12-12 16:43:22 +01:00
Andrea Cavalli
297c249243 Code cleanup 2021-12-12 16:19:50 +01:00
Andrea Cavalli
1a64d98697 Bugfixes 2021-12-12 02:40:26 +01:00
Andrea Cavalli
2a5e90d667 Update rocksdb to 6.26.1, Update netty 2021-12-12 02:17:36 +01:00
Andrea Cavalli
18b242d746 Update log4j and lucene 2021-12-11 13:20:28 +01:00
Andrea Cavalli
6644e040dd Fix unsorted unscored streaming multi searcher 2021-12-08 11:58:06 +01:00
Andrea Cavalli
b5aa8b4baa Use a LongSemaphore to avoid using a buffer 2021-11-29 23:27:55 +01:00
Andrea Cavalli
cc368aecc8 Fix deadlock 2021-11-29 14:15:31 +01:00
Andrea Cavalli
d8de969bee Fix deadlock 2021-11-29 14:01:57 +01:00
Andrea Cavalli
3d9247c969 Optimize unscored streaming searcher 2021-11-24 16:39:22 +01:00
Andrea Cavalli
044d189600 Optimize single-shard indices 2021-11-21 12:31:23 +01:00
Andrea Cavalli
7047b512fc Add numeric field 2021-11-20 16:09:00 +01:00
Andrea Cavalli
798b8a5288 Update policies 2021-11-20 01:30:06 +01:00
Andrea Cavalli
06d98040b1 Allow to use absolute values 2021-11-20 01:12:17 +01:00
Andrea Cavalli
3810c49fa1 Allow random sampling 2021-11-19 22:15:31 +01:00
Andrea Cavalli
29d9aad8bf Update buckets 2021-11-19 19:03:31 +01:00
Andrea Cavalli
ed00d474d6 Implement numeric buckets collector 2021-11-18 17:13:53 +01:00
Andrea Cavalli
e014266b8d Rename unclear parameters 2021-11-16 23:54:23 +01:00
Andrea Cavalli
c1c57388e5 Add standard lucene queries 2021-11-16 23:19:23 +01:00
Andrea Cavalli
3d7e80b4ec Update comparators 2021-11-16 23:19:13 +01:00
Andrea Cavalli
891255e18e Add more methods 2021-11-14 22:21:32 +01:00
Andrea Cavalli
f30ad372ed Bugfix 2021-11-12 02:52:42 +01:00
Andrea Cavalli
23d5f700fb Add update method to single values 2021-11-12 02:05:44 +01:00
Andrea Cavalli
42c4b6e651 Add a discard hook 2021-11-09 15:57:16 +01:00
Andrea Cavalli
d4dae4667d Remove stream executor 2021-11-09 02:14:21 +01:00
Andrea Cavalli
7fb8183c63 Update 2021-11-09 01:54:46 +01:00
Andrea Cavalli
adbbbaa646 Optimizations 2021-11-09 01:13:47 +01:00
Andrea Cavalli
50b3c897ff Add scoring shard multi manager 2021-11-09 00:54:09 +01:00
Andrea Cavalli
47aac33b22 Add count multi searcher 2021-11-09 00:05:26 +01:00
Andrea Cavalli
1ba4a1866f Reactive collector multi manager 2021-11-08 23:53:53 +01:00
Andrea Cavalli
7c0a4b2af9 Bugfix 2021-11-08 23:46:40 +01:00
Andrea Cavalli
a48bea6c63 Fix wait time 2021-11-08 23:46:02 +01:00
Andrea Cavalli
ef15bf7e8c Bugfixes 2021-11-08 18:52:52 +01:00
Andrea Cavalli
81b7df8702 Remove sends to improve performance 2021-11-08 16:33:41 +01:00
Andrea Cavalli
232e46bcea Add more discard handlers 2021-11-08 12:06:32 +01:00
Andrea Cavalli
59aa1ef5c6 Update database 2021-11-08 11:17:52 +01:00
Andrea Cavalli
63cd178988 Use merge operator when possible 2021-11-08 10:49:59 +01:00
Andrea Cavalli
b48ab0b095 Add soft update document 2021-11-07 18:34:34 +01:00
Andrea Cavalli
52be944b5f Bugfixes 2021-11-07 18:00:11 +01:00
Andrea Cavalli
b34b620082 Allow lucene partial document updates 2021-11-07 17:46:40 +01:00
Andrea Cavalli
d8801b3471 Update loggers 2021-11-07 14:46:03 +01:00
Andrea Cavalli
eba6f92d7f Remove warning 2021-11-05 16:54:58 +01:00
Andrea Cavalli
c46320cdf7 Adjust database parameters 2021-11-02 13:02:49 +01:00
Andrea Cavalli
e015c9afff Add todos 2021-10-30 17:43:59 +02:00
Andrea Cavalli
c506a7e71b Use netty direct memory if possible 2021-10-30 12:39:56 +02:00
Andrea Cavalli
98dcb39ce1 Reformat iterator 2021-10-30 12:21:28 +02:00
Andrea Cavalli
1ea004630e Use Micrometer registry, better optimistic commit failure logging 2021-10-30 11:13:46 +02:00
Andrea Cavalli
f3c63842c8 Don't close unreleasable buffers 2021-10-29 00:11:25 +02:00
Andrea Cavalli
37de47fd6c Refactor results 2021-10-28 23:57:23 +02:00
Andrea Cavalli
d861e26a57 Rename sort 2021-10-28 17:18:23 +02:00
Andrea Cavalli
417c174761 Implement ResourceSupport 2021-10-28 11:44:20 +02:00
Andrea Cavalli
e24dc4bece Update dependencies 2021-10-26 23:59:55 +02:00
Andrea Cavalli
da2b302d1b Replace sinks with references when needed 2021-10-26 00:02:08 +02:00
Andrea Cavalli
1899ef1723 Optimize array 2021-10-25 01:54:37 +02:00
Andrea Cavalli
f9b656986c Clean queues 2021-10-25 01:44:12 +02:00
Andrea Cavalli
7a01b6a2a4 Group writes together to speedup writes 2021-10-25 01:41:02 +02:00
Andrea Cavalli
e48f8b25eb Remove unused constants 2021-10-25 00:44:30 +02:00
Andrea Cavalli
db9b1bb7fb Rename constants 2021-10-25 00:42:48 +02:00
Andrea Cavalli
90aa7a2522 Use in-memory database if the limit is lower than 8192 2021-10-22 01:26:44 +02:00
Andrea Cavalli
a77af5253d Fix performance issue 2021-10-22 00:51:54 +02:00
Andrea Cavalli
213e446ff4 Separate lucene from the rest of the app to avoid thread interruptions 2021-10-21 11:55:59 +02:00
Andrea Cavalli
fb43fa78af Discard transaction data 2021-10-21 10:01:29 +02:00
Andrea Cavalli
1f04241ac6 Use optimistic db 2021-10-21 10:00:39 +02:00
Andrea Cavalli
1625a5c44b RocksDB database abstraction 2021-10-20 01:51:34 +02:00
Andrea Cavalli
80d0ced888 Optimistic transactions, inline codecs 2021-10-19 00:22:05 +02:00
Andrea Cavalli
aad5f8c96c Optimistic updates 2021-10-17 19:52:43 +02:00
Andrea Cavalli
df2b757fdd Remove unused class 2021-10-17 17:15:57 +02:00
Andrea Cavalli
3a591c38f2 Use new netty resource leak detector 2021-10-16 16:56:10 +02:00
Andrea Cavalli
e66efba834 Publish lucene 9 snapshots 2021-10-16 14:59:38 +02:00
Andrea Cavalli
f77784fc50 Improve temporary LMDB database lifecycle 2021-10-16 14:35:04 +02:00
Andrea Cavalli
a441c2fb85 Move comparator 2021-10-16 01:51:07 +02:00
Andrea Cavalli
6f8f70c23b Bugfix in sizeRange 2021-10-16 01:49:41 +02:00
Andrea Cavalli
fd565c80b2 Fix assertions 2021-10-16 01:35:09 +02:00
Andrea Cavalli
f6aab7dafb More tests 2021-10-16 00:50:06 +02:00
Andrea Cavalli
f91617c7ec Update tests 2021-10-16 00:32:04 +02:00
Andrea Cavalli
e6c0f14fda Implement some sort codecs 2021-10-15 22:03:53 +02:00
Andrea Cavalli
17a9b49755 Partial sorted implementation 2021-10-15 00:03:41 +02:00
Andrea Cavalli
36df18796b Partial sorted implementation 2021-10-14 23:04:34 +02:00
Andrea Cavalli
e1d1e1fb05 Fix LMDB PriorityQueue 2021-10-14 15:55:58 +02:00
Andrea Cavalli
d6b6a211a8 Replace scoreMode with boolean "complete" 2021-10-14 00:49:21 +02:00
Andrea Cavalli
5817d8a93f Implement more tests 2021-10-13 14:26:54 +02:00
Andrea Cavalli
c8536e45da Cleanup code 2021-10-13 13:16:56 +02:00
Andrea Cavalli
584115c5cc Rename classes and improve readonly tests performance 2021-10-13 12:25:32 +02:00
Andrea Cavalli
09f60a3a99 Implement memory-mapped sorted searcher for streaming huge query results 2021-10-13 00:23:56 +02:00
Andrea Cavalli
83e98ebce8 Fast implementation of query transformers 2021-10-08 11:50:28 +02:00
Andrea Cavalli
30a14a4aae Add todos 2021-10-08 02:14:34 +02:00
Andrea Cavalli
4d5f8b5b37 Implement MoreLikeThis transformer 2021-10-08 02:13:33 +02:00
Andrea Cavalli
81d8abe72c Add ensureBlocking in blocking method 2021-10-07 00:54:20 +02:00
Andrea Cavalli
013e4c5dd8 Remove publishOn in lucene utils 2021-10-07 00:54:11 +02:00
Andrea Cavalli
33e4cb2f14 Add ensureOpen method to avoid writing to the index when closing 2021-10-07 00:53:38 +02:00
Andrea Cavalli
818bd20ed9 Update netty 2021-10-01 19:17:33 +02:00
Andrea Cavalli
9646ddcb28 Use method references 2021-09-30 18:25:36 +02:00
Andrea Cavalli
5d2ba797eb Cleanup code 2021-09-30 00:25:09 +02:00
Andrea Cavalli
9a6f195b63 Cleanup utils 2021-09-30 00:23:41 +02:00
Andrea Cavalli
dc14192dca Reimplement MultiMoreLikeThis using BigCompositeReader 2021-09-29 13:18:06 +02:00
Andrea Cavalli
6b8c1025d1 Fix ClassCastException 2021-09-29 11:47:17 +02:00
Andrea Cavalli
74d20204ab Avoid using BufferTimeoutPublisher (it kills lucene Async IO) 2021-09-29 11:38:53 +02:00
Andrea Cavalli
2fb96eaf03 Ensure running in a blocking thread 2021-09-29 11:38:36 +02:00
Andrea Cavalli
74f2cce604 Throw error only when needed 2021-09-26 15:10:42 +02:00
Andrea Cavalli
5d7b403e55 Move reactive collector 2021-09-25 18:31:41 +02:00
Andrea Cavalli
f03f7296d4 Separate LeafCollector 2021-09-25 14:09:10 +02:00
Andrea Cavalli
5443e330bb Optimistic emission of results 2021-09-25 13:26:59 +02:00
Andrea Cavalli
8e15020f5b Improve performance of infinite queries 2021-09-25 13:07:52 +02:00
Andrea Cavalli
d06c9184bd Bugfix 2021-09-25 13:07:40 +02:00
Andrea Cavalli
d96b5a168b Execute searches on its own executor 2021-09-25 13:06:24 +02:00
Andrea Cavalli
3742eedd7f Release dropped resource 2021-09-24 04:01:28 +02:00
Andrea Cavalli
996165c3cc Do not release cached searcher 2021-09-24 02:47:52 +02:00
Andrea Cavalli
23fa46c775 Release cached searcher 2021-09-24 02:44:12 +02:00
Andrea Cavalli
20c19f662b Avoid multiple drops 2021-09-24 02:01:45 +02:00
Andrea Cavalli
be1ca997a7 Bugfix 2021-09-23 23:45:41 +02:00
Andrea Cavalli
1a73a5a33f Bugfixes 2021-09-23 22:04:01 +02:00
Andrea Cavalli
3a544d4297 Partial migration to ResourceSupport 2021-09-23 20:57:28 +02:00
Andrea Cavalli
247207febe Temporary 2021-09-23 15:48:27 +02:00
Andrea Cavalli
51ec85abe1 Improve logging 2021-09-23 15:37:13 +02:00
Andrea Cavalli
6c849fae78 Change makeInaccessible visibility 2021-09-23 15:34:56 +02:00
Andrea Cavalli
d4c977047c Allow sorted/scored counting 2021-09-23 14:50:15 +02:00
Andrea Cavalli
b77b441515 Fix all errors 2021-09-23 11:30:44 +02:00
Andrea Cavalli
552b5f78ce Fix size check assertion 2021-09-23 02:22:30 +02:00
Andrea Cavalli
29086b1939 Add some tests for low level dictionary, log tests, pass more tests 2021-09-23 02:15:58 +02:00
Andrea Cavalli
e034f3b778 Support memory segments 2021-09-22 18:33:28 +02:00
Andrea Cavalli
8b73a05177 Bugfixes and upgraded to java 17 2021-09-22 11:03:39 +02:00
Andrea Cavalli
0fee105f0b Bugfixes 2021-09-21 02:01:02 +02:00
Andrea Cavalli
bd97704a0b Better page limits 2021-09-20 18:20:59 +02:00
Andrea Cavalli
b8adbf452e Add resources flux 2021-09-20 16:22:39 +02:00
Andrea Cavalli
3c5edbc06e Finished initial refactoring 2021-09-20 12:51:27 +02:00
Andrea Cavalli
4a883ca8ea Done implementing Local and LocalMulti lucene indices 2021-09-20 11:52:21 +02:00
Andrea Cavalli
ca37d1fb68 Rewrite lucene transformers and implement MoreLikeThis sharding 2021-09-20 11:35:01 +02:00
Andrea Cavalli
5cfb5f49cd Add custom MoreLikeThis with sharding support 2021-09-20 00:22:22 +02:00
Andrea Cavalli
65db1711b5 Missing: moreLikeThis 2021-09-19 19:59:37 +02:00
Andrea Cavalli
31422847c2 Add SimpleUnsortedUnscoredLuceneMultiSearcher 2021-09-19 12:35:48 +02:00
Andrea Cavalli
d1963a1d65 Finish refactoring SimpleLuceneLocalSearcher 2021-09-19 12:01:11 +02:00
Andrea Cavalli
8bc0284f27 First safe search commit 2021-09-18 18:34:21 +02:00
Andrea Cavalli
26e9c4bbee Fix package 2021-09-17 20:37:26 +02:00
Andrea Cavalli
6782abb66d Use Net5 2021-09-17 16:56:28 +02:00
Andrea Cavalli
32300d63ac Fix ci mode 2021-09-10 13:42:02 +02:00
Andrea Cavalli
c467d5f2ba Add ci mode 2021-09-10 13:34:26 +02:00
Andrea Cavalli
7feeb4a9ce Use markers 2021-09-10 12:13:52 +02:00
Andrea Cavalli
0a378bc0f1 Fix negative value 2021-09-10 12:13:46 +02:00
Andrea Cavalli
9e09574517 Don't parallelize here 2021-09-10 01:13:39 +02:00
Andrea Cavalli
36dde3bb5a Set concurrency to 3 2021-09-10 00:05:39 +02:00
Andrea Cavalli
c47f45076a Optimization 2021-09-09 23:27:39 +02:00
Andrea Cavalli
f157e67818 Various searchers optimizations 2021-09-09 23:00:16 +02:00
Andrea Cavalli
6dda099cc1 Fix registration of global allocator 2021-09-09 16:18:47 +02:00
Andrea Cavalli
d9fd4113e0 Fix refactoring 2021-09-09 15:34:31 +02:00
Andrea Cavalli
9c167a322a Extend metric utils 2021-09-09 15:30:28 +02:00
Andrea Cavalli
2fd311f67a Support allocator metrics 2021-09-09 15:21:20 +02:00
Andrea Cavalli
48ba4d15de Bugfix 2021-09-09 12:31:49 +02:00
Andrea Cavalli
cb5a5e36f2 Fix timeout subscription 2021-09-09 11:43:37 +02:00
Andrea Cavalli
48db79359b Use boundedelastic db scheduler 2021-09-08 23:51:05 +02:00
Andrea Cavalli
5293fbe22c Close leaked data 2021-09-08 23:41:18 +02:00
Andrea Cavalli
3a79e5ba6f Undo parallelization of lucene searches 2021-09-08 23:41:05 +02:00
Andrea Cavalli
6926292904 Avoid nesting schedulers inside generators 2021-09-08 22:32:08 +02:00
Andrea Cavalli
2b21e6a864 Use blocking generator instead of Flux.create 2021-09-08 22:16:06 +02:00
Andrea Cavalli
e12e240487 Subscribe on the same thread 2021-09-08 21:35:04 +02:00
Andrea Cavalli
09c7e4f730 Performance bugfix 2021-09-08 21:34:52 +02:00