Commit Graph

422 Commits

Author SHA1 Message Date
Andrea Cavalli ca37d1fb68 Rewrite lucene transformers and implement MoreLikeThis sharding 2021-09-20 11:35:01 +02:00
Andrea Cavalli 5cfb5f49cd Add custom MoreLikeThis with sharding support 2021-09-20 00:22:22 +02:00
Andrea Cavalli 65db1711b5 Missing: moreLikeThis 2021-09-19 19:59:37 +02:00
Andrea Cavalli 31422847c2 Add SimpleUnsortedUnscoredLuceneMultiSearcher 2021-09-19 12:35:48 +02:00
Andrea Cavalli d1963a1d65 Finish refactoring SimpleLuceneLocalSearcher 2021-09-19 12:01:11 +02:00
Andrea Cavalli 8bc0284f27 First safe search commit 2021-09-18 18:34:21 +02:00
Andrea Cavalli 26e9c4bbee Fix package 2021-09-17 20:37:26 +02:00
Andrea Cavalli 6782abb66d Use Net5 2021-09-17 16:56:28 +02:00
Andrea Cavalli 7feeb4a9ce Use markers 2021-09-10 12:13:52 +02:00
Andrea Cavalli 0a378bc0f1 Fix negative value 2021-09-10 12:13:46 +02:00
Andrea Cavalli 9e09574517 Don't parallelize here 2021-09-10 01:13:39 +02:00
Andrea Cavalli 36dde3bb5a Set concurrency to 3 2021-09-10 00:05:39 +02:00
Andrea Cavalli c47f45076a Optimization 2021-09-09 23:27:39 +02:00
Andrea Cavalli f157e67818 Various searchers optimizations 2021-09-09 23:00:16 +02:00
Andrea Cavalli 6dda099cc1 Fix registration of global allocator 2021-09-09 16:18:47 +02:00
Andrea Cavalli d9fd4113e0 Fix refactoring 2021-09-09 15:34:31 +02:00
Andrea Cavalli 9c167a322a Extend metric utils 2021-09-09 15:30:28 +02:00
Andrea Cavalli 2fd311f67a Support allocator metrics 2021-09-09 15:21:20 +02:00
Andrea Cavalli 48ba4d15de Bugfix 2021-09-09 12:31:49 +02:00
Andrea Cavalli cb5a5e36f2 Fix timeout subscription 2021-09-09 11:43:37 +02:00
Andrea Cavalli 48db79359b Use boundedelastic db scheduler 2021-09-08 23:51:05 +02:00
Andrea Cavalli 5293fbe22c Close leaked data 2021-09-08 23:41:18 +02:00
Andrea Cavalli 3a79e5ba6f Undo parallelization of lucene searches 2021-09-08 23:41:05 +02:00
Andrea Cavalli 6926292904 Avoid nesting schedulers inside generators 2021-09-08 22:32:08 +02:00
Andrea Cavalli 2b21e6a864 Use blocking generator instead of Flux.create 2021-09-08 22:16:06 +02:00
Andrea Cavalli e12e240487 Subscribe on the same thread 2021-09-08 21:35:04 +02:00
Andrea Cavalli 09c7e4f730 Performance bugfix 2021-09-08 21:34:52 +02:00
Andrea Cavalli 047a471bf7 Fix last leaks 2021-09-08 00:22:39 +02:00
Andrea Cavalli 46787fa353 Fix untested testing function
A test function was not tested, causing the uneffectiveness of some tests that were passing when they should have not.
New tests to test the test function have been written, making sure to avoid false negatives.
Add editorconfig, add documentation of MetricUtils
2021-09-07 19:44:23 +02:00
Andrea Cavalli 57a0206cf1 Fully implement memory dictionary 2021-09-07 19:32:37 +02:00
Andrea Cavalli a662033228 Less verbose exception 2021-09-07 11:39:13 +02:00
Andrea Cavalli dd1ed1d18f Suppress warning 2021-09-07 11:29:28 +02:00
Andrea Cavalli d4ed99cc24 Fix compilation issue 2021-09-07 11:28:03 +02:00
Andrea Cavalli 717d13ef1f Revert commit 1882e8b300 2021-09-07 11:26:10 +02:00
Andrea Cavalli 66fa853272 Avoid memory leaks 2021-09-07 02:36:11 +02:00
Andrea Cavalli f5d3474966 Fix possible deadlock 2021-09-07 02:08:29 +02:00
Andrea Cavalli 936c07406e Implement more phasers 2021-09-06 18:52:21 +02:00
Andrea Cavalli 51b60168f7 Implement phaser 2021-09-06 18:29:10 +02:00
Andrea Cavalli d05994c8ff Rename manager 2021-09-06 18:24:36 +02:00
Andrea Cavalli 5a79d2d7cb Close the CachedIndexSearcher automatically 2021-09-06 18:23:47 +02:00
Andrea Cavalli 7230e96343 Bugfix 2021-09-06 17:42:12 +02:00
Andrea Cavalli e59705bd0b Better manage cached index searchers 2021-09-06 17:35:02 +02:00
Andrea Cavalli b7d975ccfb MultiGet window set to 16 to avoid memory problems 2021-09-06 15:54:44 +02:00
Andrea Cavalli 16bd3b263b Limit cached indexwriters 2021-09-06 15:08:07 +02:00
Andrea Cavalli d5964a7bed Pool searchers 2021-09-06 15:06:51 +02:00
Andrea Cavalli 6c73c1e86d Add memory tests
needs fix
2021-09-06 01:15:13 +02:00
Andrea Cavalli 4c348a6b2f Throw exception if running in a nonblocking scope 2021-09-05 14:23:46 +02:00
Andrea Cavalli 1882e8b300 Optimize performance by removing all superfluous subscribeOn calls 2021-09-04 16:42:47 +02:00
Andrea Cavalli 2fe8063193 Fix composite buffers 2021-09-04 02:35:12 +02:00
Andrea Cavalli f5f46edbec Use netty5 package 2021-09-04 02:19:10 +02:00
Andrea Cavalli 154086dee2 Fix compilation issues and buffer expansions 2021-09-03 16:11:14 +02:00
Andrea Cavalli 47de651c94 Bugfixes 2021-09-03 02:22:55 +02:00
Andrea Cavalli a8028024c8 Finish refactoring 2021-09-02 21:14:26 +02:00
Andrea Cavalli 3091b81d34 Fix map tests 2021-09-02 17:15:40 +02:00
Andrea Cavalli ff7823656e y 2021-09-01 00:01:56 +02:00
Andrea Cavalli 013d26387d (unfinished) Netty 5 refactoring 2021-08-31 15:50:11 +02:00
Andrea Cavalli 0faef5316e (unfinished) Netty 5 refactoring 2021-08-31 09:14:46 +02:00
Andrea Cavalli 3b55e8bd24 (unfinished) Netty 5 refactoring 2021-08-29 23:18:03 +02:00
Andrea Cavalli 9984cfff73 Bugfixes 2021-08-29 01:15:51 +02:00
Andrea Cavalli 8adc8b77b4 Bugfixes 2021-08-28 22:51:05 +02:00
Andrea Cavalli 03b5876001 Bugfixes 2021-08-28 22:42:51 +02:00
Andrea Cavalli 0c17af2ae5 Bugfixes 2021-08-27 02:49:51 +02:00
Andrea Cavalli 6fe9f9c24f Fix hanging on ScoredSimpleLuceneShardSearcher 2021-08-25 10:23:42 +02:00
Andrea Cavalli 6e31aa01f9 Ignore release errors for empty searchers 2021-08-24 14:13:43 +02:00
Andrea Cavalli 2207ae4314 Fix class names 2021-08-24 11:55:33 +02:00
Andrea Cavalli a0eb80b130 Protect against memory leaks with dropped search results 2021-08-24 11:53:19 +02:00
Andrea Cavalli a909aaaf52 LLKeyScore is now a regular record 2021-08-24 11:06:25 +02:00
Andrea Cavalli c5552d2827 Bugfixes 2021-08-22 23:50:50 +02:00
Andrea Cavalli 2a24570512 Add checked serialization exception 2021-08-22 21:23:22 +02:00
Andrea Cavalli 4e71e42d32 Code cleanup 2021-08-22 19:54:23 +02:00
Andrea Cavalli 906379923f Lazy retain generated suffixes 2021-08-22 19:52:19 +02:00
Andrea Cavalli bc759c344d Use a new approach to completely avoid memory leaks 2021-08-22 18:20:05 +02:00
Andrea Cavalli 6ed31ab2a6 Code cleanup 2021-08-16 10:36:54 +02:00
Andrea Cavalli 435e7d4886 Use a new approach to completely avoid memory leaks 2021-08-16 10:27:47 +02:00
Andrea Cavalli 9d326f5a8b Human readable representation of total hits count 2021-08-04 01:16:17 +02:00
Andrea Cavalli 9734d78839 Total hits with precision 2021-08-04 01:12:39 +02:00
Andrea Cavalli bcd99f4727 Avoid errors 2021-08-04 00:44:06 +02:00
Andrea Cavalli 328a45f40d Fix unscored-unsorted collectors 2021-07-31 23:12:13 +02:00
Andrea Cavalli 586be8751c Fix compilation issue 2021-07-31 19:18:20 +02:00
Andrea Cavalli d411675c2b Use flatMapIterable when possible 2021-07-31 18:00:53 +02:00
Andrea Cavalli 7597e54bac Remove unnecessary schedulers 2021-07-31 12:28:53 +02:00
Andrea Cavalli 4e782403f5 Fix results ordering 2021-07-30 14:01:12 +02:00
Andrea Cavalli db54773cda remove warning 2021-07-28 22:18:33 +02:00
Andrea Cavalli e738eda331 Performance optimization 2021-07-27 19:34:51 +02:00
Andrea Cavalli b816e1f9e5 Use buffertimeoutpublisher 2021-07-27 01:31:18 +02:00
Andrea Cavalli 7abcdf05f9 Optimize performance of unscored queries 2021-07-27 00:32:30 +02:00
Andrea Cavalli ea86bf7a43 Improve performance with UnscoredCollector 2021-07-26 19:21:17 +02:00
Andrea Cavalli 32b514c866 Add max open files option 2021-07-24 23:47:51 +02:00
Andrea Cavalli 1df38022fb Fix max open files with direct I/O 2021-07-23 22:43:24 +02:00
Andrea Cavalli c5d353e02a Fix discard errors 2021-07-23 22:18:08 +02:00
Andrea Cavalli abde1d1aab Fix getMulti 2021-07-23 15:20:33 +02:00
Andrea Cavalli aa1aa7a6fb Test more numbers 2021-07-18 19:37:24 +02:00
Andrea Cavalli ff9ee54857 Improve performance 2021-07-17 23:06:26 +02:00
Andrea Cavalli 43439c6f10 Optimize performance 2021-07-17 11:52:08 +02:00
Andrea Cavalli e4bbeeca3a Fix messages query not releasing 2021-07-15 15:54:34 +02:00
Andrea Cavalli 547fa68fe7 use record getters in queries 2021-07-13 23:07:39 +02:00
Andrea Cavalli 2489c50022 Make buckets unmodifiable 2021-07-13 22:58:08 +02:00
Andrea Cavalli fe5f50be10 Remove deprecated methods 2021-07-13 18:07:19 +02:00
Andrea Cavalli 61d4b00a02 Suppress warning 2021-07-11 00:39:48 +02:00
Andrea Cavalli d5c42d531f Fix lexicographic order 2021-07-11 00:37:32 +02:00
Andrea Cavalli 58a9121978 Add heap kv database implementation 2021-07-10 20:52:01 +02:00
Andrea Cavalli 5f3c8a2515 Avoid calling reduce() multiple times 2021-07-08 18:54:53 +02:00
Andrea Cavalli e2b5432b8d Fix npe 2021-07-08 17:04:53 +02:00
Andrea Cavalli 211a0b36cd Search unscored sorted queries using the non-scored shard searcher 2021-07-08 17:01:56 +02:00
Andrea Cavalli c85dcfb54e Clean rocksdb code 2021-07-06 22:27:03 +02:00
Andrea Cavalli 2a47e6f3fd Add lucene options 2021-07-06 14:33:47 +02:00
Andrea Cavalli e7e25e646d Fix scheduling 2021-07-06 02:23:06 +02:00
Andrea Cavalli 7cebcd7e92 Sort shards during merge 2021-07-06 01:52:12 +02:00
Andrea Cavalli 6c97b0f068 Bugfixes 2021-07-06 01:30:37 +02:00
Andrea Cavalli b5ccf315be Implement adaptive multi searcher for shard indexes 2021-07-06 00:30:14 +02:00
Andrea Cavalli 545c7111de Bugfix 2021-07-05 18:55:11 +02:00
Andrea Cavalli 00ff36836e Improve reactive searchers performance 2021-07-05 15:43:19 +02:00
Andrea Cavalli a5d4584a11 Adaptive reactive lucene search engine, lazy results 2021-07-05 12:05:45 +02:00
Andrea Cavalli 7929f0dc8c Fully reactive lucene queries 2021-07-04 01:34:17 +02:00
Andrea Cavalli 8a1e4028f7 Advanced lucene options 2021-07-01 21:19:52 +02:00
Andrea Cavalli b80feabb97 Use NIOFSDirectory 2021-06-30 23:43:46 +02:00
Andrea Cavalli b567f9f320 Copy map before writing to avoid errors 2021-06-30 10:58:49 +02:00
Andrea Cavalli 3758d06853 Configurable direct buffers 2021-06-29 23:31:02 +02:00
Andrea Cavalli 2df2c00e36 Fix unmodifiable read options 2021-06-27 16:52:45 +02:00
Andrea Cavalli cb8dbf2fa2 Fix direct I/O parameters 2021-06-27 16:47:59 +02:00
Andrea Cavalli c9e9cb8a01 Bugfix 2021-06-27 16:33:23 +02:00
Andrea Cavalli e3536cfd01 Remove deprecation 2021-06-27 16:19:56 +02:00
Andrea Cavalli c6d5beb33b Database options record 2021-06-27 15:40:56 +02:00
Andrea Cavalli 507101e453 Add verifyChecksum method 2021-06-27 15:06:48 +02:00
Andrea Cavalli 1d281d0305 Prevent reading corrupted database parts, reimplement badblocks 2021-06-27 11:58:12 +02:00
Andrea Cavalli 63cdb43644 Fix badblocks memory leak 2021-06-26 23:57:24 +02:00
Andrea Cavalli bd8755c180 Implement "badblocks" method 2021-06-26 02:35:33 +02:00
Andrea Cavalli ee05614115 Use Direct I/O when available 2021-06-25 23:47:53 +02:00
Andrea Cavalli bd62b83d57 Bugfix 2021-06-25 20:25:57 +02:00
Andrea Cavalli 4640d0dfad Better implementation of snapshots in lucene shards 2021-06-25 20:07:19 +02:00
Andrea Cavalli 47f6081cde better parameters for performance 2021-06-25 20:06:58 +02:00
Andrea Cavalli 03d3afa317 Reimplement query serializer 2021-06-20 01:09:06 +02:00
Andrea Cavalli 924dbf6bf6 Add immutable read/write options 2021-06-19 21:55:20 +02:00
Andrea Cavalli 1bd1cb87de Fix compatibility 2021-06-19 16:26:54 +02:00
Andrea Cavalli 92bb13336d Bugfix 2021-06-19 13:27:58 +02:00
Andrea Cavalli b1d48d36ff Bugfixes 2021-06-19 12:14:14 +02:00
Andrea Cavalli 6039241d2b Update common utils 2021-06-09 02:56:53 +02:00
Andrea Cavalli dcd7add36f Add lucene delta import 2021-06-07 16:21:12 +02:00
Andrea Cavalli adf3bef488 Fix bug with map access 2021-06-06 02:23:51 +02:00
Andrea Cavalli 187274071f Bugfix 2021-06-02 13:21:07 +02:00
Andrea Cavalli f95a46b634 Add delta isModified 2021-06-01 17:08:37 +02:00
Andrea Cavalli ccf07f4216 Change readahead size 2021-05-29 01:45:03 +02:00
Andrea Cavalli 6eb531e4f1 Add ICU 2021-05-28 16:04:59 +02:00
Andrea Cavalli 4e76073259 Update data-generator 2021-05-25 11:17:24 +02:00
Andrea Cavalli a293b65b08 Update to lucene 9 2021-05-25 01:12:41 +02:00
Andrea Cavalli 6ccff7646b Update reactor 2021-05-22 14:46:54 +02:00
Andrea Cavalli 2d24436b93 Update to java 16 2021-05-21 00:19:40 +02:00
Andrea Cavalli 211d2fc99e Bugfixes 2021-05-19 22:51:55 +02:00
Andrea Cavalli c9f64195e7 Performance tuning 2021-05-18 01:10:30 +02:00
Andrea Cavalli f8377c3e63 Bugfix 2021-05-17 04:10:41 +02:00