Andrea Cavalli
d2abe044cc
Rethink rocksdb settings
2022-01-12 16:18:31 +01:00
Andrea Cavalli
ce506f22dd
Bugfix
2022-01-11 22:28:42 +01:00
Andrea Cavalli
4e5e4423ff
Rewrite the custom analyzers
2022-01-11 22:23:07 +01:00
Andrea Cavalli
aad377acb0
Change default values
2022-01-10 22:59:36 +01:00
Andrea Cavalli
84dea46a35
Update logging
2022-01-04 12:55:33 +01:00
Andrea Cavalli
ce97f33873
Fix infostream
2021-12-31 01:00:32 +01:00
Andrea Cavalli
57bcc92512
Fix statistics
2021-12-31 00:58:47 +01:00
Andrea Cavalli
d6244ab18d
Rewind and clear the buffers
2021-12-30 22:29:17 +01:00
Andrea Cavalli
64f21409f1
Performance optimization
2021-12-30 22:29:06 +01:00
Andrea Cavalli
b493decd64
Fix metric names
2021-12-30 18:20:56 +01:00
Andrea Cavalli
68d8b5240c
Implement more micrometer metrics
2021-12-30 17:28:06 +01:00
Andrea Cavalli
5769bc7076
Print errored key
2021-12-29 00:31:35 +01:00
Andrea Cavalli
24493eb4ff
Change default timeout
2021-12-27 23:07:11 +01:00
Andrea Cavalli
7993a6210b
Fix pessimistic db
2021-12-27 18:44:54 +01:00
Andrea Cavalli
dffb8eb3eb
Set default compression types
2021-12-27 17:49:22 +01:00
Andrea Cavalli
9309692062
Configurable optimistic transactions
2021-12-27 17:45:52 +01:00
Andrea Cavalli
aa04a64c34
Fix logging, configurable compression
2021-12-27 17:34:44 +01:00
Andrea Cavalli
582813b6c7
Customizable volumes
2021-12-27 16:33:31 +01:00
Andrea Cavalli
3ae3bd7944
Fix bound
2021-12-26 20:24:02 +01:00
Andrea Cavalli
2b09f5a738
Update retry algorithm
2021-12-26 20:20:27 +01:00
Andrea Cavalli
83bc3ebb04
Update retry algorithm
2021-12-26 12:47:00 +01:00
Andrea Cavalli
03bbf3baa9
Bugfix
2021-12-23 02:20:40 +01:00
Andrea Cavalli
a1eec93c64
Rewrite unsorted lucene queries
2021-12-23 02:13:51 +01:00
Andrea Cavalli
a5666dd5b4
Increase default timeout
2021-12-20 16:38:58 +01:00
Andrea Cavalli
eaef75a304
Close LMDB databases after each full search
2021-12-18 21:01:14 +01:00
Andrea Cavalli
480ab77db8
Use sorted maps
2021-12-18 18:16:56 +01:00
Andrea Cavalli
1de5e52ffd
Use linkedhashmap
2021-12-18 15:15:19 +01:00
Andrea Cavalli
9952eaffc0
Using search instead of count is too heavy
2021-12-18 00:18:24 +01:00
Andrea Cavalli
0f62362d21
Disable reader pooling
2021-12-17 23:51:10 +01:00
Andrea Cavalli
638595f518
Pool LMDB databases
2021-12-17 23:12:35 +01:00
Andrea Cavalli
0e9c8c089e
Bugfixes
2021-12-17 16:24:18 +01:00
Andrea Cavalli
6d92ba8a68
Bugfixes
2021-12-17 03:04:01 +01:00
Andrea Cavalli
1a35930909
use a reentrantlock to avoid multiple merges at the same time
2021-12-17 02:19:51 +01:00
Andrea Cavalli
1dffb55572
Use log4j for logging, rewrite some local dictionary parts
2021-12-17 01:48:49 +01:00
Andrea Cavalli
7a712722d7
Code cleanup
2021-12-16 16:34:20 +01:00
Andrea Cavalli
6e312fe102
Use uninterruptible scheduler for lucene searches
2021-12-16 16:14:44 +01:00
Andrea Cavalli
2e1678373c
Bugfix
2021-12-16 03:00:51 +01:00
Andrea Cavalli
01099cc4d1
Fix unscored searcher
2021-12-16 02:38:56 +01:00
Andrea Cavalli
b7ca57a215
Schedule commits and merges
2021-12-15 16:47:59 +01:00
Andrea Cavalli
8ad622db0a
Reduce the number of threads
2021-12-15 16:04:33 +01:00
Andrea Cavalli
d58d696ca4
Bugfixes
2021-12-13 01:57:37 +01:00
Andrea Cavalli
907561d93c
Add requests timeout
2021-12-12 23:40:30 +01:00
Andrea Cavalli
5157656a2c
Don't use official searcher for production queries
2021-12-12 18:52:50 +01:00
Andrea Cavalli
3cdafd748e
Optionally disable LMDB
2021-12-12 16:43:22 +01:00
Andrea Cavalli
297c249243
Code cleanup
2021-12-12 16:19:50 +01:00
Andrea Cavalli
1a64d98697
Bugfixes
2021-12-12 02:40:26 +01:00
Andrea Cavalli
2a5e90d667
Update rocksdb to 6.26.1, Update netty
2021-12-12 02:17:36 +01:00
Andrea Cavalli
18b242d746
Update log4j and lucene
2021-12-11 13:20:28 +01:00
Andrea Cavalli
6644e040dd
Fix unsorted unscored streaming multi searcher
2021-12-08 11:58:06 +01:00
Andrea Cavalli
b5aa8b4baa
Use a LongSemaphore to avoid using a buffer
2021-11-29 23:27:55 +01:00
Andrea Cavalli
cc368aecc8
Fix deadlock
2021-11-29 14:15:31 +01:00
Andrea Cavalli
d8de969bee
Fix deadlock
2021-11-29 14:01:57 +01:00
Andrea Cavalli
3d9247c969
Optimize unscored streaming searcher
2021-11-24 16:39:22 +01:00
Andrea Cavalli
044d189600
Optimize single-shard indices
2021-11-21 12:31:23 +01:00
Andrea Cavalli
7047b512fc
Add numeric field
2021-11-20 16:09:00 +01:00
Andrea Cavalli
798b8a5288
Update policies
2021-11-20 01:30:06 +01:00
Andrea Cavalli
06d98040b1
Allow to use absolute values
2021-11-20 01:12:17 +01:00
Andrea Cavalli
3810c49fa1
Allow random sampling
2021-11-19 22:15:31 +01:00
Andrea Cavalli
29d9aad8bf
Update buckets
2021-11-19 19:03:31 +01:00
Andrea Cavalli
ed00d474d6
Implement numeric buckets collector
2021-11-18 17:13:53 +01:00
Andrea Cavalli
e014266b8d
Rename unclear parameters
2021-11-16 23:54:23 +01:00
Andrea Cavalli
c1c57388e5
Add standard lucene queries
2021-11-16 23:19:23 +01:00
Andrea Cavalli
3d7e80b4ec
Update comparators
2021-11-16 23:19:13 +01:00
Andrea Cavalli
891255e18e
Add more methods
2021-11-14 22:21:32 +01:00
Andrea Cavalli
f30ad372ed
Bugfix
2021-11-12 02:52:42 +01:00
Andrea Cavalli
23d5f700fb
Add update method to single values
2021-11-12 02:05:44 +01:00
Andrea Cavalli
42c4b6e651
Add a discard hook
2021-11-09 15:57:16 +01:00
Andrea Cavalli
d4dae4667d
Remove stream executor
2021-11-09 02:14:21 +01:00
Andrea Cavalli
7fb8183c63
Update
2021-11-09 01:54:46 +01:00
Andrea Cavalli
adbbbaa646
Optimizations
2021-11-09 01:13:47 +01:00
Andrea Cavalli
50b3c897ff
Add scoring shard multi manager
2021-11-09 00:54:09 +01:00
Andrea Cavalli
47aac33b22
Add count multi searcher
2021-11-09 00:05:26 +01:00
Andrea Cavalli
1ba4a1866f
Reactive collector multi manager
2021-11-08 23:53:53 +01:00
Andrea Cavalli
7c0a4b2af9
Bugfix
2021-11-08 23:46:40 +01:00
Andrea Cavalli
a48bea6c63
Fix wait time
2021-11-08 23:46:02 +01:00
Andrea Cavalli
ef15bf7e8c
Bugfixes
2021-11-08 18:52:52 +01:00
Andrea Cavalli
81b7df8702
Remove sends to improve performance
2021-11-08 16:33:41 +01:00
Andrea Cavalli
232e46bcea
Add more discard handlers
2021-11-08 12:06:32 +01:00
Andrea Cavalli
59aa1ef5c6
Update database
2021-11-08 11:17:52 +01:00
Andrea Cavalli
63cd178988
Use merge operator when possible
2021-11-08 10:49:59 +01:00
Andrea Cavalli
b48ab0b095
Add soft update document
2021-11-07 18:34:34 +01:00
Andrea Cavalli
52be944b5f
Bugfixes
2021-11-07 18:00:11 +01:00
Andrea Cavalli
b34b620082
Allow lucene partial document updates
2021-11-07 17:46:40 +01:00
Andrea Cavalli
d8801b3471
Update loggers
2021-11-07 14:46:03 +01:00
Andrea Cavalli
eba6f92d7f
Remove warning
2021-11-05 16:54:58 +01:00
Andrea Cavalli
c46320cdf7
Adjust database parameters
2021-11-02 13:02:49 +01:00
Andrea Cavalli
e015c9afff
Add todos
2021-10-30 17:43:59 +02:00
Andrea Cavalli
c506a7e71b
Use netty direct memory if possible
2021-10-30 12:39:56 +02:00
Andrea Cavalli
98dcb39ce1
Reformat iterator
2021-10-30 12:21:28 +02:00
Andrea Cavalli
1ea004630e
Use Micrometer registry, better optimistic commit failure logging
2021-10-30 11:13:46 +02:00
Andrea Cavalli
f3c63842c8
Don't close unreleasable buffers
2021-10-29 00:11:25 +02:00
Andrea Cavalli
37de47fd6c
Refactor results
2021-10-28 23:57:23 +02:00
Andrea Cavalli
d861e26a57
Rename sort
2021-10-28 17:18:23 +02:00
Andrea Cavalli
417c174761
Implement ResourceSupport
2021-10-28 11:44:20 +02:00
Andrea Cavalli
e24dc4bece
Update dependencies
2021-10-26 23:59:55 +02:00
Andrea Cavalli
da2b302d1b
Replace sinks with references when needed
2021-10-26 00:02:08 +02:00
Andrea Cavalli
1899ef1723
Optimize array
2021-10-25 01:54:37 +02:00
Andrea Cavalli
f9b656986c
Clean queues
2021-10-25 01:44:12 +02:00
Andrea Cavalli
7a01b6a2a4
Group writes together to speedup writes
2021-10-25 01:41:02 +02:00
Andrea Cavalli
e48f8b25eb
Remove unused constants
2021-10-25 00:44:30 +02:00
Andrea Cavalli
db9b1bb7fb
Rename constants
2021-10-25 00:42:48 +02:00
Andrea Cavalli
90aa7a2522
Use in-memory database if the limit is lower than 8192
2021-10-22 01:26:44 +02:00
Andrea Cavalli
a77af5253d
Fix performance issue
2021-10-22 00:51:54 +02:00
Andrea Cavalli
213e446ff4
Separate lucene from the rest of the app to avoid thread interruptions
2021-10-21 11:55:59 +02:00
Andrea Cavalli
fb43fa78af
Discard transaction data
2021-10-21 10:01:29 +02:00
Andrea Cavalli
1f04241ac6
Use optimistic db
2021-10-21 10:00:39 +02:00
Andrea Cavalli
1625a5c44b
RocksDB database abstraction
2021-10-20 01:51:34 +02:00
Andrea Cavalli
80d0ced888
Optimistic transactions, inline codecs
2021-10-19 00:22:05 +02:00
Andrea Cavalli
aad5f8c96c
Optimistic updates
2021-10-17 19:52:43 +02:00
Andrea Cavalli
df2b757fdd
Remove unused class
2021-10-17 17:15:57 +02:00
Andrea Cavalli
3a591c38f2
Use new netty resource leak detector
2021-10-16 16:56:10 +02:00
Andrea Cavalli
e66efba834
Publish lucene 9 snapshots
2021-10-16 14:59:38 +02:00
Andrea Cavalli
f77784fc50
Improve temporary LMDB database lifecycle
2021-10-16 14:35:04 +02:00
Andrea Cavalli
a441c2fb85
Move comparator
2021-10-16 01:51:07 +02:00
Andrea Cavalli
6f8f70c23b
Bugfix in sizeRange
2021-10-16 01:49:41 +02:00
Andrea Cavalli
fd565c80b2
Fix assertions
2021-10-16 01:35:09 +02:00
Andrea Cavalli
f6aab7dafb
More tests
2021-10-16 00:50:06 +02:00
Andrea Cavalli
f91617c7ec
Update tests
2021-10-16 00:32:04 +02:00
Andrea Cavalli
e6c0f14fda
Implement some sort codecs
2021-10-15 22:03:53 +02:00
Andrea Cavalli
17a9b49755
Partial sorted implementation
2021-10-15 00:03:41 +02:00
Andrea Cavalli
36df18796b
Partial sorted implementation
2021-10-14 23:04:34 +02:00
Andrea Cavalli
e1d1e1fb05
Fix LMDB PriorityQueue
2021-10-14 15:55:58 +02:00
Andrea Cavalli
d6b6a211a8
Replace scoreMode with boolean "complete"
2021-10-14 00:49:21 +02:00
Andrea Cavalli
5817d8a93f
Implement more tests
2021-10-13 14:26:54 +02:00
Andrea Cavalli
c8536e45da
Cleanup code
2021-10-13 13:16:56 +02:00
Andrea Cavalli
584115c5cc
Rename classes and improve readonly tests performance
2021-10-13 12:25:32 +02:00
Andrea Cavalli
09f60a3a99
Implement memory-mapped sorted searcher for streaming huge query results
2021-10-13 00:23:56 +02:00
Andrea Cavalli
83e98ebce8
Fast implementation of query transformers
2021-10-08 11:50:28 +02:00
Andrea Cavalli
30a14a4aae
Add todos
2021-10-08 02:14:34 +02:00
Andrea Cavalli
4d5f8b5b37
Implement MoreLikeThis transformer
2021-10-08 02:13:33 +02:00
Andrea Cavalli
81d8abe72c
Add ensureBlocking in blocking method
2021-10-07 00:54:20 +02:00
Andrea Cavalli
013e4c5dd8
Remove publishOn in lucene utils
2021-10-07 00:54:11 +02:00
Andrea Cavalli
33e4cb2f14
Add ensureOpen method to avoid writing to the index when closing
2021-10-07 00:53:38 +02:00
Andrea Cavalli
818bd20ed9
Update netty
2021-10-01 19:17:33 +02:00
Andrea Cavalli
9646ddcb28
Use method references
2021-09-30 18:25:36 +02:00
Andrea Cavalli
5d2ba797eb
Cleanup code
2021-09-30 00:25:09 +02:00
Andrea Cavalli
9a6f195b63
Cleanup utils
2021-09-30 00:23:41 +02:00
Andrea Cavalli
dc14192dca
Reimplement MultiMoreLikeThis using BigCompositeReader
2021-09-29 13:18:06 +02:00
Andrea Cavalli
6b8c1025d1
Fix ClassCastException
2021-09-29 11:47:17 +02:00
Andrea Cavalli
74d20204ab
Avoid using BufferTimeoutPublisher (it kills lucene Async IO)
2021-09-29 11:38:53 +02:00
Andrea Cavalli
2fb96eaf03
Ensure running in a blocking thread
2021-09-29 11:38:36 +02:00
Andrea Cavalli
74f2cce604
Throw error only when needed
2021-09-26 15:10:42 +02:00
Andrea Cavalli
5d7b403e55
Move reactive collector
2021-09-25 18:31:41 +02:00
Andrea Cavalli
f03f7296d4
Separate LeafCollector
2021-09-25 14:09:10 +02:00
Andrea Cavalli
5443e330bb
Optimistic emission of results
2021-09-25 13:26:59 +02:00
Andrea Cavalli
8e15020f5b
Improve performance of infinite queries
2021-09-25 13:07:52 +02:00
Andrea Cavalli
d06c9184bd
Bugfix
2021-09-25 13:07:40 +02:00
Andrea Cavalli
d96b5a168b
Execute searches on its own executor
2021-09-25 13:06:24 +02:00
Andrea Cavalli
3742eedd7f
Release dropped resource
2021-09-24 04:01:28 +02:00
Andrea Cavalli
996165c3cc
Do not release cached searcher
2021-09-24 02:47:52 +02:00
Andrea Cavalli
23fa46c775
Release cached searcher
2021-09-24 02:44:12 +02:00
Andrea Cavalli
20c19f662b
Avoid multiple drops
2021-09-24 02:01:45 +02:00
Andrea Cavalli
be1ca997a7
Bugfix
2021-09-23 23:45:41 +02:00
Andrea Cavalli
1a73a5a33f
Bugfixes
2021-09-23 22:04:01 +02:00
Andrea Cavalli
3a544d4297
Partial migration to ResourceSupport
2021-09-23 20:57:28 +02:00
Andrea Cavalli
247207febe
Temporary
2021-09-23 15:48:27 +02:00
Andrea Cavalli
51ec85abe1
Improve logging
2021-09-23 15:37:13 +02:00
Andrea Cavalli
6c849fae78
Change makeInaccessible visibility
2021-09-23 15:34:56 +02:00
Andrea Cavalli
d4c977047c
Allow sorted/scored counting
2021-09-23 14:50:15 +02:00
Andrea Cavalli
b77b441515
Fix all errors
2021-09-23 11:30:44 +02:00
Andrea Cavalli
552b5f78ce
Fix size check assertion
2021-09-23 02:22:30 +02:00
Andrea Cavalli
29086b1939
Add some tests for low level dictionary, log tests, pass more tests
2021-09-23 02:15:58 +02:00
Andrea Cavalli
e034f3b778
Support memory segments
2021-09-22 18:33:28 +02:00
Andrea Cavalli
8b73a05177
Bugfixes and upgraded to java 17
2021-09-22 11:03:39 +02:00
Andrea Cavalli
0fee105f0b
Bugfixes
2021-09-21 02:01:02 +02:00
Andrea Cavalli
bd97704a0b
Better page limits
2021-09-20 18:20:59 +02:00
Andrea Cavalli
b8adbf452e
Add resources flux
2021-09-20 16:22:39 +02:00
Andrea Cavalli
3c5edbc06e
Finished initial refactoring
2021-09-20 12:51:27 +02:00
Andrea Cavalli
4a883ca8ea
Done implementing Local and LocalMulti lucene indices
2021-09-20 11:52:21 +02:00
Andrea Cavalli
ca37d1fb68
Rewrite lucene transformers and implement MoreLikeThis sharding
2021-09-20 11:35:01 +02:00
Andrea Cavalli
5cfb5f49cd
Add custom MoreLikeThis with sharding support
2021-09-20 00:22:22 +02:00
Andrea Cavalli
65db1711b5
Missing: moreLikeThis
2021-09-19 19:59:37 +02:00
Andrea Cavalli
31422847c2
Add SimpleUnsortedUnscoredLuceneMultiSearcher
2021-09-19 12:35:48 +02:00
Andrea Cavalli
d1963a1d65
Finish refactoring SimpleLuceneLocalSearcher
2021-09-19 12:01:11 +02:00
Andrea Cavalli
8bc0284f27
First safe search commit
2021-09-18 18:34:21 +02:00
Andrea Cavalli
26e9c4bbee
Fix package
2021-09-17 20:37:26 +02:00
Andrea Cavalli
6782abb66d
Use Net5
2021-09-17 16:56:28 +02:00
Andrea Cavalli
32300d63ac
Fix ci mode
2021-09-10 13:42:02 +02:00
Andrea Cavalli
c467d5f2ba
Add ci mode
2021-09-10 13:34:26 +02:00
Andrea Cavalli
7feeb4a9ce
Use markers
2021-09-10 12:13:52 +02:00
Andrea Cavalli
0a378bc0f1
Fix negative value
2021-09-10 12:13:46 +02:00
Andrea Cavalli
9e09574517
Don't parallelize here
2021-09-10 01:13:39 +02:00
Andrea Cavalli
36dde3bb5a
Set concurrency to 3
2021-09-10 00:05:39 +02:00
Andrea Cavalli
c47f45076a
Optimization
2021-09-09 23:27:39 +02:00
Andrea Cavalli
f157e67818
Various searchers optimizations
2021-09-09 23:00:16 +02:00
Andrea Cavalli
6dda099cc1
Fix registration of global allocator
2021-09-09 16:18:47 +02:00
Andrea Cavalli
d9fd4113e0
Fix refactoring
2021-09-09 15:34:31 +02:00
Andrea Cavalli
9c167a322a
Extend metric utils
2021-09-09 15:30:28 +02:00
Andrea Cavalli
2fd311f67a
Support allocator metrics
2021-09-09 15:21:20 +02:00
Andrea Cavalli
48ba4d15de
Bugfix
2021-09-09 12:31:49 +02:00
Andrea Cavalli
cb5a5e36f2
Fix timeout subscription
2021-09-09 11:43:37 +02:00
Andrea Cavalli
48db79359b
Use boundedelastic db scheduler
2021-09-08 23:51:05 +02:00
Andrea Cavalli
5293fbe22c
Close leaked data
2021-09-08 23:41:18 +02:00
Andrea Cavalli
3a79e5ba6f
Undo parallelization of lucene searches
2021-09-08 23:41:05 +02:00
Andrea Cavalli
6926292904
Avoid nesting schedulers inside generators
2021-09-08 22:32:08 +02:00
Andrea Cavalli
2b21e6a864
Use blocking generator instead of Flux.create
2021-09-08 22:16:06 +02:00
Andrea Cavalli
e12e240487
Subscribe on the same thread
2021-09-08 21:35:04 +02:00
Andrea Cavalli
09c7e4f730
Performance bugfix
2021-09-08 21:34:52 +02:00
Andrea Cavalli
047a471bf7
Fix last leaks
2021-09-08 00:22:39 +02:00
Andrea Cavalli
46787fa353
Fix untested testing function
...
A test function was not tested, causing the uneffectiveness of some tests that were passing when they should have not.
New tests to test the test function have been written, making sure to avoid false negatives.
Add editorconfig, add documentation of MetricUtils
2021-09-07 19:44:23 +02:00
