Commit Graph

10538 Commits

Author SHA1 Message Date
Andrea Cavalli
c0ddce316b Rename package 2021-09-17 16:28:14 +02:00
Chris Vest
0973673fce
Make CompositeBuffer an interface (#11687)
Motivation:
Having composite buffer as a class has caused problems in the past.
Making it an interface makes it possible to stub or decorate the composite buffer implementation.

Modification:
Make CompositeBuffer an interface, and move the implementation to DefaultCompositeBuffer.
Then hide the implementation and permit construction only through static factory methods.

Result:
CompositeBuffer is now an interface.
2021-09-16 13:25:45 +02:00
Norman Maurer
8339a00fd6 Ensure bom don't define extra dependencies (#11682)
Motivation:

The bom shouldnt depend on the parent as it may define extra dependencies that would be "pulled" in.

See https://github.com/netty/netty/pull/11672#discussion_r707619462

Modifications:

- Revert commit d6383bf2478267eb62b1f71807c2a3c34794d1b0.
- Add tcnative version and add comments to ensure we keep the version in-sync

Result:

Correct bom for netty
2021-09-16 09:31:39 +02:00
Chris Vest
000f2a0934
Add CompositeBuffer.decomposeBuffer method (#11683)
Motivation:
It may in some cases be useful to unwrap a composite buffer and work on the array of buffers directly.
The decomposeBuffer method makes this possible safely, by killing the composite buffer in the process.

Modification:
Add a CompositeBuffer.decomposeBuffer method, which returns the array of the constituent component buffers of the composite buffer, and at the same time closes the composite buffer without closing its components.
The caller effectively takes ownership of the component buffers away from the composite buffer.

Result:
This API makes buffer composition fully reversible.
2021-09-15 16:38:43 +02:00
Chris Vest
cf1ab852d1 Add pinnedHeap/DirectMemory methods to ByteBufAllocatorMetric (#11667)
Motivation:
The "used memory" is the amount of memory that a pooled allocator has currently allocated and committed for itself.
Ths is useful for managing resource usage of the pool versus the available system resources.
However, it is not useful for managing resources of the currently circulating buffer instances versus the pool.
The pinned memory is the memory currently in use by buffers in circulation, plus memory held in the thread-local caches.

Modification:
Add pinned memory accounting to PoolChunk.
We cannot just use the existing freeBytes because that field is only updated when pool subpages are retired, and a chunk will never retire its last subpage instance.
The accounting statistics are available on the PooledByteBufAllocator only, since the metrics interfaces cannot be changed due to backwards compatibility.

Result:
It is now possible to get a fairly accurate (with slight over-counting due to the thread-local caches) picture of how much memory is held up in buffer instances at any given moment.

Fixes #11637
2021-09-15 16:32:52 +02:00
Nitesh Kant
29cae0445a
Introduce ByteToMessageDecoderForBuffer (#11654)
__Motivation__

In order to migrate all codec incrementally to use `Buffer`, we need a version of `ByteToMessageDecoder` that uses `Buffer`.

__Modification__

- Added the new version of `ByteToMessageDecoder` with a new name so that both old and new version can co-exist and we can incrementally migrate all codecs
- Migrated `FixedLengthFrameDecoder` as it was simple and used in tests.

__Result__

We have the basic building block to start migrating all codecs to the new `Buffer` API.
2021-09-14 17:06:49 -07:00
Norman Maurer
a76842dcd5 Use cpu_relax() implementation for aarch64 (#11677)
Motivation:

We only implement cpu_relax() for x86_64 atm, we should also do for aarch64

Modifications:

Add implementation

Result:

Possible better performance on aarch64 when busy spinning with epoll is used
2021-09-14 08:33:19 +02:00
Norman Maurer
1eb128e2a9 Upgrade to netty-tcnative 2.0.43.Final (#11679)
Motivation:

netty-tcnative 2.0.42.Final did not include all native libs due a problem during the release. This was fixed with the new release.

Modifications:

Upgrade to 2.0.43.Final

Result:

Depend on a netty-tcnative version that includes all native libs.
2021-09-14 08:32:37 +02:00
Nitesh Kant
43f3956030
CompositeBuffer#split() should correctly set offsets (#11671)
__Motivation__

While computing offsets from within `CompositeBuffer#split()`, we do not consider a case when the constituents `buffer` array is empty but existing read/write offsets are non-zero. This is possible when the buffer is full.

__Modification__

Correctly set offsets even for the aforementioned case.

__Result__

Read/write offsets are correctly set while splitting composite buffer.
2021-09-10 12:33:16 -07:00
Scott Mitchell
efd576e43e
Fix bom indentation, 4 spaces -> 2 (#11676)
Motivation:
Cherry-pick of d6383bf2478267eb62b1f71807c2a3c34794d1b0 added extra
indentation, which fails checkstyle.
2021-09-10 11:33:34 -07:00
Scott Mitchell
62a23f9e29 netty-bom to provide resolved tcnative version (#11672)
Motivation:
Netty's bom includes netty-tcnative dependencies with a variable for the
version [1]. However that variable isn't defined/resolved and therefore
leads to undefined dependencies.

[1] https://search.maven.org/artifact/io.netty/netty-bom/4.1.68.Final/pom

Modifications:
- Netty's bom file should inherit from netty-parent pom so variables can
  be resolved at inclusion time.

Result:
netty-bom allows for netty-tcnative version to be resolved in 3rd party
projects.
2021-09-10 10:01:23 -07:00
Norman Maurer
d523e68d83 Fix netty-tcnative* entries in bom
Motivation:

We did use the incorrect classifier and also missed to add the osx-aarch_64 version of netty-tcnative-boringssl-static

Modifications:

Fix entries

Result:

Correct and complete bom
2021-09-09 16:13:29 +02:00
Norman Maurer
537a0d4d81 Merge pull request from GHSA-9vjp-v76f-g363
Motivation:

e Snappy frame decoder function doesn't restrict the size of the compressed data (and the uncompressed data) which may lead to excessive memory usage. Beside this it also may buffer reserved skippable chunks until the whole chunk was received which may lead to excessive memory usage as well.

Modifications:

- Add various validations for the max allowed size of a chunk
- Skip bytes on the fly when an skippable chunk is handled

Result:

No more risk of OOME. Thanks to Ori Hollander of JFrog Security for reporting the issue.
2021-09-09 16:08:33 +02:00
Norman Maurer
f2cc94c7d4 Merge pull request from GHSA-grg4-wf29-r9vv
Motivation:

We should do the Bzip2 decoding in a streaming fashion and so ensure we propagate the buffer as soon as possible through the pipeline. This allows the users to release these buffers as fast as possible.

Modification:

- Change the Bzip2Decoder to do the decompression of data in a streaming fashion.
- Add some safety check to ensure the block length never execeeds the maximum (as defined in the spec)

Result:

No more risk of an OOME by decompress some large data via bzip2.

Thanks to Ori Hollander of JFrog Security for reporting the issue.

(we got acquired during the process and now Vdoo is part of JFrog company)
2021-09-09 16:08:21 +02:00
Norman Maurer
ebd3f8b4fb Respect jdk.tls.namedGroups when using native SSL implementation (#11660)
Motivation:

When using the JDK implementation for SSL its possible to adjust the used named groups. We should allow to do this as well and also select some default groups that will reduce the number of roundtrips.

Modifications:

- Upgrade netty-tcnative so we can set the curves
- Respect jdk.tls.namedGroups
- Use default groups of "P-256", "P-384", "X25519" so its compatible with what the JDK versions < 13 support as well.

Result:

Be able to set the used groups

Co-authored-by: Nitesh Kant <nitesh_kant@apple.com>
2021-09-09 14:36:29 +02:00
Norman Maurer
dcf1e12556 Add support for mac m1 (#11666)
Motivation:

As more and more people switch to a mac m1 we should support it

Modifications:

- Add profiles for cross-compile for mac m1
- Adjust script to finish release

Result:

Mac m1 is supported
2021-09-09 08:40:35 +02:00
Chris Vest
1eb9a9764e
Buffer should not expose nativeAddress() directly (#11665)
Motivation:
Accessing the native address, if a buffer has any, violates the no-aliasing rule for Buffers.
Also, it inherently assumes that a buffer has a single, native memory allocation.
The native address, or addresses, are already available via the Readable- and WritableComponents.
Accessing these via forEachReadable and forEachWritable will also ensure that composite buffers will be handled correctly.

Modification:
Remove the nativeAddress() method from the Buffer API.
Update the ByteBufAdaptor and a few tests to cope with this change.

Result:
Less error prone code, and make unsafe APIs a bit more hidden.
2021-09-08 20:02:59 +02:00
Ran
3152ec76db Throw exceptions when rule violating UDS paths been passed in. (#11663)
Motivation:

Currently, Netty is silently truncating all over the limit UDS paths and ignoring the `sun_path`'s null-termination role, which hurts compatibility with other UDS clients and servers.

Modifications:

Adding a validation in the JNI code, if the UDS path is not satisfying the system limit or Linux spec throw a NativeIoException.

Result:

All UDS paths Netty can successfully bind are connectable by other programs.
2021-09-08 09:22:37 +02:00
Chris Vest
59275fba52
Netty Future no longer extends JDK Future (#11647)
Motivation:
It is important to avoid blocking method calls in an event loop thread, since that can stall the system.
Netty's Future interface was extending the JDK Future interface, which included a number of blocking methods of questionable use in Netty.
We wish to reduce the number of blocking methods on the Future API in order to discourage their use a little.
Further more, the Netty Future specification of the behaviour of the cancel() and isDone() methods are inconsistent with those of the JDK Future.
If Netty's Future stop extending the JDK Future interface, it will also no longer be bound by its specification.

Modification:
Make Netty's Future no longer extend the JDK Future interface.
Change the EvenExecutorGroup interface to no longer extend ScheduledExecutorService.
The EventExecutorGroup still extends Executor, because Executor does not dictate any return type of the `execute()` method — this is also useful in the DefaultFutureCompletionStage implementation.
The Netty ScheduledFuture interface has been removed since it provided no additional features that were actually used.
Numerous changes to use sites that previously relied on the JDK types.
Remove the `Future.cancel()` method that took a boolean argument — this argument was always ignored in our implementations, which was another spec deviation.
Various `invoke*` and `shutdown*` methods have been removed from the EvenExecutorGroup API since it no longer extends ScheduledExecutorService — these were either not used anywhere, or deprecated with better alternatives available.
Updates to cancellation javadocs.

Result:
Cleaner code, leaner API.
2021-09-08 09:06:28 +02:00
Chris Vest
3cbb41a478
Make ByteCursor implementations static final inner classes (#11662)
Motivation:
People might be tempted to use mocking tools like Mockito.spy() on the ByteCursors.
By returning instances where the concrete classes are final, we will be forcing integrators to use stub-like wrappers instead.
Such stubs are more well-behaved since they are implemented in terms of the real instance.
This prevents the mocked objects from (easily) producing behaviour that violates the API specification.

Modification:
All ByteCursor implementations have changed from using anonymous inner classes, to using static-final named inner classes.

Result:
The concrete ByteCursor classes can no longer be extended via byte code generation, such as from mocking tools.
2021-09-08 09:04:16 +02:00
kushalagrawal
3a23094b81 Updated "CipherSuitesConverter" to make it public. (#11656)
Motivation:
While using netty there is sometimes need to handle the cipher suites and
signature algorithm in more strict environment. CipherSuiteConverter is
quite helpful in converting Cipher-Suites to and from java to openSSL.
Making it public would be helpful for the project which are using netty.

Modification:
Updated "CipherSuitesConverter" to make it public.
updated "CipherSuitesConverter.toOpenssl" to public.
updated "CipherSuitesConverter.toJava" to public.

Result:
Fixes #11655
2021-09-06 15:08:06 +02:00
Norman Maurer
ee54ea725a
Add default methods to EventExecutor / EventExecutorGroup / EventLoop / EventLoopGroup (#11649)
Motivation:

We can remove some classes and duplication if we add default methods

Modifications:

- Add default methods to EventExecutor / EventExecutorGroup / EventLoop / EventLoopGroup
- Remove code duplication
- Remove AbstractEventExecutorGroup as it is not needed anymore

Result:

Cleanup and removal of code-duplication. Also makes it easier for people to implement their custom executors / groups
2021-09-05 20:05:33 +02:00
Nitesh Kant
683ff4230e
Add Channel#bufferAllocator() (#11651)
__Motivation__

As we start to migrate codecs to use the new `Buffer` API, we need a way for them to get a handle of `BufferAllocator`.

__Modification__

Added `bufferAllocator()` method to `ChannelConfig`, `Channel` and `ChannelHandlerContext`

__Result__

Codecs can allocate `Buffer` instances
2021-09-03 11:05:26 -07:00
Norman Maurer
e97cb12b24
Remove deprecated EventLoopGroups (#11648)
Motivation:

The way how an EventLoopGroup is created for a specific transport has changed. We should remove all the old deprecated implementations and change all our code to use the new way how to init groups

Modifications:

- Remove LocalEventLoopGroup, NioEventLoopGroup, EpollEventLoopGroup and KQueueEventLoopGroup
- Adjust code to use the new way how to setup EventLoopGroups

Result:

Remove deprecate classes and usages
2021-09-03 09:21:46 -07:00
Norman Maurer
06bc52f59e
Remove UnaryPromiseNotifier and just use Future.cascadeTo(...) (#11650) 2021-09-03 09:35:20 +02:00
Idel Pivnitskiy
3a1a3de4e9 Include number of maximum active streams in exception message (#11644)
Motivation:

When users receive "Maximum active streams violated for this endpoint"
exception, it's useful to know what is the current max streams limit on
HTTP/2 connection.

Modifications:

- Include current number of maximum active streams in exception message;

Result:

Easier debugging of HTTP/2 connections.
2021-09-03 08:54:00 +02:00
Violeta Georgieva
ced712d4f7 Ensure HttpData#addContent/setContent releases the buffer before throwing IOException (#11621)
Motivation:
When the ByteBuf size exceeds the max limit/defined size, IOException is thrown.
HttpData#addContent/setContent should release the buffer in such cases otherwise memory leak will happen.

Modification:
- Release the provided ByteBuf before throwing IOException
- Add unit tests

Result:
Fixes #11618
2021-09-02 14:01:45 -07:00
skyguard1
6339b240d4 Remove useless imports and redundant type cast (#11639)
Motivation:

There are some redundant imports and unnecessary type cast

Modification:

Remove useless imports and unnecessary type cast

Result:

The code is cleaner than original

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
2021-09-02 19:23:15 +02:00
Idel Pivnitskiy
6a2b9a76fe Always log Http2UnknownStreamError at DEBUG level (#11643)
Motivation:

Exception logged by `onHttp2UnknownStreamError` is propagated to the
upper layers anyway. Receiver of the exception is responsible for
correct handling. Inside netty, it's enough to log at `DEBUG` level.

Modifications:

- `Http2FrameCodec#onHttp2UnknownStreamError` always logs at `DEBUG`
level;

Result:

Less noise in logs when `Http2UnknownStreamError` is properly handled by
upper layers.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
Co-authored-by: Chris Vest <mr.chrisvest@gmail.com>
2021-09-02 19:12:11 +02:00
Norman Maurer
cba3b4dd57
Change visibility of DefaultPromise to push users to use EventExecutor methods (#11646)
Motivation:

While we use DefaultPromise as our implementation of Promise and Future users should not really use it directly. Users should always use the EventExecutor / EventLoop to create a Promise / Future.

Modifications:

- Change static Promise methods to be package-private
- Add default implementations for Promise and Future creation to EventExecutor
- Change public constructor to protected
- Remove usage of DefaultPromise in our tests

Result:

Less likely users will depend on the actual Promise implementation
2021-09-02 19:11:01 +02:00
Chris Vest
0cb4cc4e49
Make Promise not extend Future (#11634)
Motivation:
We wish to separate these two into clearer write/read interfaces.
In particular, we don't want to be able to add listeners to promises, because it makes it easy to add them out of order.
We can't prevent it entirely, because any promise can be freely converted to a future where listeners can be added.
We can, however, discourage this in the API.

Modification:
The Promise interface no longer extends the Future interface.
Numerous changes to make the project compile and its tests run.

Result:
Clearer separation of concerns in the code.
2021-09-02 10:46:54 +02:00
Francesco Nigro
23601902ab O(1) buffer next capacity computation (#11641)
Motivation:

Enlarging buffers approaching 4 MiB size requires n iterations

Modification:

Use a single instruction to compute the next buffer capacity

Result:

Faster/Simpler calculateNewCapacity
2021-09-01 17:48:47 +02:00
Norman Maurer
a39fea736d Fix release problems caused by the testsuite-shading module (#11640)
Motivation:

Last time when we tried to do a release it failed due the fact that no javadoc jar was generated. Beside this how we did configure the shading module did introduce a lot of duplication

Modifications:

- Generate an empty javadoc jar
- Share config for shading plugin

Result:

No more problems during release process
2021-09-01 17:43:35 +02:00
Chris Vest
782d70281e
Reduce reliance on ScheduledFuture (#11635)
Motivation:
If we don't need the scheduled future, then it will be one less complication when we change Netty Future to no longer extend JDK Future.
It would also result in fewer elements in our API.

Modification:
There was only one real use of ScheduledFuture in our code, in Cache.
This has been changed to wrap an ordinary future with a deadline for implementing the Delayed interface.
All other places were effectively overspecifying by relying on ScheduledFuture.
A few places were also specifying JDK Futures - these have been changed to specify Netty Futures.

Result:
Reduced dependency on the ScheduledFuture interfaces.
2021-08-31 16:06:34 +02:00
Norman Maurer
a3c44f5a99
Adjust usage of ChannelFutureListeners.CLOSE to use the ChannelHandlerContext (#11631)
Motivation:

Usually the outbound operation should start at the "current" ChanneöHandlercontext which was often not the case

Modifications:

Use the ChannelHandlerContext for closing the connection

Result:

Start the operation on the right position of the pipeline
2021-08-31 12:49:30 +02:00
Chun-Han, Hsiao
8cc34e5c4c [HTTP2] Fix memory leak while writing empty data frame with padding (#11633)
Motivation:

There is a memory leak while writing empty data frame with padding.

The empty data frame was occurred because we are running a proxy server
built by netty, and found that google services always sent data frames
followed by an empty data frame.

Modifications:

Calls the ctx.write even the payload is empty.

Result:

Fix memory leak.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-08-31 12:49:12 +02:00
Violeta Georgieva
14102eb98c Ensure DiskFileUpload#toString returns a string that uses the correct deleteOnExit (#11628)
Motivation:
DiskFileUpload#toString should use the value provided when constructing DiskFileUpload
and not the default one

Modification:
- DiskFileUpload#toString is modified to use the correct deleteOnExit value

Result:
DiskFileUpload#toString returns a string that uses the correct deleteOnExit
2021-08-31 09:01:53 +02:00
Norman Maurer
c039f82483 Add missing annotations to test overrides (#11630)
Motivation:

When we override a test with params we also need to ensure we add the correct annotations to the override

Modifications:

Add the correct annotations so the tests are actually run in intellij

Result:

Each test can be run
2021-08-31 08:08:08 +02:00
Norman Maurer
887526374b Fix leak in TcpDnsTest (#11632)
Motivation:

We did get a report for a leak on the CI.

Modifications:

Fix leak in test by correctly releasing the query

Result:

No more leaks
2021-08-31 08:06:45 +02:00
kushalagrawal
4f1c5b9f95 Added "RSASSA-PSS" algorithm in allowed algorithm list. (#11626)
Motivation:
While OpenSSK is provided support for the "RSASSA-PSS" algorithm this was still not valid from netty. Which was causing issue in validating certificates which was signed using this algorithm.

Modification:
Added "RSASSA-PSS" in LOCAL_SUPPORTED_SIGNATURE_ALGORITHMS.
validation:
Validated and tested with CA and User cert singed with RSASSA-PSS algorithm.

Result:
Fixes #11360

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-08-30 17:24:48 -07:00
Norman Maurer
73d2492269
Replace master with main branch (#11629)
Motivation:

We should not use master as branch name, better to switch to main

Modifications:

Replace master branch name with main

Result:

Use main as replacement for master
2021-08-30 15:37:49 +02:00
Norman Maurer
d7580b526a
Remove public API's that take Promise (#11625)
Motivation:

We did recently change the Channel / ChannelHandler API to always act on the Future only. We should do the same for our handlers.

Modifications:

- Adjust http2 API
- Adjust other handlers API

Result:

Easier to use API and more consistent
2021-08-30 13:15:14 +02:00
Norman Maurer
73377f0fd5
Make our ChannelFutureListeners less restrictive (#11627)
Motivation:

Our current ChannelFutureListeners are too restrictive in terms of the type of the context. We should lift this a bit

Modifications:

Refactor ChannelFutureListeners to not be an enum and so allow most of its instances to be used with any ChannelOutboundInvoker.

Result:

More flexible usage
2021-08-30 13:14:33 +02:00
Norman Maurer
e04f48d802
Add cascadeTo methods to Future (#11623)
Motivation:

The need of cascade from a Future to a Promise exists. We should add some default implementation for it.

Modifications:

- Merge PromiseNotifier into Futures
- Add default cascadeTo(...) methods to Future
- Add tests to FuturesTest
- Replace usage of PromiseNotifier with Future.cascadeTo
- Use combination of map(...) and cascadeTo(...) in *Bootstrap to reduce code duplication

Result:

Provide default implementation of cascadeTo.
2021-08-29 15:44:34 +02:00
skyguard1
3fc44b0a8e Fix QueryStringEncoder encodes tilde (#11590)
Motivation:

In this issue(#11578 ) discussed that tilde should not be encoded in QueryStringEncoder, this pr is to fix this problem

Modification:

Modified QueryStringEncoder so that it does not encode tilde, and added a test case

Result:

Fixes #11578
2021-08-26 12:56:03 -07:00
Chris Vest
445f747ce3
Add a Futures class with static map() and flatMap() methods (#11607)
Motivation:
Making futures easier to compose, combine, and extend is useful to have as part of the API, since implementing this correctly and efficiently can be tricky.

Modification:
Add `Future.map(Function<V,R>) -> Future<R>` and `Future.flatMap(Function<V,Future<R>>) -> Future<R>` default methods to the `Future` interface.
These methods return new Future instance, that will be completed when the original future completes, and the result will be processed through the given mapping function.
These two methods take care to propagate cancellation and exceptions correctly:
Cancellation propagates both ways between the new and original future.
Failures only propagate from the original future to the returned new Future instance.

Result:
A few convenient methods for modifying and composing futures.

This PR fixes #8523, and perhaps also #2105
2021-08-26 11:23:12 +02:00
Chris Vest
584a275a7b Try to make the PR Reports workflow more robust (#11620)
Motiviation:
The workflow from the default branch is the one that gets to run for all PRs, regardless of what branch they are targeting.
This means the 4.1 PR Reports workflow needs to tollerate master branch PRs, where there is no Java 8 builds.

Modification:
Make the PR Reports tolerate that the Java 8 matrix fails if they are missing.

Result:
Hopefully we'll get working PR Reports for master branch PRs now.
2021-08-26 11:20:53 +02:00
Chris Vest
edf4e28afa
Change !future.isSuccess() to future.isFailed() where it makes sense (#11616)
Motivation:
The expression "not is success" can mean that either the future failed, or it has not yet completed.
However, many places where such an expression is used is expecting the future to have completed.
Specifically, they are expecting to be able to call `cause()` on the future.
It is both more correct, and semantically clearer, to call `isFailed()` instead of `!isSuccess()`.

Modification:
Change all places that used `!isSuccess()` to  mean that the future had failed, to use `isFailed()`.
A few places are relying on `isSuccess()` returning `false` for _incomplete_ futures, and these places have been left unchanged.

Result:
Clearer code, with potentially fewer latent bugs.
2021-08-26 09:43:17 +02:00
Norman Maurer
2a7cc36536
Fix test-failures caused by NPE (#11619)
Motivation:

We did miss to update some code that setup the EventLoop mock and so did see test-failures due NPE's

Modifications:

Correctly mock the newPromise() method.

Result:

No more test-failures
2021-08-26 09:41:41 +02:00
Norman Maurer
e21591fa25 Don't throw if null is given as ByteBuf when adding components. (#11613)
Motivation:

232c669fa4 did add some overflow protection but did not handle null elements in the array the same as before.

Modifications:

- Break the loop if a null element was found
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/11612
2021-08-26 08:21:06 +02:00