Commit Graph

10464 Commits

Author SHA1 Message Date
Aayush Atharva
3085ea3d0b Bump Brotli4j to 1.5.0 (#11509)
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.

See https://github.com/hyperxpro/Brotli4j/pull/27

Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.

Result:
Up-to-date Broti4j library.
2021-07-26 08:30:45 +02:00
Norman Maurer
85eb96c68e
Use junit5 methods (#11508)
Motivation:

We missed to update one junit4 method usage to junit5

Modifications:

Use junit5 methods

Result:

No more usage of junit4
2021-07-26 08:28:32 +02:00
Norman Maurer
119f34889d Be able to build on JDK17 (#11500)
Motivation:

As the release of JDK17 is getting closer and there are ea builds already we should ensure we can actually build netty with it.

Modifications:

- Add profile for JDK17
- Remove test-code that would fail with JDK17 due the changes in 4f4d0f5366.

Result:

Be able to build and run testsuite with JDK17
2021-07-23 14:16:18 +02:00
Norman Maurer
6c4c8fd6d4 Update japicmp plugin to support Java16+ (#11507)
Motivation:

japicmp did fix some issues related to Java16+. Let's update so we can also enable it in later java versions

Modifications:

Update to 0.15.3

Result:

Be able to use japicmp with all java versions
2021-07-23 11:51:38 +02:00
Norman Maurer
ae41a5c28b FastLzFrameDecoder should use allocator to allocate output buffer (#11499)
Motivation:

FastLzFrameDecoder currently not use the allocator to alocate the output buffer. This means that if you use the PooledByteBufAllocator you still can't make use of the pooling. Beside this the decoder also does an uncessary memory copy when no compression is used.

Modifications:

- Allocate the output buffer via the allocator
- Don't allocate and copy if we handle an uncompressed chunk
- Make use of ByteBufChecksum for a few optimizations when running on a recent JDK

Result:

Less allocations when using FastLzFrameDecoder
2021-07-21 22:17:05 +02:00
skyguard1
a1f13e17db Add StringDecoder test (#11496)
Motivation:

There is no test case of `StringDecoder` here

Modification:

Need to add `StringDecoder` test case

Result:

Added test case of `StringDecoder`

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-21 14:31:45 +02:00
Norman Maurer
23d8fde855 Add PromiseNotifier static method which takes care of cancel propagation (#11494)
Motivation:

At the moment we not correctly propagate cancellation in some case when we use the PromiseNotifier.

Modifications:

- Add PromiseNotifier static method which takes care of cancellation
- Add unit test
- Deprecate ChannelPromiseNotifier

Result:

Correctly propagate cancellation of operation

Co-authored-by: Nitesh Kant <nitesh_kant@apple.com>
2021-07-21 13:47:43 +02:00
Aayush Atharva
9398e6481d Update BouncyCastle Libraries (#11495)
Motivation:
New versions of Bouncy Castle libraries are out and we should upgrade to them.

Modification:
Upgraded all Bouncy Castle libraries to the latest version.

Result:
The latest versions of Bouncy Castle libraries.
2021-07-20 15:20:35 +02:00
Heng Zhang
d920cbf143 Distribue weight to children when closing stream (#11490)
Motivation:

As suggested in [section 5.3.4 in http2 spec](https://datatracker.ietf.org/doc/html/rfc7540#section-5.3.4):

> When a stream is removed from the dependency tree, its dependencies can be moved to become dependent on the parent of the closed stream. The weights of new dependencies are recalculated by distributing the weight of the dependency of the closed stream proportionally based on the weights of its dependencies.

For example, we have stream A and B depend on connection stream with default weights (16), and stream C depends on A with max weight (256). When stream A was closed, we move stream C to become dependent on the connection stream, then we should distribute the weight of stream A to its children (only stream C), so the new weight of stream C will be 16. If we keep the weight of stream C unchanged, it will get more resource than stream B

Modification:
- distribute weight to its children when closing a stream
- add a unit test for the case above and fix other related unit tests

Result:

More spec-compliant and more appropriate stream reprioritization

Co-authored-by: Heng Zhang <zhangheng@imo.im>
2021-07-20 14:18:13 +02:00
Chris Vest
c982d45493 Fix a bug where SslHandler clients would not process Server Hello messages in a timely manner (#11472)
Motivation:
The TLS handshake must be able to finish on its own, without being driven by outside read calls.
This is currently not the case when TCP FastOpen is enabled.
Reads must be permitted and marked as pending, even when a channel is not active.

This is important because, with TCP FastOpen, the handshake processing of a TLS connection will start
before the connection has been established -- before the process of connecting has even been started.

The SslHandler on the client side will add the Client Hello message to the ChannelOutboundBuffer, then
issue a `ctx.read` call for the anticipated Server Hello response, and then flush the Client Hello
message which, in the case of TCP FastOpen, will cause the TCP connection to be established.

In this transaction, it is important that the `ctx.read` call is not ignored since, if auto-read is
turned off, this could delay or even prevent the Server Hello message from being processed, causing
the server-side handshake to time out.

Modification:
Attach a listener to the SslHandler.handshakeFuture in the EchoClient, that will call ctx.read.

Result:
The SocketSslEchoTest now tests that the SslHandler can finish handshakes on its own, without being driven by 3rd party ctx.read calls.
The various channel implementations have been updated to comply with this behaviour.
2021-07-15 12:01:16 +02:00
Norman Maurer
23bf55206c Migrate the rest of the ssl package to junit5 (#11483)
Motivation:

We should use junit5 everywhere.

Modifications:

- Refactor rest of tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-15 08:58:50 +02:00
Norman Maurer
d5ddd8b680 Disable flaky test for now (#11488)
Motivation:

JdkOpenSslEngineInteroptTest.testMutualAuthSameCerts() is flaky on the CI and so fails the PR build quite often.
Let's disable it for now until we were able to reproduce it locally and fix it.

Modifications:

Disable flaky test

Result:

More stable CI builds
2021-07-14 16:51:52 +02:00
Nitesh Kant
ef7224480c Improve name matching in DNS answers (#11474)
__Motivation__

Upon receiving a DNS answer, we match whether the name in the question matches the name in the record. Some DNS servers we have encountered append a search domain to the record name which fails this match. eg: for question name `netty` and search domains `io` and `com`, we will do 2 queries: `netty.io.` and `netty.com.`, if the answer for `netty.io` contains `netty.com` then we ignore this record.

__Modification__

If the name in the record does not match the name in the question, append configured search domains to the question name to see if it matches the record name.

__Result__

Records names with appended search domains are still returned as valid answers.
2021-07-14 14:20:20 +02:00
Norman Maurer
0c411859eb SelfSignedCertificate should work in http2 tests (#11486)
Motivation:

We need to add `--add-exports java.base/sun.security.x509=ALL-UNNAMED` when running the tests for codec-http2 as some of the tests use SelfSignedCertificate.

Modifications:

- Add `--add-exports java.base/sun.security.x509=ALL-UNNAMED` when running the tests for codec-http2
- Ensure we export correct when running with JDK12, 13, 14 and 15 as well

Result:

No more tests failure due not be able to access classes
2021-07-14 07:46:31 +02:00
Aayush Atharva
70b3315181 Use Static Constants as Sample Data (#11485)
Motivation:
#11468 was merged but didn't fix tests completely. There is a fight between `LF` and `CRLF`. So to eliminate this, we should just get rid of them.

Modification:
Use a small sample dataset without `LF` and `CRLF`.

Result:
Simple and passing test.
2021-07-14 07:45:48 +02:00
Norman Maurer
1f6577ee92 Remove rest of junit4 usage (#11484)
Motivation:

We did migrate all these modules to junit5 before but missed a few usages of junit4

Modifications:

Replace all junit4 imports by junit5 apis

Result:

Part of  https://github.com/netty/netty/issues/10757
2021-07-13 21:00:53 +02:00
Aayush Atharva
bd2799cb9e Fix JavaDoc and Run Brotli tests on Windows (#11468)
Motivation:
JavaDoc of StandardCompressionOptions should point towards public methods. Also, Brotli tests were failing on Windows.

Modification:
Fixed JavaDoc and enabled Brotli tests on Windows.

Result:
Better JavaDoc and Brotli tests will run on Windows

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-13 16:25:13 +02:00
Aayush Atharva
c5b5ea5e75 Update HttpContentCompressor to pass correct message to ObjectUtil (#11482)
Motivation:
In #11256, We introduced `Iterable` as a parameter but later in review, it was removed. But we forgot to change `compressionOptionsIterable` to just `compressionOptions`.

Modification:
Changed `compressionOptionsIterable` to `compressionOptions`.

Result:
Correct ObjectUtil message
2021-07-13 08:11:54 +02:00
Adrian Antkowiak
aa003f8fec HAProxyMessage.sourceAddress() can be null (#11475)
Motivation:

If `send-proxy-v2` is used `sourceAddress()` can be `null`.

Modification:

Update docs to reflect this.

Result:

Docs are more correct.
2021-07-12 11:56:08 +02:00
skyguard1
154a3e0cab Add zstd http content compression support (#11470)
Motivation:

netty needs to support zstd content-encoding http content compression

Modification:

Add ZstdOptions, and modify HttpContentCompressor and CompressorHttp2ConnectionEncoder to support zstd compression

Result:

netty supports zstd http content compression

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-12 09:03:44 +02:00
Violeta Georgieva
cc92b6c1e6
Add support for Unix domain datagram sockets when using native epoll/kqueue transport (#11476)
Motivation:

There are use cases when Unix domain datagram sockets are needed for communication.
This PR adds such support for Epoll/KQueue.

Modification:

- Expose Channel, Config and Packet interfaces/classes for Unix domain datagram sockets.
All interfaces/classes are in `transport-native-unix-common` module in order to be available
for KQueue and Epoll implementations
- Add JNI code for Unix domain datagram sockets
- Refactor `DatagramUnicastTest` so that it can be used for testing also Unix domain datagram sockets
- Add Unix domain datagram sockets implementation for KQueue transport
- Add Unix domain datagram sockets implementation for Epoll transport

Result:

Fixes #6737
2021-07-12 08:45:10 +02:00
Ikko Ashimine
980b6a0801 Fix typo in ReferenceCountedOpenSslEngine (#11467)
Motivation:

There should be no typos in comments

Modifications:
```
alway -> always
```

Result:

Fixed typo.
2021-07-08 16:30:13 +02:00
Norman Maurer
f8796c7eaf Introduce OpenSslAsyncPrivateKeyMethod which allows to asynchronously sign / decrypt the private key (#11390) (#11460)
Motivation:

At the moment we only support signing / decrypting the private key in a synchronous fashion. This is quite limited as we may want to do a network call to do so on a remote system for example.

Modifications:

- Update to latest netty-tcnative which supports running tasks in an asynchronous fashion.
- Add OpenSslAsyncPrivateKeyMethod interface
- Adjust SslHandler to be able to handle asynchronous task execution
- Adjust unit tests to test that asynchronous task execution works in all cases

Result:

Be able to asynchronous do key signing operations
2021-07-08 16:28:58 +02:00
Aayush Atharva
55c4e2ca82 Introduce BrotliEncoder (#11256)
Motivation:
Currently, Netty only has BrotliDecoder which can decode Brotli encoded data. However, BrotliEncoder is missing which will encode normal data to Brotli encoded data.

Modification:
Added BrotliEncoder and CompressionOption

Result:
Fixes #6899.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-08 12:01:28 +02:00
skyguard1
95a59af549 Add zstd http header value (#11463)
Motivation:

ZSTD has a wide range of uses on the Internet, so should consider adding `application/zstd` HTTP media-type and `zstd` content-encoding, see  https://tools.ietf.org/html/rfc8478

Modification:

Add `application/zstd` HTTP media-type and `zstd` content-encoding

Result:

netty provides `application/zstd` HTTP media-type and `zstd content-encoding` as http headers

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-08 11:45:28 +02:00
Norman Maurer
ed834254ce Add script which can be used to build affected modules only (#11461)
Motivation:

At the moment we always build all modules. This script can be used to only build affected modules for a given change

Modifications:

Add script that will only build modules that are affected by a change

Result:

More targeted build
2021-07-08 10:18:28 +02:00
Norman Maurer
6ac8ef54f7
Remove throws Exception from ChannelHandler methods that handle o… (#11417)
Motivation:

At the moment all methods in `ChannelHandler` declare `throws Exception` as part of their method signature. While this is fine for methods that handle inbound events it is quite confusing for methods that handle outbound events. This comes due the fact that these methods also take a `ChannelPromise` which actually need to be fullfilled to signal back either success or failure. Define `throws...` for these methods is confusing at best. We should just always require the implementation to use the passed in promise to signal back success or failure. Doing so also clears up semantics in general. Due the fact that we can't "forbid" throwing `RuntimeException` we still need to handle this in some way tho. In this case we should just consider it a "bug" and so log it and close the `Channel` in question. The user should never have an exception "escape" their implementation and just use the promise. This also clears up the ownership of the passed in message etc.

As `flush(ChannelHandlerContext)` and `read(ChannelHandlerContext)` don't take a `ChannelPromise` as argument this also means that these methods can never produce an error. This makes kind of sense as these really are just "signals" for the underlying transports to do something. For `RuntimeException` the same rule is used as for other outbound event handling methods, which is logging and closing the `Channel`.

Motifications:

- Remove `throws Exception` from signature
- Adjust code to not throw and just notify the promise directly
- Adjust unit tests

Result:

Much cleaner API and semantics.
2021-07-08 10:16:00 +02:00
Norman Maurer
54aa4d9b68 Only run one SSL task per delegation (#11462)
Motivation:

We should only run one SSL task per delegation to allow more SSLEngines to make progress in a timely manner

Modifications:

- Only run one task per delegation to the executor
- Only create new SSL task if really needed
- Only schedule if not on the EventExecutor thread

Result:

More fair usage of resources and less allocations
2021-07-08 08:06:07 +02:00
Aayush Atharva
04cb23626d Add SslProtocols and Cipher suites constants (#11457)
Motivation:
Protocols and Cipher suites constants to prevent typos in protocol and cipher suites names and ease of use.

Modification:
Added Protocols and Cipher suites as constants in their respective classes.

Result:
Fixes #11393
2021-07-07 21:20:58 +02:00
Norman Maurer
93d428eb23 Revert "Introduce OpenSslAsyncPrivateKeyMethod which allows to asynchronously sign / decrypt the private key (#11390)"
This reverts commit 2b9f4836be.
2021-07-07 08:27:31 +02:00
Norman Maurer
37d4b5a2f7 Call fireUserEventTriggered(...) before we try to modify the pipeline
Motivation:

We should call fireUserEventTriggered(...) before we try to modify the pipeline as otherwise we may end up in the situation that the handler was already removed.

Modifications:

Change ordering of calls

Result:

Test pass again
2021-07-06 16:43:27 +02:00
skyguard1
520d85da11 [Feature] Add zstd encoder (#11437)
Motivation:

As discussed in #10422, ZstdEncoder can be added separately

Modification:

Add ZstdEncoder separately

Result:

netty supports ZSTD with ZstdEncoder

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-06 14:57:39 +02:00
Nitesh Kant
c62eb26b09 ApplicationProtocolNegotiationHandler should drain buffer messages on channel close (#11445)
__Motivation__

`ApplicationProtocolNegotiationHandler` buffers messages which are read before SSL handshake complete event is received and drains them when the handler is removed. However, the channel may be closed (or input shutdown) before SSL handshake  event is received in which case we may fire channel read after channel closure (from `handlerRemoved()`).

__Modification__

Intercept `channelInactive()` and input closed event and drain the buffer.

__Result__

If channel is closed before SSL handshake complete event is received, we still maintain the order of message read and channel closure.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-06 14:03:54 +02:00
Norman Maurer
2b9f4836be Introduce OpenSslAsyncPrivateKeyMethod which allows to asynchronously sign / decrypt the private key (#11390)
Motivation:

At the moment we only support signing / decrypting the private key in a synchronous fashion. This is quite limited as we may want to do a network call to do so on a remote system for example.

Modifications:

- Update to latest netty-tcnative which supports running tasks in an asynchronous fashion.
- Add OpenSslAsyncPrivateKeyMethod interface
- Adjust SslHandler to be able to handle asynchronous task execution
- Adjust unit tests to test that asynchronous task execution works in all cases

Result:

Be able to asynchronous do key signing operations
2021-07-06 09:59:15 +02:00
Norman Maurer
5014f6d7a6
Change reflection config to match constructor (#11453)
Motivation:

We need to change the reflection config to match the constructor that is used

Modifications:

Adjust config

Result:

Graal PR jobs pass again
2021-07-05 21:05:06 +02:00
Norman Maurer
cb82277d36 Use Junit5 for handler module (#11444)
Motivation:

We should aim to use junit5 everywhere

Modifications:

Migrate most of the handler module to use junit5

Result:

Part of #10757
2021-07-02 17:43:13 +02:00
Kasimir Torri
ef231fda50 Improve PerMessageDeflateClientExtensionHandler (#11413)
Motivation:

The `PerMessageDeflateClientExtensionHandler` has the following strange behaviors currently:

* The `requestedServerNoContext` parameter doesn't actually add the `server_no_context_takeover` parameter to the client offer; instead it depends on the requested server window size.
* The handshake will fail if the server responds with a `server_no_context_takeover` parameter and `requestedServerNoContext` is false. According to RFC 7692 (7.1.1.1) the server may do this, and this means that to cover both cases one needs to use two handshakers in the channel pipeline: one with `requestedServerNoContext = true` and one with `requestedServerNoContext = false`.
* The value of the `server_max_window_bits` parameter in the server response is never checked (should be between 8 and 15). And the value of `client_max_window_bits` is checked only in the branch handling the server window parameter.

Modification:

* Add the `server_no_context_takeover` parameter if `requestedServerNoContext` is true.
* Accept a server handshake response which includes the server no context takeover parameter even if we did not request it.
* Check the values of the client and server window size in their respective branches and fail the handshake if they are out of bounds.

Result:

There will be no need to use two handshakers in the pipeline to be lenient in what handshakes are accepted.
2021-07-02 14:51:12 +02:00
Norman Maurer
dbdf9f16c2 Migrate microbenchmark to junit5 (#11440) (#11443)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust microbenchmark to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-02 08:05:18 +02:00
Norman Maurer
f84bfd2dd2 Migrate codec-http to junit5 (#11440)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec-http tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-01 18:56:30 +02:00
Norman Maurer
26a0efcb93 Migrate common to junit5 (#11431) (#11438)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust common tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-01 18:46:46 +02:00
Aleksandar Gradinac
5dc238341a Delay initialization of io.netty.handler.codec.compression.BrotliDecoder to runtime (#11428)
Motivation:
Including codec-http in the project and building a native-image out of it using a GraalVM 21.2 nightly can result in a failure.

Modification:
By delaying the initialization of `io.netty.handler.codec.compression.BrotliDecoder` to runtime, native-image will not try to eagerly initialize the class during the image build, avoiding the build failure described in the issue.

Result:
Fixes #11427
2021-07-01 15:09:11 +02:00
Aayush Atharva
7f690783d1 Add ALPN Buffering to support HTTP/2 Prior Knowledge (#11407)
Motivation:
Currently, Netty cannot handle HTTP/2 Preface messages if the client used the Prior knowledge technique. In Prior knowledge, the client sends an HTTP/2 preface message immediately after finishing TLS Handshake. But in Netty, when TLS Handshake is finished, ALPNHandler is triggered to configure the pipeline. And between these 2 operations, if an HTTP/2 preface message arrives, it gets dropped.

Modification:

Buffer messages until we are done with the ALPN handling.

Result:
Fixes #11403.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-01 14:14:32 +02:00
Alex Budovski
5b27750338 Fix deprecated GC logging switch (#11395)
Motivation:

This caused test failures due to the deprecation warning and produced a
dumpstream.

Modification:

Replace deprecated flag with recommended one.

Result:
Fix deprecation and cause of test failure in codec project.
2021-07-01 12:10:10 +02:00
Norman Maurer
00e482ce2d Skip Brotli related tests on platforms which not support Brotli (#11435)
Motivation:

The native module is not yet available on aarch64 Mac / Windows thus causing tests in codec/ to fail (specifically all the Brotli ones, since the module could not be loaded).

Modification:

Disable Brotli tests when platform is not supported

Result:

Tests under codec/ now pass under Mac/aarch64 and Windows/aarch64
2021-07-01 09:59:41 +02:00
wujimin
ea2654e9cc Add support for GMSSL (#11406) (#11410)
__Motivation__

Add support for GMSSL protocol to SslUtils.

__Modification__

Modify `SslUtils.getEncryptedPacketLength(ByteBuf buffer, int offset)` to get packet length when protocol is GMSSL.
Modify `SslUtils.getEncryptedPacketLength(ByteBuffer buffer)` to get packet length when protocol is GMSSL.

__Result__

`SslUtils.getEncryptedPacketLength` now supports GMSSL protocol. Fixes https://github.com/netty/netty/issues/11406
2021-07-01 08:17:58 +02:00
Stephane Landelle
7aaf9c1b2e Fix HttpHeaderValue#XML_HTTP_REQUEST case (#11433)
Motivation:

HTTP header values are case sensitive. The expected value for `x-request-with` header is `XMLHttpRequest`, not `XmlHttpRequest`.

Modification:

Fix constant's case.

Result:

Correct `XMLHttpRequest` HTTP header value.
2021-07-01 08:13:22 +02:00
Tamara Braun
3afece1e13 Fix Buffer Overflow During Lz4FrameEncoder Close (#11429)
Motivation:

We failed to account for the last header when estimating the buffer
size. If the data does not compress enough to make space for the
last header we would exceed the ByteBuf's capacity.

Modifications:

Call #ensureWritable with appropriate capacity for footer ByteBuf
befor writing footer.

Result:

If there is not enough space left in the buffer, the buffer will be
expanded.
2021-06-30 18:47:32 +02:00
Norman Maurer
fe1ed36cc4 Migrate codec-mqtt to junit5 (#11431)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec-mqtt tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 15:41:01 +02:00
Norman Maurer
1049b02f67 Migrate codec-redis to junit5 (#11426)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec-redis tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 14:00:13 +02:00
Norman Maurer
7c9180e62b Migrate resolver to junit5 (#11425)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust resolver tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 12:20:27 +02:00