Commit Graph

10469 Commits

Author SHA1 Message Date
Norman Maurer
98c446c493 Revert "Only do a partial build for a few jobs when verify PRs (#11464)"
This reverts commit e5adc1131a.
2021-07-13 15:32:42 +02:00
skyguard1
df22356a3a
Refactor HttpContentCompressor using CompressionEncoderFactory (#11480)
Motivation:

The `HttpContentCompressor.beginEncode()` method has too many if else, so consider refactoring

Modification:

Create the corresponding `CompressionEncoderFactory` according to the compression algorithm, remove the if else

Result:

The code of `HttpContentCompressor` is cleaner than the previous implementation

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-13 12:25:30 +02:00
Aayush Atharva
6085cd7ab6
Update HttpContentCompressor to pass correct message to ObjectUtil (#11482)
Motivation:
In #11256, We introduced `Iterable` as a parameter but later in review, it was removed. But we forgot to change `compressionOptionsIterable` to just `compressionOptions`.

Modification:
Changed `compressionOptionsIterable` to `compressionOptions`.

Result:
Correct ObjectUtil message
2021-07-13 08:11:01 +02:00
Adrian Antkowiak
8cbdefae23
HAProxyMessage.sourceAddress() can be null (#11475)
Motivation:

If `send-proxy-v2` is used `sourceAddress()` can be `null`.

Modification:

Update docs to reflect this.

Result:

Docs are more correct.
2021-07-12 11:55:39 +02:00
skyguard1
6ce36f1909
Add zstd http content compression support (#11470)
Motivation:

netty needs to support zstd content-encoding http content compression

Modification:

Add ZstdOptions, and modify HttpContentCompressor and CompressorHttp2ConnectionEncoder to support zstd compression

Result:

netty supports zstd http content compression

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-12 08:49:04 +02:00
Violeta Georgieva
3f9a5f50be
Add support for Unix domain datagram sockets when using native epoll/kqueue transport (#11423)
Motivation:

There are use cases when Unix domain datagram sockets are needed for communication. This PR adds such support for Epoll/KQueue.

Modification:

- Expose Channel, Config and Packet interfaces/classes for Unix domain datagram sockets. All interfaces/classes are in `transport-native-unix-common` module in order to be available for KQueue and Epoll implementations
- Add JNI code for Unix domain datagram sockets
- Refactor `DatagramUnicastTest` so that it can be used for testing also Unix domain datagram sockets
- Add Unix domain datagram sockets implementation for KQueue transport
- Add Unix domain datagram sockets implementation for Epoll transport

Result:

Fixes #6737
2021-07-09 09:45:43 +02:00
Norman Maurer
e5adc1131a
Only do a partial build for a few jobs when verify PRs (#11464)
Motivation:

Currently the PR verification takes a lot of time. We should try to only run builds for the affected modules in most cases to speed up the build.

Modifications:

Adjust docker files and workflow to only run build for affected modules for a lot of jobs while still run full builds for windows + java11 with BoringSSL

Result:

Hopefully quicker feedback loop when verify PRs
2021-07-09 06:47:06 +02:00
Ikko Ashimine
d1ef28a22f
Fix typo in ReferenceCountedOpenSslEngine (#11467)
Motivation:

There should be no typos in comments

Modifications:
```
alway -> always
```

Result:

Fixed typo.
2021-07-08 16:29:58 +02:00
Norman Maurer
40fb6026ef
Introduce OpenSslAsyncPrivateKeyMethod which allows to asynchronously sign / decrypt the private key (#11390) (#11460)
Motivation:

At the moment we only support signing / decrypting the private key in a synchronous fashion. This is quite limited as we may want to do a network call to do so on a remote system for example.

Modifications:

- Update to latest netty-tcnative which supports running tasks in an asynchronous fashion.
- Add OpenSslAsyncPrivateKeyMethod interface
- Adjust SslHandler to be able to handle asynchronous task execution
- Adjust unit tests to test that asynchronous task execution works in all cases

Result:

Be able to asynchronous do key signing operations
2021-07-08 16:19:22 +02:00
Aayush Atharva
fef761d03e
Introduce BrotliEncoder (#11256)
Motivation:
Currently, Netty only has BrotliDecoder which can decode Brotli encoded data. However, BrotliEncoder is missing which will encode normal data to Brotli encoded data.

Modification:
Added BrotliEncoder and CompressionOption

Result:
Fixes #6899.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-08 11:51:27 +02:00
skyguard1
7825fa8d7a
Add zstd http header value (#11463)
Motivation:

ZSTD has a wide range of uses on the Internet, so should consider adding `application/zstd` HTTP media-type and `zstd` content-encoding, see  https://tools.ietf.org/html/rfc8478

Modification:

Add `application/zstd` HTTP media-type and `zstd` content-encoding

Result:

netty provides `application/zstd` HTTP media-type and `zstd content-encoding` as http headers

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-08 11:44:54 +02:00
Norman Maurer
3e350993b7
Add script which can be used to build affected modules only (#11461)
Motivation:

At the moment we always build all modules. This script can be used to only build affected modules for a given change

Modifications:

Add script that will only build modules that are affected by a change

Result:

More targeted build
2021-07-08 10:18:16 +02:00
Norman Maurer
7e39b96402
Only run one SSL task per delegation (#11462)
Motivation:

We should only run one SSL task per delegation to allow more SSLEngines to make progress in a timely manner

Modifications:

- Only run one task per delegation to the executor
- Only create new SSL task if really needed
- Only schedule if not on the EventExecutor thread

Result:

More fair usage of resources and less allocations
2021-07-08 07:56:15 +02:00
Aayush Atharva
ede7a604f1
Add SslProtocols and Cipher suites constants (#11457)
Motivation:
Protocols and Cipher suites constants to prevent typos in protocol and cipher suites names and ease of use.

Modification:
Added Protocols and Cipher suites as constants in their respective classes.

Result:
Fixes #11393
2021-07-07 21:15:43 +02:00
Norman Maurer
0b2e955aff Revert "Introduce OpenSslAsyncPrivateKeyMethod which allows to asynchronously sign / decrypt the private key (#11390)"
This reverts commit 7c57c4be17.
2021-07-07 08:26:27 +02:00
Norman Maurer
ac91eaaae8
Ensure we only try to wrap if handler was not removed yet. (#11455)
Motivation:
7c57c4be17 did add a way to async sign keys but did not guard against the handler been removed before try to wrap in cause of an error which could lead to a harmless NPE.

Modifications:

Add check

Result:

No more harmless NPE
2021-07-06 15:01:49 +02:00
skyguard1
266c987339
[Feature] Add zstd encoder (#11437)
Motivation:

As discussed in #10422, ZstdEncoder can be added separately

Modification:

Add ZstdEncoder separately

Result:

netty supports ZSTD with ZstdEncoder

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-06 14:57:09 +02:00
Nitesh Kant
f2295628e9
ApplicationProtocolNegotiationHandler should drain buffer messages on channel close (#11445)
__Motivation__

`ApplicationProtocolNegotiationHandler` buffers messages which are read before SSL handshake complete event is received and drains them when the handler is removed. However, the channel may be closed (or input shutdown) before SSL handshake  event is received in which case we may fire channel read after channel closure (from `handlerRemoved()`).

__Modification__

Intercept `channelInactive()` and input closed event and drain the buffer.

__Result__

If channel is closed before SSL handshake complete event is received, we still maintain the order of message read and channel closure.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-06 14:01:17 +02:00
Norman Maurer
7c57c4be17
Introduce OpenSslAsyncPrivateKeyMethod which allows to asynchronously sign / decrypt the private key (#11390)
Motivation:

At the moment we only support signing / decrypting the private key in a synchronous fashion. This is quite limited as we may want to do a network call to do so on a remote system for example.

Modifications:

- Update to latest netty-tcnative which supports running tasks in an asynchronous fashion.
- Add OpenSslAsyncPrivateKeyMethod interface
- Adjust SslHandler to be able to handle asynchronous task execution
- Adjust unit tests to test that asynchronous task execution works in all cases

Result:

Be able to asynchronous do key signing operations
2021-07-06 08:25:31 +02:00
Norman Maurer
ae7944459d Fix test failure introduced by 7cc31b8653 2021-07-05 08:37:41 +02:00
Norman Maurer
7cc31b8653
Use Junit5 for handler module (#11444)
Motivation:

We should aim to use junit5 everywhere

Modifications:

Migrate most of the handler module to use junit5

Result:

Part of #10757
2021-07-02 15:06:06 +02:00
Kasimir Torri
c97981403d
Improve PerMessageDeflateClientExtensionHandler (#11413)
Motivation:

The `PerMessageDeflateClientExtensionHandler` has the following strange behaviors currently:

* The `requestedServerNoContext` parameter doesn't actually add the `server_no_context_takeover` parameter to the client offer; instead it depends on the requested server window size.
* The handshake will fail if the server responds with a `server_no_context_takeover` parameter and `requestedServerNoContext` is false. According to RFC 7692 (7.1.1.1) the server may do this, and this means that to cover both cases one needs to use two handshakers in the channel pipeline: one with `requestedServerNoContext = true` and one with `requestedServerNoContext = false`.
* The value of the `server_max_window_bits` parameter in the server response is never checked (should be between 8 and 15). And the value of `client_max_window_bits` is checked only in the branch handling the server window parameter.

Modification:

* Add the `server_no_context_takeover` parameter if `requestedServerNoContext` is true.
* Accept a server handshake response which includes the server no context takeover parameter even if we did not request it.
* Check the values of the client and server window size in their respective branches and fail the handshake if they are out of bounds.

Result:

There will be no need to use two handshakers in the pipeline to be lenient in what handshakes are accepted.
2021-07-02 14:47:59 +02:00
Norman Maurer
842e73f8d3
Migrate microbenchmark to junit5 (#11440) (#11443)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust microbenchmark to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-02 08:05:03 +02:00
Norman Maurer
7c6be99da0
Migrate rest of epoll module to junit5 (#11442)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust missing epoll tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-02 08:04:48 +02:00
Norman Maurer
5179e53294
Migrate codec-http to junit5 (#11440)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec-http tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-01 18:53:41 +02:00
Norman Maurer
c82c17782c
Migrate common to junit5 (#11431) (#11438)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust common tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-01 18:19:44 +02:00
Norman Maurer
8003ea8a03
Migrate transport-udt to junit5 (#11439)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust transport-udt tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-01 18:19:24 +02:00
Aleksandar Gradinac
bc206ff173
Delay initialization of io.netty.handler.codec.compression.BrotliDecoder to runtime (#11428)
Motivation:
Including codec-http in the project and building a native-image out of it using a GraalVM 21.2 nightly can result in a failure.

Modification:
By delaying the initialization of `io.netty.handler.codec.compression.BrotliDecoder` to runtime, native-image will not try to eagerly initialize the class during the image build, avoiding the build failure described in the issue.

Result:
Fixes #11427
2021-07-01 15:08:51 +02:00
Aayush Atharva
750d23583c
Add ALPN Buffering to support HTTP/2 Prior Knowledge (#11407)
Motivation:
Currently, Netty cannot handle HTTP/2 Preface messages if the client used the Prior knowledge technique. In Prior knowledge, the client sends an HTTP/2 preface message immediately after finishing TLS Handshake. But in Netty, when TLS Handshake is finished, ALPNHandler is triggered to configure the pipeline. And between these 2 operations, if an HTTP/2 preface message arrives, it gets dropped.

Modification:

Buffer messages until we are done with the ALPN handling.

Result:
Fixes #11403.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-01 14:10:52 +02:00
Norman Maurer
126e9f873b
Revert "Fix deprecated GC logging switch (#11395)" (#11436)
This reverts commit 077e5f282c.
2021-07-01 12:09:52 +02:00
Alex Budovski
077e5f282c
Fix deprecated GC logging switch (#11395)
Motivation:

This caused test failures due to the deprecation warning and produced a
dumpstream.

Modification:

Replace deprecated flag with recommended one.

Result:
Fix deprecation and cause of test failure in codec project.
2021-07-01 11:30:30 +02:00
Maksym Ostroverkhov
2abe20a6b5
Tail tasks queue: configure separately from tasks queue (#11400)
Motivation:

IO transports (primarily epoll, but also applies to kqueue, nio) cant be configured with separate tail tasks queue factory -
instead single queue factory is used for both normal tasks and tail tasks.

Modifications:

Add constructor accepting tail EventLoopTaskQueueFactory to aforementioned transports

Result:

IO transports can be configured with separate tail tasks
2021-07-01 10:51:10 +02:00
Norman Maurer
20e4ccbd33
Skip Brotli related tests on platforms which not support Brotli (#11435)
Motivation:

The native module is not yet available on aarch64 Mac / Windows thus causing tests in codec/ to fail (specifically all the Brotli ones, since the module could not be loaded).

Modification:

Disable Brotli tests when platform is not supported

Result:

Tests under codec/ now pass under Mac/aarch64 and Windows/aarch64
2021-07-01 09:59:24 +02:00
wujimin
3226e77485
Add support for GMSSL (#11406) (#11410)
__Motivation__

Add support for GMSSL protocol to SslUtils.

__Modification__

Modify `SslUtils.getEncryptedPacketLength(ByteBuf buffer, int offset)` to get packet length when protocol is GMSSL.
Modify `SslUtils.getEncryptedPacketLength(ByteBuffer buffer)` to get packet length when protocol is GMSSL.

__Result__

`SslUtils.getEncryptedPacketLength` now supports GMSSL protocol. Fixes https://github.com/netty/netty/issues/11406
2021-07-01 08:17:45 +02:00
Stephane Landelle
801819b359
Fix HttpHeaderValue#XML_HTTP_REQUEST case (#11433)
Motivation:

HTTP header values are case sensitive. The expected value for `x-request-with` header is `XMLHttpRequest`, not `XmlHttpRequest`.

Modification:

Fix constant's case.

Result:

Correct `XMLHttpRequest` HTTP header value.
2021-07-01 08:13:10 +02:00
Tamara Braun
194a81ff4f
Fix Buffer Overflow During Lz4FrameEncoder Close (#11429)
Motivation:

We failed to account for the last header when estimating the buffer
size. If the data does not compress enough to make space for the
last header we would exceed the ByteBuf's capacity.

Modifications:

Call #ensureWritable with appropriate capacity for footer ByteBuf
befor writing footer.

Result:

If there is not enough space left in the buffer, the buffer will be
expanded.
2021-06-30 18:47:07 +02:00
Norman Maurer
8d76f402b1
Migrate transport to junit5 (#11431) (#11432)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust transport tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 16:17:57 +02:00
Norman Maurer
94a4880358
Migrate codec-mqtt to junit5 (#11431)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec-mqtt tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 15:32:16 +02:00
Norman Maurer
3716575102
Migrate codec to junit5 (#11426) (#11430)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 14:12:50 +02:00
Norman Maurer
be606df4f0
Migrate codec-redis to junit5 (#11426)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec-redis tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 13:59:57 +02:00
Norman Maurer
2c3e78566e
Migrate resolver to junit5 (#11425)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust resolver tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 12:19:02 +02:00
Norman Maurer
29be99c538
Migrate the rest of codec-http2 to junit5 (#11424)
Motivation:

8c73dbe9bd did migrate the codec-http2 code to use junit5 but missed two classes.

Modifications:

Adjust the rest of codec-http2 tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 11:11:25 +02:00
Norman Maurer
8c73dbe9bd
Migrate codec-http2 to junit5 (#11422)
Motivation:

We should update to use junit5 in all modules.

Modifications:

Adjust codec-http2 tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-06-30 10:32:01 +02:00
Unev
d8ad931488 ByteBufFormat constructor for LoggingHandler (#11420)
__Motivation__

`LoggingHandler` misses a constructor variant that only takes `ByteBufFormat`

__Modification__

Added the missing constructor variant.

__Result__

`LoggingHandler` can be constructed with `ByteBufFormat` only.

Co-authored-by: Nitesh Kant <nitesh_kant@apple.com>
2021-06-29 10:28:27 -07:00
skyguard1
d99a8f75b4
Use Two way algorithm to optimize ByteBufUtil.indexOf() method (#11367)
Use Two way algorithm to optimize ByteBufUtil.indexOf() method

Motivation:

ByteBufUtil.indexOf can be inefficient for substring search on
ByteBuf, in terms of algorithm complexity (O(needle.readableBytes * haystack.readableBytes)), consider using the Two Way algorithm to optimize the ByteBufUtil.indexOf() method

Modification:

Use the Two Way algorithm to optimize ByteBufUtil.indexOf() method.

Result:

The performance of the ByteBufUtil.indexOf() method is higher than the original implementation
2021-06-28 11:07:17 +02:00
Norman Maurer
6c618e30af
CombinedChannelDuplexHandler.removeOutboundHandler() cause connect(...) to not pass the correct parameters (#11414)
Motivation:

Due a bug we did not pass the correct remote and localaddress to the next handler if the outbound portion of the CombinedChannelDuplexHandler was removed

Modifications:

- Call the correct connect(...) method
- Refactor tests to test that the parameters are correctly passed on
- Remvoe some code duplication in the tests

Result:

CombinedChannelDuplexHandler correctly pass parameters on
2021-06-24 13:58:17 +02:00
Norman Maurer
a71ec15fc4
Correctly use HandshakeStatus.NEED_WRAP when a handshake failed and a alert was produced (#11412)
Motivation:

We need to ensure we always "consumed" all alerts etc via SSLEngine.wrap(...) before we teardown the engine. Failing to do so may lead to a situation where the remote peer will not be able to see the actual cause of the handshake failure but just see the connection being closed.

Modifications:

Correctly return HandshakeStatus.NEED_WRAP when we need to wrap some data first before we shutdown the engine because of a handshake failure.

Result:

Fixes https://github.com/netty/netty/issues/11388
2021-06-24 10:05:47 +02:00
Norman Maurer
6edb231bf5
Update to netty-tcnative 2.0.40.Final (#11411)
Motivation:

A new version of netty-tcnative was released

Modifications:

Update to 2.0.40.Final

Result:

Use latest version
2021-06-23 14:21:59 +02:00
Nitesh Kant
5a658bb887
HttpUtil#normalizeAndGetContentLength() should handle empty value (#11409)
__Motivation__

`HttpUtil#normalizeAndGetContentLength()` throws `StringIndexOutOfBoundsException` for empty `content-length` values, it should instead throw `IllegalArgumentException` for all invalid values.

__Modification__

- Throw `IllegalArgumentException` if the `content-length` value is empty.
- Add tests

__Result__

Fixes https://github.com/netty/netty/issues/11408
2021-06-23 12:07:16 +02:00
ping
98a3a0c0cb
Recycler.WeakOrderQueue drop Object hasBeenRecycled (#11402)
Motivation:

WeakOrderQueue would drop object that has been recycled, even when it has space for it.
WeakOrderQueue#add should check DefaultHandler.hasBeenRecycler field  first

Modifications:

WeakOrderQueue test the DefaultHandler.hasBeenRecycler first

Result:

WeakOrderQueue would not drop object that has been recycled when there is space


Co-authored-by: Norman Maurer <norman_maurer@apple.com>
Co-authored-by: Trustin Lee <t@motd.kr>
2021-06-23 11:32:44 +02:00