Commit Graph

10506 Commits

Author SHA1 Message Date
Norman Maurer
8bcc27a169
Ensure we always wrap if there is something left to be send to the remote peer (#11535)
Motivation:

We need to ensure we call wrap as long as there is something left to be send to the remote peer in cases of non-application data (like for example alerts).

Modifications:

Check the pending data and based on it return NEED_WRAP even when the handshake was done.

Result:

Always produce alerts etc
2021-08-02 10:12:35 +02:00
Aayush Atharva
82418dafec
Add Increment and Decrement count positive check in ReferenceCountUtil (#11523)
Motivation:
We should have guards in place to check increment or decrement count in `ReferenceCountUtil`. Increment and Decrement counts must be a positive integer.

Modification:
Added `ObjectUtil#checkPositive` checks.

Result:
Prevent release due to invalid count.
2021-07-29 20:51:01 +02:00
Norman Maurer
b6da6f5828
Ensure we include hs_err* files when build failed (#11525)
Motivation:

We the build failed we should ensure we also include hs_err* files as it may have failed due a native crash

Modifications:

Adjust path matching to include hs_err as well

Result:

Easier to debug native crashes
2021-07-29 08:48:12 +02:00
Norman Maurer
a2e34ec5e8
Add @UnstableApi to TcpDnsResponseEncoder (#11526)
Motivation:

We should mark TcpDnsResponseEncoder as unstable

Modifications:

Add annotation

Result:

Mark the encoder as unstable
2021-07-28 19:53:08 +02:00
Norman Maurer
8314e2fb04 Fix false-positive report of japicmp-maven-plugin introduced by OCccef8feedd726743d0355b34799e4915536d72ab 2021-07-28 12:06:10 +02:00
Norman Maurer
e419dedc2c Add @UnstableApi annotation to TcpDnsQueryDecoder introduced by ccef8feedd 2021-07-28 11:37:29 +02:00
RockyLOMO
ccef8feedd
Add TcpDnsQueryDecoder and TcpDnsResponseEncoder (#11415)
Motivation:
There is no decoder and encoder for TCP based DNS.

Result:
- Added decoder and encoder
- Added tests
- Added example

Result:

Be able to decode and encode TCP based dns 

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-28 09:24:28 +02:00
Norman Maurer
c83c5b7d7c
JdkZlibDecoder may corrupt data when GZIP footer is fragmented (#11521)
Motivation:

Due a bug in the statemachine we produced an decoding error when the GZIP footer was fragmented in some cases

Modifications:

- Fix statemachine
- Add testcase

Result:

Correctly decode GZIP in all cases
2021-07-28 09:15:37 +02:00
Norman Maurer
412af40f5d
Make DnsNameResolver.cnameCache() public (#11520)
Motivation:

Users may want to clear the cache manually. For this it should be possible to access it first.

Modifications:

Change the visibility to public

Result:

Be able to clear the cache manually. Related to https://github.com/netty/netty/issues/11519
2021-07-27 12:05:28 +02:00
Norman Maurer
e1be815a94
FastLzFrameEncoder should not depend on the buffer type (#11516)
Motivation:

At the moment FastLzFrameEncoder depends on the buffer type. This is not needed and may cause memory copies

Modifications:

Rewrite to be able to just act on the ByteBuf

Result:

Less memory copies
2021-07-27 09:21:02 +02:00
Norman Maurer
c423891fb5
Remove ApplicationProtocolNegotiationHandler when no SslHandler is present (#11503)
Motivation:

750d235 did introduce a change in ApplicationProtocolNegotiationHandler that let the handler buffer all inbound data until the SSL handshake was completed. Due of this change a user might get buffer data forever if the SslHandler is not in the pipeline but the ApplicationProtocolNegotiationHandler was added. We should better guard the user against this "missconfiguration" of the ChannelPipeline.

Modifications:

- Remove the ApplicationProtocolNegotiationHandler when no SslHandler is present when the first message was received
- Add unit test

Result:

No possible that we buffer forever
2021-07-26 20:20:40 +02:00
Norman Maurer
165a035a15
Add some size checks to make code more robust and more clear (#11512)
Motivation:

While its technical impossible that a chunk is larger than 64kb it still makes things easier to read and more robust to add some size checks to LzfDecoder.

Modifications:

Check the maximum length

Result:

More robust and easier to reason about code
2021-07-26 17:11:18 +02:00
Norman Maurer
8af59e4b40
Disable mutual auth tests on windows for now (#11513)
Motivation:

We did observe that the mutal auth tests are flaky on windows when running on the CI. Let's disable these for now.

Modifications:

Disable mutual auth tests on windows

Result:

More stable build. Related to https://github.com/netty/netty/issues/11489
2021-07-26 14:10:26 +02:00
Norman Maurer
ae875c35ec
FastLzFrameDecoder should not need to do any extra memory copies even when direct buffers are used (#11511)
Modifications:

Change code to not depend on heap buffers

Result:

Less memory copies
2021-07-26 11:59:02 +02:00
Norman Maurer
3549c4048c Use junit5 methods (#11508)
Motivation:

We missed to update one junit4 method usage to junit5

Modifications:

Use junit5 methods

Result:

No more usage of junit4
2021-07-26 08:54:01 +02:00
old driver
8f2fe8659b
Optimize method io.netty.util.concurrent.DefaultPromise.await0(...) (#11504)
Motivation:

For the code pattern of `Object.wait(...)`  in `io.netty.util.concurrent.DefaultPromise.await0(...)`, it's better to follow the recommended code pattern according to [Object.wait(...)'s doc](https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait()):
```
synchronized (obj) {
     while (<condition does not hold>)
     obj.wait();
     ... // Perform action appropriate to condition
}
```

Modification:
Change the `Object.wait(...)`'s code pattern in `io.netty.util.concurrent.DefaultPromise.await0(...)`.

Result:

The `Object.wait(...)`'s code pattern in `io.netty.util.concurrent.DefaultPromise.await0(...)` meets the Java doc.
2021-07-26 08:32:53 +02:00
Aayush Atharva
eb99f6c74a
Fix JavaDoc of SelfSignedCertiticate regarding Private Key type (#11510)
Motivation:
SelfSignedCertificate generates EC/RSA key pair and this should be explicitly mentioned in JavaDoc. Currently, only "RSA" was mentioned not "EC".

Modification:
Changed RSA to EC/RSA

Result:
Correct JavaDoc
2021-07-26 08:31:49 +02:00
Aayush Atharva
7110997072
Bump Brotli4j to 1.5.0 (#11509)
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.

See https://github.com/hyperxpro/Brotli4j/pull/27

Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.

Result:
Up-to-date Broti4j library.
2021-07-26 08:29:07 +02:00
Norman Maurer
0cb25f9fdd
Be able to build on JDK17 (#11500)
Motivation:

As the release of JDK17 is getting closer and there are ea builds already we should ensure we can actually build netty with it.

Modifications:

- Add profile for JDK17
- Remove test-code that would fail with JDK17 due the changes in 4f4d0f5366.

Result:

Be able to build and run testsuite with JDK17
2021-07-23 14:14:30 +02:00
Norman Maurer
a1c3e3670d
Update japicmp plugin to support Java16+ (#11507)
Motivation:

japicmp did fix some issues related to Java16+. Let's update so we can also enable it in later java versions

Modifications:

Update to 0.15.3

Result:

Be able to use japicmp with all java versions
2021-07-23 11:51:15 +02:00
Norman Maurer
b14dfbb74c
FastLzFrameDecoder should use allocator to allocate output buffer (#11499)
Motivation:

FastLzFrameDecoder currently not use the allocator to alocate the output buffer. This means that if you use the PooledByteBufAllocator you still can't make use of the pooling. Beside this the decoder also does an uncessary memory copy when no compression is used.

Modifications:

- Allocate the output buffer via the allocator
- Don't allocate and copy if we handle an uncompressed chunk
- Make use of ByteBufChecksum for a few optimizations when running on a recent JDK

Result:

Less allocations when using FastLzFrameDecoder
2021-07-21 22:09:12 +02:00
skyguard1
3859805126
Add StringDecoder test (#11496)
Motivation:

There is no test case of `StringDecoder` here

Modification:

Need to add `StringDecoder` test case

Result:

Added test case of `StringDecoder`

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-21 14:31:14 +02:00
Norman Maurer
26cdfe2cde
Add PromiseNotifier static method which takes care of cancel propagation (#11494)
Motivation:

At the moment we not correctly propagate cancellation in some case when we use the PromiseNotifier.

Modifications:

- Add PromiseNotifier static method which takes care of cancellation
- Add unit test
- Deprecate ChannelPromiseNotifier

Result:

Correctly propagate cancellation of operation

Co-authored-by: Nitesh Kant <nitesh_kant@apple.com>
2021-07-21 13:37:32 +02:00
Mads Johannessen
3a41a97b0e
Support large or variable chunk sizes (#11469)
Motivation:

Chunks are splitted up into even smaller chunks when the underlying
buffer's readable bytes are less than the chunk size.

The underlying buffer can be smaller than a chunk size if:

- The chunk size is larger than the maximum plaintext chunk allowed by the TLS RFC,
  see: io.netty.handler.ssl.SslHandler.MAX_PLAINTEXT_LENGTH.

- The chunk sizes are variable in size,
  which may cause Netty guess a buffer size that is smaller than a chunk size.

Modification:

Create a variable in HttpObjectDecoder: ByteBuf chunkedContent

- Initialize chunkedContent in READ_CHUNK_SIZE with chunkSize as buffer size.

- In READ_CHUNKED_CONTENT write bytes into chunkedContent

  - If the remaining chunk size is not 0 and toRead ==maxChunkSize,
    create a chunk using the chunkedContent and add it to the output messages
    before re-initializing chunkedContent with the remaining chunkSize as buffer size.

  - If the remaining chunk size is not 0 and toRead != maxChunkSize,
    return without adding any output messages.

  - If the remaining chunk size is 0,
    create a chunk using the chunkedContent and add it to the output messages;
    set chunkedContent = null and fall-through.

Result:

Support chunk sizes higher than the underlying buffer's readable bytes.

Co-authored-by: Nitesh Kant <nitesh_kant@apple.com>
Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-20 15:21:26 +02:00
Aayush Atharva
1ce76e7e99
Update BouncyCastle Libraries (#11495)
Motivation:
New versions of Bouncy Castle libraries are out and we should upgrade to them.

Modification:
Upgraded all Bouncy Castle libraries to the latest version.

Result:
The latest versions of Bouncy Castle libraries.
2021-07-20 15:09:58 +02:00
Heng Zhang
b3e0e09a82
Distribue weight to children when closing stream (#11490)
Motivation:

As suggested in [section 5.3.4 in http2 spec](https://datatracker.ietf.org/doc/html/rfc7540#section-5.3.4):

> When a stream is removed from the dependency tree, its dependencies can be moved to become dependent on the parent of the closed stream. The weights of new dependencies are recalculated by distributing the weight of the dependency of the closed stream proportionally based on the weights of its dependencies.

For example, we have stream A and B depend on connection stream with default weights (16), and stream C depends on A with max weight (256). When stream A was closed, we move stream C to become dependent on the connection stream, then we should distribute the weight of stream A to its children (only stream C), so the new weight of stream C will be 16. If we keep the weight of stream C unchanged, it will get more resource than stream B

Modification:
- distribute weight to its children when closing a stream
- add a unit test for the case above and fix other related unit tests

Result:

More spec-compliant and more appropriate stream reprioritization

Co-authored-by: Heng Zhang <zhangheng@imo.im>
2021-07-20 14:17:56 +02:00
Netty Project Bot
2a184ab932 [maven-release-plugin] prepare for next development iteration 2021-07-16 06:23:23 +00:00
Netty Project Bot
e43d0d99f1 [maven-release-plugin] prepare release netty-4.1.66.Final 2021-07-16 06:23:17 +00:00
Chris Vest
9e8b8ff53a
Fix a bug where SslHandler clients would not process Server Hello messages in a timely manner (#11472)
Motivation:
The TLS handshake must be able to finish on its own, without being driven by outside read calls.
This is currently not the case when TCP FastOpen is enabled.
Reads must be permitted and marked as pending, even when a channel is not active.

This is important because, with TCP FastOpen, the handshake processing of a TLS connection will start
before the connection has been established -- before the process of connecting has even been started.

The SslHandler on the client side will add the Client Hello message to the ChannelOutboundBuffer, then
issue a `ctx.read` call for the anticipated Server Hello response, and then flush the Client Hello
message which, in the case of TCP FastOpen, will cause the TCP connection to be established.

In this transaction, it is important that the `ctx.read` call is not ignored since, if auto-read is
turned off, this could delay or even prevent the Server Hello message from being processed, causing
the server-side handshake to time out.

Modification:
Attach a listener to the SslHandler.handshakeFuture in the EchoClient, that will call ctx.read.

Result:
The SocketSslEchoTest now tests that the SslHandler can finish handshakes on its own, without being driven by 3rd party ctx.read calls.
The various channel implementations have been updated to comply with this behaviour.
2021-07-15 09:02:03 +02:00
Norman Maurer
55957d3e75
SelfSignedCertificate should try BouncyCastle first (#11487)
Motivation:

In JDK version >= 9 the access to sun.* is not permitted anymore by default. Because of this we should better first try the BouncyCastle based implementation before falling back to the JDK based version.

Modifications:

Switch ordering of usage of BouncyCastle vs JDK internals.

Result:

Less surprising errors when using SelfSignedCertificate in Java >9
2021-07-15 08:59:20 +02:00
Norman Maurer
e236e99006
Migrate the rest of the ssl package to junit5 (#11483)
Motivation:

We should use junit5 everywhere.

Modifications:

- Refactor rest of tests to use junit5

Result:

Part of https://github.com/netty/netty/issues/10757
2021-07-15 08:50:24 +02:00
Norman Maurer
2914134522
Disable flaky test for now (#11488)
Motivation:

JdkOpenSslEngineInteroptTest.testMutualAuthSameCerts() is flaky on the CI and so fails the PR build quite often.
Let's disable it for now until we were able to reproduce it locally and fix it.

Modifications:

Disable flaky test

Result:

More stable CI builds
2021-07-14 16:42:34 +02:00
Nitesh Kant
8337e3a973
Improve name matching in DNS answers (#11474)
__Motivation__

Upon receiving a DNS answer, we match whether the name in the question matches the name in the record. Some DNS servers we have encountered append a search domain to the record name which fails this match. eg: for question name `netty` and search domains `io` and `com`, we will do 2 queries: `netty.io.` and `netty.com.`, if the answer for `netty.io` contains `netty.com` then we ignore this record.

__Modification__

If the name in the record does not match the name in the question, append configured search domains to the question name to see if it matches the record name.

__Result__

Records names with appended search domains are still returned as valid answers.
2021-07-14 14:11:22 +02:00
Norman Maurer
876624a22b
SelfSignedCertificate should work in http2 tests (#11486)
Motivation:

We need to add `--add-exports java.base/sun.security.x509=ALL-UNNAMED` when running the tests for codec-http2 as some of the tests use SelfSignedCertificate.

Modifications:

- Add `--add-exports java.base/sun.security.x509=ALL-UNNAMED` when running the tests for codec-http2
- Ensure we export correct when running with JDK12, 13, 14 and 15 as well

Result:

No more tests failure due not be able to access classes
2021-07-14 07:46:17 +02:00
Aayush Atharva
3b8e7c3e60
Use Static Constants as Sample Data (#11485)
Motivation:
#11468 was merged but didn't fix tests completely. There is a fight between `LF` and `CRLF`. So to eliminate this, we should just get rid of them.

Modification:
Use a small sample dataset without `LF` and `CRLF`.

Result:
Simple and passing test.
2021-07-14 07:45:21 +02:00
Norman Maurer
043e9e309e
Remove rest of junit4 usage (#11484)
Motivation:

We did migrate all these modules to junit5 before but missed a few usages of junit4

Modifications:

Replace all junit4 imports by junit5 apis

Result:

Part of  https://github.com/netty/netty/issues/10757
2021-07-13 20:59:57 +02:00
Aayush Atharva
7ec198ddf9
Fix JavaDoc and Run Brotli tests on Windows (#11468)
Motivation:
JavaDoc of StandardCompressionOptions should point towards public methods. Also, Brotli tests were failing on Windows.

Modification:
Fixed JavaDoc and enabled Brotli tests on Windows.

Result:
Better JavaDoc and Brotli tests will run on Windows

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-13 16:24:58 +02:00
Norman Maurer
98c446c493 Revert "Only do a partial build for a few jobs when verify PRs (#11464)"
This reverts commit e5adc1131a.
2021-07-13 15:32:42 +02:00
skyguard1
df22356a3a
Refactor HttpContentCompressor using CompressionEncoderFactory (#11480)
Motivation:

The `HttpContentCompressor.beginEncode()` method has too many if else, so consider refactoring

Modification:

Create the corresponding `CompressionEncoderFactory` according to the compression algorithm, remove the if else

Result:

The code of `HttpContentCompressor` is cleaner than the previous implementation

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-13 12:25:30 +02:00
Aayush Atharva
6085cd7ab6
Update HttpContentCompressor to pass correct message to ObjectUtil (#11482)
Motivation:
In #11256, We introduced `Iterable` as a parameter but later in review, it was removed. But we forgot to change `compressionOptionsIterable` to just `compressionOptions`.

Modification:
Changed `compressionOptionsIterable` to `compressionOptions`.

Result:
Correct ObjectUtil message
2021-07-13 08:11:01 +02:00
Adrian Antkowiak
8cbdefae23
HAProxyMessage.sourceAddress() can be null (#11475)
Motivation:

If `send-proxy-v2` is used `sourceAddress()` can be `null`.

Modification:

Update docs to reflect this.

Result:

Docs are more correct.
2021-07-12 11:55:39 +02:00
skyguard1
6ce36f1909
Add zstd http content compression support (#11470)
Motivation:

netty needs to support zstd content-encoding http content compression

Modification:

Add ZstdOptions, and modify HttpContentCompressor and CompressorHttp2ConnectionEncoder to support zstd compression

Result:

netty supports zstd http content compression

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-12 08:49:04 +02:00
Violeta Georgieva
3f9a5f50be
Add support for Unix domain datagram sockets when using native epoll/kqueue transport (#11423)
Motivation:

There are use cases when Unix domain datagram sockets are needed for communication. This PR adds such support for Epoll/KQueue.

Modification:

- Expose Channel, Config and Packet interfaces/classes for Unix domain datagram sockets. All interfaces/classes are in `transport-native-unix-common` module in order to be available for KQueue and Epoll implementations
- Add JNI code for Unix domain datagram sockets
- Refactor `DatagramUnicastTest` so that it can be used for testing also Unix domain datagram sockets
- Add Unix domain datagram sockets implementation for KQueue transport
- Add Unix domain datagram sockets implementation for Epoll transport

Result:

Fixes #6737
2021-07-09 09:45:43 +02:00
Norman Maurer
e5adc1131a
Only do a partial build for a few jobs when verify PRs (#11464)
Motivation:

Currently the PR verification takes a lot of time. We should try to only run builds for the affected modules in most cases to speed up the build.

Modifications:

Adjust docker files and workflow to only run build for affected modules for a lot of jobs while still run full builds for windows + java11 with BoringSSL

Result:

Hopefully quicker feedback loop when verify PRs
2021-07-09 06:47:06 +02:00
Ikko Ashimine
d1ef28a22f
Fix typo in ReferenceCountedOpenSslEngine (#11467)
Motivation:

There should be no typos in comments

Modifications:
```
alway -> always
```

Result:

Fixed typo.
2021-07-08 16:29:58 +02:00
Norman Maurer
40fb6026ef
Introduce OpenSslAsyncPrivateKeyMethod which allows to asynchronously sign / decrypt the private key (#11390) (#11460)
Motivation:

At the moment we only support signing / decrypting the private key in a synchronous fashion. This is quite limited as we may want to do a network call to do so on a remote system for example.

Modifications:

- Update to latest netty-tcnative which supports running tasks in an asynchronous fashion.
- Add OpenSslAsyncPrivateKeyMethod interface
- Adjust SslHandler to be able to handle asynchronous task execution
- Adjust unit tests to test that asynchronous task execution works in all cases

Result:

Be able to asynchronous do key signing operations
2021-07-08 16:19:22 +02:00
Aayush Atharva
fef761d03e
Introduce BrotliEncoder (#11256)
Motivation:
Currently, Netty only has BrotliDecoder which can decode Brotli encoded data. However, BrotliEncoder is missing which will encode normal data to Brotli encoded data.

Modification:
Added BrotliEncoder and CompressionOption

Result:
Fixes #6899.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-07-08 11:51:27 +02:00
skyguard1
7825fa8d7a
Add zstd http header value (#11463)
Motivation:

ZSTD has a wide range of uses on the Internet, so should consider adding `application/zstd` HTTP media-type and `zstd` content-encoding, see  https://tools.ietf.org/html/rfc8478

Modification:

Add `application/zstd` HTTP media-type and `zstd` content-encoding

Result:

netty provides `application/zstd` HTTP media-type and `zstd content-encoding` as http headers

Signed-off-by: xingrufei <xingrufei@sogou-inc.com>
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
2021-07-08 11:44:54 +02:00
Norman Maurer
3e350993b7
Add script which can be used to build affected modules only (#11461)
Motivation:

At the moment we always build all modules. This script can be used to only build affected modules for a given change

Modifications:

Add script that will only build modules that are affected by a change

Result:

More targeted build
2021-07-08 10:18:16 +02:00
Norman Maurer
7e39b96402
Only run one SSL task per delegation (#11462)
Motivation:

We should only run one SSL task per delegation to allow more SSLEngines to make progress in a timely manner

Modifications:

- Only run one task per delegation to the executor
- Only create new SSL task if really needed
- Only schedule if not on the EventExecutor thread

Result:

More fair usage of resources and less allocations
2021-07-08 07:56:15 +02:00