Commit Graph

10247 Commits

Author SHA1 Message Date
Aayush Atharva
94485d7e13 Remove unnecessary this. (#11035)
Motivation:
There are unnecessary `this` keyword usages in `HttpStaticFileServerHandler`.

Modification:
Removed unnecessary `this` keywords.

Result:
Clean code
2021-02-26 12:13:59 +01:00
Chris Vest
f3134080ff Fix bug in Recycler with racing calls to recycle (#11037)
Motivation:
It is possible for two separate threads to race on recycling an object.
If this happens, the object might be added to a WeakOrderQueue when it shouldn't be.
The end result of this is that an object could be acquired multiple times, without a recycle in between.
Effectively, it ends up in circulation twice.

Modification:
We fix this by making the update to the lastRecycledId field of the handle, an atomic state transition.
Only the thread that "wins" the race and succeeds in their state transition will be allowed to recycle the object.
The others will bail out on their recycling.
We use weakCompareAndSet because we only need the atomicity guarantee, and the program order within each thread is sufficient.
Also, spurious failures just means we won't recycle that particular object, which is fine.

Result:
Objects no longer risk circulating twice due to a recycle race.

This fixes #10986
2021-02-26 10:03:49 +01:00
Idel Pivnitskiy
a1172bf9c6 Less noisy logging in DnsServerAddressStreamProviders (#11031)
Motivation:

It is not uncommon to run Netty on OS X without the specific
`MacOSDnsServerAddressStreamProvider`. The current log message is too
verbose because it prints a full stack trace on the console while a
simple logging message would have been enough.

Modifications:

- Print a `WARN` message when `MacOSDnsServerAddressStreamProvider`
class is not found;
- Print a `ERROR` message with a stack trace when the class was found
but could not be loaded due to some other reasons;

Result:

Less noise in logs.
2021-02-23 11:17:47 +01:00
Chris Vest
95e4d6f3fe Fix hash collision handling in DefaultHeaders iterator remove (#11028)
Motivation:
If two different headers end up in the same hash bucket, and you are iterating the header that is not the first in the bucket, and you use the iterator to remove the first element returned from the iterator, then you would get a NullPointerException.

Modification:
Change the DefaultHeaders iterator remove method, to re-iterate the hash bucket and unlink the entry once found, if we don't have any existing iteration starting point.

Also made DefaultHeaders.remove0 package private to avoid a synthetic method indirection.

Result:
Removing from iterators from DefaultHeaders is now robust towards hash collisions.
2021-02-19 13:31:55 +01:00
Norman Maurer
7f2925350a Propagate SSLException to the Http2StreamChannels (#11023)
Motivation:

When TLSv1.3 is used (or TLS_FALSE_START) together with mTLS the handshake is considered successful before the server actually did verify the key material that was provided by the client. If the verification fails we currently will just close the stream without any extra information which makes it very hard to debug on the client side.

Modifications:

- Propagate SSLExceptions to the active streams
- Add unit test

Result:

Better visibility into why a stream was closed
2021-02-19 08:41:55 +01:00
吴迪
5e0617e49e Fix incorrect comment in code (#11029)
Motivation:
Comment on method is outdated / incorrect.

Modification:
Adjust comment

Result:
Correct docs
2021-02-19 08:07:43 +01:00
wangyuwei
dbee8c9b89 Avoid letting ipv6 addresses join ipv4 groups (#11015)
Motivation:

#10995 

when `io.netty.channel.unix.Socket` is ipv6 and join a multicast group with ipv4 address will cause `io.netty.channel.ChannelException: setsockopt() failed: Invalid argument` (at least in `Linux centos.dev 4.18.0-240.10.1.el8_3.x86_64 #1 SMP Mon Jan 18 17:05:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux`)


Modification:

check if target group address is ipv6 before call  `io.netty.channel.epoll.LinuxSocket#joinGroup(java.net.InetAddress, java.net.NetworkInterface, java.net.InetAddress)` 

I'm not sure if this modification is currect, but i checked source code of java NIO

```
Java_sun_nio_ch_Net_canJoin6WithIPv4Group0(JNIEnv* env, jclass cl)
{
#if defined(__APPLE__)
    /* IPV6_ADD_MEMBERSHIP can be used to join IPv4 multicast groups */
    return JNI_TRUE;
#else
    /* IPV6_ADD_MEMBERSHIP cannot be used to join IPv4 multicast groups */
    return JNI_FALSE;
#endif
}
```
seems ipv6 address can't join ipv4 group except osx

Result:

test on `Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux` exception  ` setsockopt() failed: Invalid argument` has fixed

Fixes #10995
2021-02-16 14:22:02 +01:00
Idel Pivnitskiy
b29df0f87a Simplify flushAtEnd flag computation in SslHandler#handlerAdded (#11025)
Motivation:

The `!fastOpen` part of `active || !fastOpen` is always false.

Modification:

- Remove `!fastOpen` and keep only `active` as a `flushAtEnd` flag for
`startHandshakeProcessing`;
- Update comment;

Result:

Simplified `flushAtEnd` flag computation in `SslHandler#handlerAdded`.
2021-02-16 14:05:26 +01:00
Chris Vest
d60b1651fc TCP Fast Open for clients (#11006)
Support TCP Fast Open for clients and make SslHandler take advantage

Motivation:
- TCP Fast Open allow us to send a small amount of data along side the initial SYN packet when establishing a TCP connection.
- The TLS Client Hello packet is small enough to fit in there, and is also idempotent (another requirement for using TCP Fast Open), so if we can save a round-trip when establishing TLS connections when using TFO.

Modification:
- Add support for client-side TCP Fast Open for Epoll, and also lowers the Linux kernel version requirements to 3.6.
- When adding the SslHandler to a pipeline, if TCP Fast Open is enabled for the channel (and the channel is not already active) then start the handshake early by writing it to the outbound buffer.
- An important detail to note here, is that the outbound buffer is not flushed at this point, like it would for normal handshakes. The flushing happens later as part of establishing the TCP connection.

Result:
- It is now possible for clients (on epoll) to open connections with TCP Fast Open.
- The SslHandler automatically detects when this is the case, and now send its Client Hello message as part of the initial data in the TCP Fast Open flow when available, saving a round-trip when establishing TLS connections.

Co-authored-by: Colin Godsey <crgodsey@gmail.com>
2021-02-15 14:29:03 +01:00
赵延
1d1087243f
Unify the exception for bind operations in AbstractBootstrap (#11018)
Motivation:

How we used exceptions was not consistent.

Modifications:

Use the same type of exception in the bind methods.

Result:

Fixes #11014
2021-02-15 08:59:10 +01:00
skyguard1
899e5a6850 Upgrade graalvm version to the latest version (#11012)
Motivation:
The current netty's graalvm dependency version is too low, so you need to upgrade the plugin

Modification:
Upgrade Graalvm version to the latest version, please review this pr, thank you

Result:

Use up-to-date version.

Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
2021-02-11 13:30:00 +01:00
Norman Maurer
b43b67553e Disable flaky test (#11017)
Motivation:

The testGlobalWriteThrottle is flaky and failed our build multiple times now. Lets disable it for now until we had time to investigate

Modifications:

Disable flaky test

Result:

Less failures during build
2021-02-11 13:27:38 +01:00
Violeta Georgieva
b1ce20c080 Allow blocking calls when parsing etcResolver/hosts files (#11009)
Motivation:

When etcResolver/hosts files are parsed, FileInputStream.read(...) is internally called by

- UnixResolverDnsServerAddressStreamProvider#parseEtcResolverSearchDomains
- UnixResolverDnsServerAddressStreamProvider#parseEtcResolverOptions
- HostsFileParser#parse

This will cause the error below when BlockHound is enabled

reactor.blockhound.BlockingOperationError: Blocking call! java.io.FileInputStream#readBytes
     	at java.io.FileInputStream.readBytes(FileInputStream.java)
     	at java.io.FileInputStream.read(FileInputStream.java:255)

Modifications:

- Add whitelist entries to BlockHound configuration
- Fix typos in UnixResolverDnsServerAddressStreamProvider
- Add tests

Result:

Fixes #11004
2021-02-11 11:05:14 +01:00
strugcoder
c1b3ffafcf Simplity some code (#11000)
Motivation:

There was some code that could be simplified.

Modification:

Simplify code.

Result:

Code cleanup
2021-02-11 09:06:16 +01:00
Chris Vest
4177e2c445 Revert test debugging flags and update surefire/failsafe (#11016)
* Revert "Add a profile for debugging tests that run from Maven (#11011)"

This reverts commit 83895f0f

The same functionality is already natively available in surefire, by adding the `-Dmaven.surefire.debug` flag to Maven.

* Update surefire/failsafe version

These new versions copes better when our tests prints to STDOUT, and disturbs the progress processing that these plugins do.
2021-02-10 15:37:06 +01:00
Norman Maurer
42a2c9c831 Enable stateless resumption for TLSv1.3 by default when using OpenSSL / BoringSSL (#10997)
Motivation:

At the moment we always set SSL_OP_NO_TICKET when building our context. The problem with this is that this also disables resumption for TLSv1.3 in BoringSSL as it only supports stateless resumption for TLSv1.3 which uses tickets.
We should better clear this option when TLSv1.3 is enabled to be able to resume sessions. This is also inline with the OpenJDK which enables this for TLSv1.3 by default as well.

Modifications:

Check for enabled protocols and if TLSv1.3 is set clear SSL_OP_NO_TICKET.

Result:

Be able to resume sessions for TLSv1.3 when using BoringSSL.
2021-02-08 20:56:42 +01:00
Norman Maurer
9c2de76add Use Files.createTempFile(...) to ensure the file is created with proper permissions
Motivation:

File.createTempFile(String, String)` will create a temporary file in the system temporary directory if the 'java.io.tmpdir'. The permissions on that file utilize the umask. In a majority of cases, this means that the file that java creates has the permissions: `-rw-r--r--`, thus, any other local user on that system can read the contents of that file.
This can be a security concern if any sensitive data is stored in this file.

This was reported by Jonathan Leitschuh <jonathan.leitschuh@gmail.com> as a security problem.

Modifications:

Use Files.createTempFile(...) which will use safe-defaults when running on java 7 and later. If running on java 6 there isnt much we can do, which is fair enough as java 6 shouldnt be considered "safe" anyway.

Result:

Create temporary files with sane permissions by default.
2021-02-08 18:17:31 +01:00
Norman Maurer
4d3bd06212 Update to latest os-maven-plugin (#11003)
Motivation:

A new version of the os-maven-plugin was released

Modifications:

Update to latest version

Result:

Use latest version
2021-02-08 08:43:44 +01:00
Scott Mitchell
4a5d7a5a17 ReadOnlyByteBuf writable bytes
Motivation:
ReadOnlyByteBuf and ReadOnlyByteBuffer are not writable, but their writableBytes
related methods return non-zero values. This is inconsistent with the behavior
of these buffer types.

Modifications:
- ReadOnlyByteBuf and ReadOnlyByteBuffer writableBytes related methods should
  return 0

Result:
More correct ReadOnlyByteBuf and ReadOnlyByteBuffer behavior with respect to
writability.
2021-02-05 20:26:28 +01:00
Francesco Nigro
517da28740 DecodeHexBenchmark is too branch-predictor friendly (#9942)
Motivation:

DecodeHexBenchmark needs to be less branch-predictor friendly
to mimic the "real" behaviour while decoding

Modifications:

DecodeHexBenchmark uses a larger sets of inputs, picking them at
random on each iteration and the benchmarked method is made !inlineable

Result:

DecodeHexBenchmark is more trusty while showing the performance
difference between different decoding methods
2021-02-05 15:28:25 +01:00
Zxy
87392634d2 Fix memory release failure when "maxNumElems == 1" of PoolSubpage (#10988)
Motivation:

when customer need large of 'byteBuf.capacity' in [7168, 8192], the size of 'chunk.subpages' may be inflated when large of byteBuf be released, not consistent with other 'byteBuf.capacity'

Modification:

when maxNumElems == 1 need consider remove from pool

Result:

Fixes #10896. 

Co-authored-by: zxingy <zxingy@servyou.com.cn>
2021-02-05 14:54:44 +01:00
Norman Maurer
d7fb0f5c0b Revert HttpPostMultipartRequestDecoder and HttpPostStandardRequestDecoder to e5951d46fc (#10989)
Motivation:

The changes introduced in 1c230405fd did cause various issues while the fix itself is not considered critical. For now it is considered the best to just rollback and investigate more.

Modifications:

- Revert changes done in 1c230405fd (and later) for
the post decoders.
- Ensure we give memory back to the system as soon as possible in a safe manner

Result:

Fixes https://github.com/netty/netty/issues/10973
2021-02-03 20:41:28 +01:00
Norman Maurer
f25b12077a Clarify who is responsible closing the InputStream once SslContext w… (#10983)
Motivation:

It was not 100% clear who is responsible calling close() on the InputStream.

Modifications:

Clarify javadocs.

Result:

Related to https://github.com/netty/netty/issues/10974

Co-authored-by: Chris Vest <christianvest_hansen@apple.com>
2021-02-01 11:50:55 +01:00
Norman Maurer
980ea9c351 Revert "Enable SSL_MODE_ENABLE_FALSE_START if jdkCompatibilityMode is false (#10407)" (#10980)
Motivation:

TLS_FALSE_START slightly changes the "flow" during handshake which may cause suprises for the end-user. We should better disable it by default again and later add a way to enable it for the user.

Modification:

This reverts commit 514d349e1f.

Result:

Restore "old flow" during TLS handshakes.
2021-01-29 19:56:12 +01:00
Bennett Lynch
5a725edf4a
[master] Allow and skip null handlers when adding a vararg list of handlers (#10776)
Allow and skip null handlers when adding a vararg list of handlers

Motivation

Allowing null handlers allows for more convenient idioms in
conditionally adding handlers, e.g.,

ch.pipeline().addLast(
        new FooHandler(),
        condition ? new BarHandler() : null,
        new BazHandler()
);

Modifications

* Change addFirst(..) and addLast(..) to skip null handlers, rather than
break or short-circuit.
* Add new unit tests.

Result

* Makes addFirst(..) and addLast(..) behavior more consistent
* Resolves https://github.com/netty/netty/issues/10728
2021-01-29 11:30:45 +01:00
Norman Maurer
3e5a08cefb Fix possible SEGV when using native dns resolver on macos (#10971)
Motivation:

We should add some more NULL checks to ensure we not SEGV in some cases

Modifications:

Add more NULL checks

Result:

More robust implementation
2021-01-28 15:15:36 +01:00
Norman Maurer
99fc0e486d Correctly filter out TLSv1.3 ciphers if TLSv1.3 is not enabled (#10919)
Motivation:

We didnt correctly filter out TLSv1.3 ciphers when TLSv1.3 is not enabled.

Modifications:

- Filter out ciphers that are not supported due the selected TLS version
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/10911

Co-authored-by: Bryce Anderson <banderson@twitter.com>
2021-01-28 11:01:57 +01:00
Norman Maurer
a3d100a7e8 Ignore SocketShutdownOutputBySelfTest.testWriteAfterShutdownOutputNoWritabilityChange (#10970)
Motivation:

The testWriteAfterShutdownOutputNoWritabilityChange() failed a few times on the CI randomly. Let's skip it for now while we investigate and see if there is anything we can do to make the test less flaky on the CI.

Modifications:

Add @Ignore on the testWriteAfterShutdownOutputNoWritabilityChange method

Result:

Less flaky CI
2021-01-28 07:30:36 +01:00
Norman Maurer
fb9412b0ad Update dependency declaration in all pom.xml (#10967)
Motivation:

We did have the architecture hardcoded in the dependency which is not correct as this will let the build fail on Applie Silicion (m1). Also we did miss some dependencies on other BSDs

Modifications:

- Fix classifier
- Add missing dependencies

Result:

Be able to build on Apple Silicon (m1)
2021-01-27 10:25:36 +01:00
Norman Maurer
b10e514615 Ensure we create a copy of the attributes and options when bootstrap a channel (#10965)
Motivation:

We need to ensure we copy the attributes and options when bootstrap the channel as otherwise we may change the underlying Entry.
This is similar to what was reported in https://github.com/netty/netty-incubator-codec-quic/issues/152.

Modifications:

- Do a copy and re-use methods
- Add unit tests

Result:

Don't affect attributes / options of channels that are already bootstrapped
2021-01-26 19:33:25 +01:00
Carl Mastrangelo
c68af5c8f4 Don't use Fixed ports for UDP test (#10961)
Motivation:
If the given port is already bound, the PcapWriteHandlerTest will sometimes fail.

Modification:
Use a dynamic port using `0`, which is more reliable

Result:
Less Flaky
2021-01-26 08:31:22 +01:00
Stephane Landelle
6c041c3104 Merge WebSocket extensions, close #10792 (#10956)
Motivation:

We currently append extensions to the user defined "sec-websocket-extensions" headers. This can cause duplicated entries.

Modifications:

* Replace existing `WebSocketExtensionUtil#appendExtension` private helper with a new `computeMergeExtensionsHeaderValue`. User defined parameters have higher precedence.
* Add tests (existing method wasn't tested)
* Reuse code for both client and server side (code was duplicated).

Result:

No more duplicated entries when user defined extensions overlap with the ones Netty generated.
2021-01-22 08:50:25 +01:00
Duke Bartholomew
8eb8ad8e9d Allow to use int while build MqttMessageIdVariableHeader (#10930)
Motivation:

Exception occurs in the MQTT message builder class (`io.netty.handler.codec.mqtt.MqttMessageBuilders`) when trying to create a message with packetId > 32767

Modification:

-add method that takes int
- deprecate old methods that take short. 

Result:


Fixes #10929 .
2021-01-21 14:41:25 +01:00
Norman Maurer
56d2635733 Override ALPN methods on ReferenceCountedOpenSslEngine (#10954)
Motivation:

We should override the get*ApplicationProtocol() methods in ReferenceCountedOpenSslEngine to make it easier for users to obtain the selected application protocol

Modifications:

Add missing overrides

Result:

Easier for the user to get the selected application protocol (if any)
2021-01-21 14:07:44 +01:00
Norman Maurer
6ae8cd6e44 Mark some methods as protected to make it easier to write own SslContext implementations (#10953)
Motivation:

We should expose some methods as protected to make it easier to write custom SslContext implementations.
This will be reused by the code for https://github.com/netty/netty-incubator-codec-quic/issues/97

Modifications:

- Add protected to some static methods which are useful for sub-classes
- Remove some unused methods
- Move *Wrapper classes to util package and make these public

Result:

Easier to write custom SslContext implementations
2021-01-21 14:07:28 +01:00
Norman Maurer
b19b8c39cb Ignore priority frames for non existing streams and so prevent a NPE (#10943)
Motivation:

https://github.com/netty/netty/pull/10765 added support for push promise and priority frames when using the Http2FrameCodec. Unfortunally it didnt correctly guard against the possibility to receive a priority frame for an non-existing stream, which resulted in a NPE

Modifications:

- Ignore priority frame for non existing stream
- Correctly implement equals / hashcode for DefaultHttp2PriorityFrame
- Add unit tests

Result:

Fixes https://github.com/netty/netty/issues/10941
2021-01-18 14:14:00 +01:00
Norman Maurer
35ac770f3d Correctly handle fragmentation in JdkZlibDecoder (#10948)
Motivation:

We had multiple bugs in JdkZlibDecoder which could lead to decoding errors when the data was received in a fragmentated manner.

Modifications:

- Correctly handle skipping of comments
- Correctly handle footer / header decoding
- Add unit test that verifies the correct handling of fragmentation

Result:

Fixes https://github.com/netty/netty/issues/10875
2021-01-18 14:03:38 +01:00
Norman Maurer
2aaa468a22 Make native loading logging less confusing (#10944)
Motivation:

We produced a lot of noise during loading native libraries as we always included the stacktrace if we could not load by one mechanism. We should better just not include the stacktrace in the debugging logging if one mechanism fails. We will log all the stacks anyway when all of the mechanisms fail.

Modifications:

Make logging less aggressive

Result:

Less confusing behaviour for the end-user
2021-01-16 10:15:12 +01:00
Norman Maurer
f6c1c0ea9c Use GracefulShutdown when stream space is exhausted (#10946)
Motivation:

We should use GracefulShutdown when we try to create a stream and fail it because the stream space is exhausted as we may still want to process the active streams.

Modifications:

- Use graceful shutdown
- Add unit test

Result:

More graceful handling of stream creation failure due stream space exhaustation
2021-01-16 09:55:53 +01:00
Francesco Nigro
5337d3eeb4 Implement SWAR indexOf byte search (#10737)
Motivation:

Faster indexOf

Modification:

Create generic SWAR indexOf that any ByteBuf implementation can use

Result:

Fixes #10731
2021-01-15 15:09:50 +01:00
Chris Vest
aed20979e6 Update the javadocs on FastThreadLocal (#10942)
Motivation: The writing was unclear.
Modification: Clarified the documentation.
Result: No more ambiguity about what FTL.remove() does.

Fixes #10914
2021-01-15 12:05:09 +01:00
Norman Maurer
ede5eecf35 Fix compile error introduced by dd89d1b6bf 2021-01-14 18:56:32 +01:00
Norman Maurer
b21a08ea03 Fix compile error introduced by bad cherry-pick in a7e8a33833 2021-01-14 18:44:12 +01:00
Violeta Georgieva
dd89d1b6bf Allow blocking calls in UnixResolverDnsServerAddressStreamProvider#parse (#10935)
Motivation:

Internally UnixResolverDnsServerAddressStreamProvider#parse calls FileInputStream.read(...)
when parsing the etcResolverFiles.
This will cause the error below when BlockHound is enabled
reactor.blockhound.BlockingOperationError: Blocking call! java.io.FileInputStream#readBytes
     	at java.io.FileInputStream.readBytes(FileInputStream.java)
     	at java.io.FileInputStream.read(FileInputStream.java:255)

Modifications:

- Add whitelist entry to BlockHound configuration
- Add test

Result:

Fixes #10925
2021-01-14 18:33:26 +01:00
Norman Maurer
a7e8a33833 Ensure native methods for unix-native-common are only registered once. (#10932)
Motiviation:

We need to ensure we only register the methods for unix-native-common once as otherwise it may have strange side-effects.

Modifications:

- Add extra method that should be called to signal that we need to register the methods. The registration will only happen once.
- Adjust code to make use of it.

Result:

No more problems due incorrect registration of these methods.
2021-01-14 17:53:40 +01:00
Norman Maurer
78646b3d7e Verify we can load native modules and add job that verifies on aarch64 as well (#10933)
Motivation:

As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions.

Modifications:

- Add new maven module which tests loading of native modules
- Add job that will also test loading on aarch64

Result:

Less likely to introduce regressions related to loading native code in the future
2021-01-14 17:15:29 +01:00
Norman Maurer
dc632e378f We need to ensure we always drain the error stack when a callback throws (#10920)
Motivation:

We need to ensure we always drain the error stack when a callback throws as otherwise we may pick up the error on a different SSL instance which uses the same thread.

Modifications:

- Correctly drain the error stack if native method throws
- Add a unit test which failed before the change

Result:

Always drain the error stack
2021-01-11 20:57:16 +01:00
Aki
e8f3e88526 Fix url in README.md (#10915)
Motivation:

We had a typo in the markdown

Modification:

Fix typo

Result:

Corect markdown
2021-01-11 07:50:43 +01:00
Norman Maurer
2aa33165cf Update README
Motivation:

We can add some status badge and also should clarify requirements.

Modifictations:

- Add status badge
- Clarify requirements

Result:

Cleanup docs
2021-01-11 07:50:35 +01:00
Aayush Atharva
c1095a0ac2 Update BouncyCastle Libraries (#10906)
Motivation:
New versions of `Bouncy Castle` libraries are out and we should upgrade to them.

Modification:
Upgraded all `Bouncy Castle` libraries to the latest version.

Result:
The latest versions of `Bouncy Castle` libraries.

Fixes #10905.
2021-01-01 19:33:03 +01:00