Commit Graph

604 Commits

Author SHA1 Message Date
Riley Park
bc7434aedb Migrate testsuite, transport-native-epoll, transport-native-kqueue, and transport-native-unix-common-tests tests to JUnit 5 (#11320)
Motivation:

JUnit 5 is more expressive, extensible, and composable in many ways, and it's better able to run tests in parallel.

Modifications:

Use JUnit5 in tests

Result:

Related to https://github.com/netty/netty/issues/10757
2021-05-27 15:55:58 +02:00
Netty Project Bot
02142a2510 [maven-release-plugin] prepare for next development iteration 2021-05-19 12:09:18 +00:00
Netty Project Bot
9e06241ddb [maven-release-plugin] prepare release netty-4.1.65.Final 2021-05-19 12:09:14 +00:00
Norman Maurer
6e866317d2
Ensure we can load the native library or fail the build (#11262)
Motivation:

We used assumeTrue(...) in some places before to detect if we could load the native library but this could lead to the sitation that we not notice if we break native loading.

Modifications:

Always fail if we cant load the native library

Result:

Ensure we not cause any regression in the native loading code in the future
2021-05-18 08:14:32 +02:00
Netty Project Bot
d5c81d9a3e [maven-release-plugin] prepare for next development iteration 2021-05-17 07:04:14 +00:00
Netty Project Bot
67bedf0efe [maven-release-plugin] prepare release netty-4.1.64.Final 2021-05-17 07:04:08 +00:00
Idel Pivnitskiy
3ecb32357c
Use PlatformDependent#normalizedOs() instead of reading os.name prop (#11239)
Motivation:

`PlatformDependent#normalizedOs()` already caches normalized variant of
the value of `os.name` system property. Instead of inconsistently
normalizing it in every case, use the utility method.

Modifications:

- `PlatformDependent`: `isWindows0()` and `isOsx0()` use `NORMALIZED_OS`;
- `PlatformDependent#normalizeOs(String)` define `darwin` as `osx`;
- `OpenSsl#loadTcNative()` does not require `equalsIgnoreCase` bcz `os`
is already normalized;
- Epoll and KQueue: `Native#loadNativeLibrary()` use `normalizedOs()`;
- Use consistent `Locale.US` for lower case conversion of `os.name`;
- `MacOSDnsServerAddressStreamProvider#loadNativeLibrary()` uses
`PlatformDependent.isOsx()`;

Result:

Consistent approach for `os.name` parsing.
2021-05-11 08:52:29 +02:00
Norman Maurer
6b3ec62e6e
Preload classes before calling native OnLoad function to prevent clas… (#11215)
Motivation:

It turns out it is quite easy to cause a classloader deadlock in more recent java updates if you cause classloading while you are in native code. Because of this we should just workaround this issue by pre-load all the classes that needs to be accessed in the OnLoad function.

Modifications:

- Preload all classes that would otherwise be loaded by native OnLoad functions.

Result:

Workaround for https://github.com/netty/netty/issues/11209 and https://bugs.openjdk.java.net/browse/JDK-8266310
2021-05-03 10:22:22 +02:00
Boris Unckel
c3416d8ad2
Utilize i.n.u.internal.ObjectUtil to assert Preconditions (transport*) (#11170) (#11181)
Motivation:

NullChecks resulting in a NullPointerException or IllegalArgumentException, numeric ranges (>0, >=0) checks, not empty strings/arrays checks must never be anonymous but with the parameter or variable name which is checked. They must be specific and should not be done with an "OR-Logic" (if a == null || b == null) throw new NullPointerEx.

Modifications:

* import static relevant checks
* Replace manual checks with ObjectUtil methods

Result:

All checks needed are done with ObjectUtil, some exception texts are improved.

Fixes #11170
2021-04-22 14:13:14 +02:00
Norman Maurer
e27831d767
Fix support for IP_RECVORIGDSTADDR when using native epoll transport (#11173)
Motivation:

While adding support for GRO (b05fdf3ff8) we broke support for IP_RECVORIGDSTADDR when using the native transport. Beside this we also didnt correctly handle IP_RECVORIGDSTADDR when recvmmsg was used.

Modifications:

- Fix support for IP_RECVORIGDSTADDR when using the native epoll transport for normal reads (recvmsg) but also for scattering reads (recvmmsg)
- Remove code from unix code-base as the support is linux specific and we not need the code there anymore

Result:

Fixes https://github.com/netty/netty/issues/11141
2021-04-21 16:04:12 +02:00
Chris Vest
df53de5b68
Less blocking in ChunkedStream (#11150)
Motivation:
We should avoid blocking in the event loop as much as possible.
The InputStream.read() is a blocking method, and we don't need to call it if available() returns a positive number.

Modification:
Bypass calling InputStream.read() if available() returns a positive number.

Result:
Fewer blocking calls in the event loop, in general, when ChunkedStream is used.
2021-04-12 13:40:14 +02:00
Netty Project Bot
6724786dcc [maven-release-plugin] prepare for next development iteration 2021-04-01 10:50:23 +00:00
Netty Project Bot
b78d8f2abd [maven-release-plugin] prepare release netty-4.1.63.Final 2021-04-01 10:50:18 +00:00
Netty Project Bot
e4dd6ee532 [maven-release-plugin] prepare for next development iteration 2021-03-30 17:22:25 +00:00
Netty Project Bot
573d1f4bf5 [maven-release-plugin] prepare release netty-4.1.62.Final 2021-03-30 17:22:18 +00:00
Netty Project Bot
7192ab0f0a [maven-release-plugin] prepare for next development iteration 2021-03-30 07:42:06 +00:00
Netty Project Bot
ccb3ff388f [maven-release-plugin] prepare release netty-4.1.61.Final 2021-03-30 07:42:01 +00:00
Norman Maurer
6c741932aa
Move SegmentedDatagramPacket to transport-native-unix-common (#11121)
Motivation:

As we can supported SegmentedDatagramPacket in multiple native
transports (like in epoll and io_uring) we should just move it to
unix-common so we can share code.

Modification:

- Move SegmentedDatagrampPacket to transport-native-unixu
- Mark the SegmentedDatagramPacket in epoll as deprecated
- Update code to use updated package.

Result:

Possibility of code re-use
2021-03-29 13:55:43 +02:00
Norman Maurer
b05fdf3ff8
Add support for UDP_GRO (#11120)
Motivation:

UDP_GRO can improve performance when reading UDP datagrams. This patch adds support for it.

See https://lwn.net/Articles/768995/

Modifications:

- Add recvmsg(...)
- Add support for UDP_GRO in recvmsg(...) and recvmmsg(...)
- Remove usage of recvfrom(...) and just always use recvmsg(...) or recvmmsg(...) to simplify things
- Refactor some code for sharing
- Add EpollChannelOption.UDP_GRO and the getter / setter in EpollDatagramConfig

Result:

UDP_GRO is supported when the underlying system supports it.
2021-03-29 08:51:08 +02:00
Norman Maurer
7ed840e62a
Use netty-jni-util 0.0.3.Final (#11115)
netty-jni-util 0.0.2.Final is incompatible with static linking. Before
the netty-jni-util dependency was introduced netty-tcnative supported
static linking via NETTY_BUILD_STATIC. netty-jni-util 0.0.3.Final adds
static linking compatibility.

Modifications:

Bump netty-jni-util to version 0.0.3.Final and update to its new API
which requires the caller to manage packagePrefix.

Result:

Using latest version of netty-jni-util and restored static linking
compatibility.
2021-03-25 11:40:22 +01:00
Norman Maurer
72d7ec9d8c
Let's use gcc10 when cross-compiling for LSE support (#11112)
Motivation:

LSE (https://mysqlonarm.github.io/ARM-LSE-and-MySQL/) can have a huge performance difference. Let's ensure we use a compiler that can support it.

Modifications:

Update to gc10 when cross-compiling as it supports LSE and enables it by default

Result:

More optimized builds for aarch64
2021-03-25 09:32:57 +01:00
Norman Maurer
a82af64047
Continue reading when the number of bytes is less then the configured… (#11089)
... number of bytes when using DatagramChannels

Motivation:

In our FixedRecvByteBufAllocator we dont continue to read if the number of bytes is less then what was configured. This is correct when using it for TCP but not when using it for UDP. When using UDP the number of bytes is the maximum of what we want to support but we often end up processing smaller datagrams in general. Because of this we should use contineReading(UncheckedBooleanSupplier) to determite if we should continue reading

Modifications:

- use contineReading(UncheckedBooleanSupplier) for DatagramChannels

Result:

Read more then once in the general case for DatagramChannels with the default config
2021-03-17 13:10:57 +01:00
Norman Maurer
95652fef12
Allow to configure the maximum number of message to write per eventloop (#11086)
Motivation:

Allow to configure the maximum number of messages to write per eventloop run. This can be useful to ensure we read data in a timely manner and not let writes dominate the CPU time. This is especially useful in protocols like QUIC where you need to read "fast enough" as otherwise you may not read the ACKs fast enough.

Modifications:

- Add new ChannelOption / config that allows to limit the number of messages to write per eventloop run.
- Respect this setting for DatagramChannels

Result:

Reduce the risk of having WRITES block the processing of other events in a timely manner

Co-authored-by: terrarier2111 <58695553+terrarier2111@users.noreply.github.com>
2021-03-17 12:58:06 +01:00
Scott Mitchell
0b0c234bed
SslHandler flushing with TCP Fast Open fix (#11077)
Motivation:
SslHandler owns the responsibility to flush non-application data
(e.g. handshake, renegotiation, etc.) to the socket. However when
TCP Fast Open is supported but the client_hello cannot be written
in the SYN the client_hello may not always be flushed. SslHandler
may not wrap/flush previously written/flushed data in the event
it was not able to be wrapped due to NEED_UNWRAP state being
encountered in wrap (e.g. peer initiated renegotiation).

Modifications:
- SslHandler to flush in channelActive() if TFO is enabled and
  the client_hello cannot be written in the SYN.
- SslHandler to wrap application data after non-application data
  wrap and handshake status is FINISHED.
- SocketSslEchoTest only flushes when writes are done, and waits
  for the handshake to complete before writing.

Result:
SslHandler flushes handshake data for TFO, and previously flushed
application data after peer initiated renegotiation finishes.
2021-03-14 14:18:27 +01:00
Chris Vest
3f9750173c
Make EpollSocketConnectTest correctly configure TFO (#11083)
Motivation:
The EpollSocketConnectTest was not correctly configuring TCP Fast Open on the server socket.
It's an option, not a child option.

Modification:
EpollSocketConnectTest now correctly enables TCP Fast Open on the server side, when available, for the test that needs it.

Result:
Test covers what it was intended to.
2021-03-14 14:17:06 +01:00
Norman Maurer
773ecb81fe
Also support CompositeByteBuf with SegmentedDatagramPacket (#11081)
Motivation:

c22c6b845d introduced support for
UDP_SEGMENT but did restrict it to continous buffers. This is not needed
as it is also fine to use CompositeByteBuf

Modifications:

- Allow to use CompositeByteBuf as well
- Add unit test

Result:

More flexible usage of segmented datagrams possible
2021-03-12 15:37:46 +01:00
Netty Project Bot
e83132fcf2 [maven-release-plugin] prepare for next development iteration 2021-03-09 08:18:31 +00:00
Netty Project Bot
eef26e8bb5 [maven-release-plugin] prepare release netty-4.1.60.Final 2021-03-09 08:18:26 +00:00
Netty Project Bot
b8d843757b [maven-release-plugin] rollback the release of netty-4.1.60.Final 2021-03-09 07:39:13 +00:00
Netty Project Bot
4f099dd11f [maven-release-plugin] prepare for next development iteration 2021-03-09 07:21:55 +00:00
Netty Project Bot
44e392e851 [maven-release-plugin] prepare release netty-4.1.60.Final 2021-03-09 07:21:51 +00:00
Norman Maurer
c22c6b845d
Add support for UDP_SEGMENT (GSO) when using sendmmsg (#11038)
Motivation:

For protocols like QUIC using UDP_SEGMENT (GSO) can help to reduce the
overhead quite a bit. We should support it.

Modifications:

- Add a SegmentedDatagramPacket which can be used to use UDP_SEGMENT
- Add unit test

Result:

Be able to make use of UDP_SEGMENT
2021-02-27 21:55:37 +01:00
wangyuwei
5d012954fd
Avoid letting ipv6 addresses join ipv4 groups (#11015)
Motivation:

#10995 

when `io.netty.channel.unix.Socket` is ipv6 and join a multicast group with ipv4 address will cause `io.netty.channel.ChannelException: setsockopt() failed: Invalid argument` (at least in `Linux centos.dev 4.18.0-240.10.1.el8_3.x86_64 #1 SMP Mon Jan 18 17:05:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux`)


Modification:

check if target group address is ipv6 before call  `io.netty.channel.epoll.LinuxSocket#joinGroup(java.net.InetAddress, java.net.NetworkInterface, java.net.InetAddress)` 

I'm not sure if this modification is currect, but i checked source code of java NIO

```
Java_sun_nio_ch_Net_canJoin6WithIPv4Group0(JNIEnv* env, jclass cl)
{
#if defined(__APPLE__)
    /* IPV6_ADD_MEMBERSHIP can be used to join IPv4 multicast groups */
    return JNI_TRUE;
#else
    /* IPV6_ADD_MEMBERSHIP cannot be used to join IPv4 multicast groups */
    return JNI_FALSE;
#endif
}
```
seems ipv6 address can't join ipv4 group except osx

Result:

test on `Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux` exception  ` setsockopt() failed: Invalid argument` has fixed

Fixes #10995
2021-02-16 14:21:32 +01:00
Chris Vest
56adab2743
TCP Fast Open for clients (#11006)
Support TCP Fast Open for clients and make SslHandler take advantage

Motivation:
- TCP Fast Open allow us to send a small amount of data along side the initial SYN packet when establishing a TCP connection.
- The TLS Client Hello packet is small enough to fit in there, and is also idempotent (another requirement for using TCP Fast Open), so if we can save a round-trip when establishing TLS connections when using TFO.

Modification:
- Add support for client-side TCP Fast Open for Epoll, and also lowers the Linux kernel version requirements to 3.6.
- When adding the SslHandler to a pipeline, if TCP Fast Open is enabled for the channel (and the channel is not already active) then start the handshake early by writing it to the outbound buffer.
- An important detail to note here, is that the outbound buffer is not flushed at this point, like it would for normal handshakes. The flushing happens later as part of establishing the TCP connection.

Result:
- It is now possible for clients (on epoll) to open connections with TCP Fast Open.
- The SslHandler automatically detects when this is the case, and now send its Client Hello message as part of the initial data in the TCP Fast Open flow when available, saving a round-trip when establishing TLS connections.

Co-authored-by: Colin Godsey <crgodsey@gmail.com>
2021-02-15 13:13:44 +01:00
root
411f76d3ad [maven-release-plugin] prepare for next development iteration 2021-02-08 10:48:37 +00:00
root
97d044812d [maven-release-plugin] prepare release netty-4.1.59.Final 2021-02-08 10:47:46 +00:00
Norman Maurer
c735357bf2 Use Files.createTempFile(...) to ensure the file is created with proper permissions
Motivation:

File.createTempFile(String, String)` will create a temporary file in the system temporary directory if the 'java.io.tmpdir'. The permissions on that file utilize the umask. In a majority of cases, this means that the file that java creates has the permissions: `-rw-r--r--`, thus, any other local user on that system can read the contents of that file.
This can be a security concern if any sensitive data is stored in this file.

This was reported by Jonathan Leitschuh <jonathan.leitschuh@gmail.com> as a security problem.

Modifications:

Use Files.createTempFile(...) which will use safe-defaults when running on java 7 and later. If running on java 6 there isnt much we can do, which is fair enough as java 6 shouldnt be considered "safe" anyway.

Result:

Create temporary files with sane permissions by default.
2021-02-08 11:44:05 +01:00
Norman Maurer
5b41f3d25b
Ensure native methods for unix-native-common are only registered once. (#10932)
Motiviation:

We need to ensure we only register the methods for unix-native-common once as otherwise it may have strange side-effects.

Modifications:

- Add extra method that should be called to signal that we need to register the methods. The registration will only happen once.
- Adjust code to make use of it.

Result:

No more problems due incorrect registration of these methods.
2021-01-14 17:52:04 +01:00
root
a137ce2042 [maven-release-plugin] prepare for next development iteration 2021-01-13 10:28:54 +00:00
root
10b03e65f1 [maven-release-plugin] prepare release netty-4.1.58.Final 2021-01-13 10:27:17 +00:00
Norman Maurer
d58ce7a151
Revert "Ensure we only register native methods once (#10876)" (#10928)
Motivation:

This reverts commit 7fb62a93b8 as it broke native loading in some cases due maven dependencies.

Modification:

Revert the commit.

Result:

Native loading works again
2021-01-13 11:07:39 +01:00
root
c6b894d03d [maven-release-plugin] prepare for next development iteration 2021-01-12 11:10:44 +00:00
root
b016568e21 [maven-release-plugin] prepare release netty-4.1.57.Final 2021-01-12 11:10:20 +00:00
Norman Maurer
11d2ce7614
Add fallback for android when trying to access the filedescriptor via JNI (#10882)
Motivation:

Android seems to use a different field name so we should also try to access it with the name used by android.

Modifications:

Try first fd and if this fails try descriptor as field name

Result:

Workaround for android.
2020-12-22 20:42:52 +01:00
Norman Maurer
7fb62a93b8
Ensure we only register native methods once (#10876)
Motivation:

We need to ensure we only register native methods once as otherwise we may end up in an "invalid" state. The problem here was that before it was basically the responsibility the user of transport-native-unix-common to register the methods. This is error prone as there may be multiple users of these on the classpath at the same time.

Modifications:

- Provide a way to init native lib without register the native methods of the provided classes. This is needed to be able to re-use functionality which is exposed to our internal native code
- Use flatten plugin to correctly resolve classifier and so have the correct dependency
- Call Unix.* method to ensure we register the methods correctly once
- Include native lib as well in the native jars of unix-common

Result:

Be able to have multiple artifacts of the classpath that depends on the unix-common. Related to https://github.com/netty/netty-incubator-transport-io_uring/issues/15
2020-12-18 10:37:49 +01:00
root
a9ec3d86f6 [maven-release-plugin] prepare for next development iteration 2020-12-17 06:11:39 +00:00
root
1188d8320e [maven-release-plugin] prepare release netty-4.1.56.Final 2020-12-17 06:11:18 +00:00
root
f57d64f1c7 [maven-release-plugin] prepare for next development iteration 2020-12-08 11:51:39 +00:00
root
38da45ffe1 [maven-release-plugin] prepare release netty-4.1.55.Final 2020-12-08 11:51:25 +00:00
Norman Maurer
3ac9685580
Use aarch_64 in a consistent way (#10845)
Motivation:

We should use aarch_64 in our classifier / jni libname on aarch64 as  os.detected.arch uses the name. Being non consistent (especially across our different projects) already gave us a lot of trouble in the past.
Let's fix this once for all.

Modifications:

Use aarch_64

Result:

More consistent classifier usage on aarch64
2020-12-07 12:14:23 +01:00