Commit Graph

40 Commits

Author SHA1 Message Date
Tatsushi Inagaki
ed142442a4 Enable Netty on a big endian platform
Motivation:

We would like to enable Netty also on a big endian platform such as
s390_64. We need to fix a function which assumes that the target
platform is little endian.

Modifications:

Modify netty_unix_socket_accept() to write an address length as jbyte
instead of jsize.

Result:

Netty can be enabled on a big endian platform.

Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>
2019-09-27 12:27:37 +02:00
Norman Maurer
5e69a13c21
Cleanup JNI code to always correctly free memory when loading fails and also correctly respect out of memory in all cases (#9596)
Motivation:

At the moment we not consistently (and also not correctly) free allocated native memory in all cases during loading the JNI library. This can lead to native memory leaks in the unlikely case of failure while trying to load the library.

Beside this we also not always correctly handle the case when a new java object can not be created in native code because of out of memory.

Modification:

- Copy some macros from netty-tcnative to be able to handle errors in a more easy fashion
- Correctly account for New* functions to return NULL
- Share code

Result:

More robust and clean JNI code
2019-09-24 07:18:35 +02:00
Norman Maurer
7a547aab65
Correctly handle IPV6-mapped-IPV4 addresses in native code when receiving datagrams (#9560)
Motivation:

291f80733a introduced a change to use a byte[] to construct the InetAddress when receiving datagram messages to reduce the overhead. Unfortunally it introduced a regression when handling IPv6-mapped-IPv4 addresses and so produced an IndexOutOfBoundsException when trying to fill the byte[] in native code.

Modifications:

- Correctly use the offset on the pointer of the address.
- Add testcase
- Make tests more robust and include more details when the test fails

Result:

No more IndexOutOfBoundsException
2019-09-11 20:30:28 +02:00
Norman Maurer
6fc7c589f0
Correctly handle ipv6 mapped ipv4 addresses when using recvmmsg (#9541)
Motivation:

394a1b3485 introduced the possibility to use recvmmsg(...) but did not correctly handle ipv6 mapped ip4 addresses to make it consistent with other transports.

Modifications:

- Correctly handle ipv6 mapped ipv4 addresses by only copy over the relevant bytes
- Small improvement on how to detect ipv6 mapped ipv4 addresses by using memcmp and not byte by byte compare
- Adjust test to cover this bug

Result:

Correctly handle ipv6 mapped ipv4 addresses
2019-09-06 13:54:29 +02:00
Norman Maurer
394a1b3485
Add support for recvmmsg when using epoll transport (#9509)
Motivation:

When using datagram sockets which need to handle a lot of packets it makes sense to use recvmmsg to be able to read multiple datagram packets with one syscall.

Modifications:

- Add support for recvmmsg on linux
- Add new EpollChannelOption.MAX_DATAGRAM_PACKET_SIZE
- Add tests

Result:

Fixes https://github.com/netty/netty/issues/8446.
2019-09-03 08:40:17 +02:00
Norman Maurer
291f80733a
Use byte[] to create DatagramSocketAddress and so reduce overhead (#9516)
Motivation:

At the moment we use the String representation of the IP to create the DatagramSocketAddress. This is not for free and we should better use the byte[] directly to reduce the overhead of parsing the String (and creating it in the first place)

Modifications:

Directly use byte[] as input for the DatagramSocketAddress

Result:

Less overhead when using Datagrams with native transports
2019-08-30 09:22:44 +02:00
Steve Buzzard
70731bfa7e Added UDP multicast (with caveats: getInterface, getNetworkInterface, block or loopback-mode-disabled operations).
Motivation:

Provide epoll/native multicast to support high load multicast users (we are using it for a high load telecomm app at my day job).

Modification:

Added support for source specific and any source multicast for epoll transport. Some caveats: no support for disabling loop back mode, retrieval of interface and block operation, all of which tend to be less frequently used.

Result:

Provides epoll transport multicast for common use cases.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2019-05-25 08:00:16 +02:00
Norman Maurer
f17bfd0f64
Only use static Exception instances when we can ensure addSuppressed … (#9152)
Motivation:

OOME is occurred by increasing suppressedExceptions because other libraries call Throwable#addSuppressed. As we have no control over what other libraries do we need to ensure this can not lead to OOME.

Modifications:

Only use static instances of the Exceptions if we can either dissable addSuppressed or we run on java6.

Result:

Not possible to OOME because of addSuppressed. Fixes https://github.com/netty/netty/issues/9151.
2019-05-17 22:23:02 +02:00
Scott Mitchell
ac023da16d Correctly handle overflow in Native.kevent(...) when EINTR is detected (#9024)
Motivation:
When kevent(...) returns with EINTR we do not correctly decrement the timespec
structure contents to account for the time duration. This may lead to negative
values for tv_nsec which will result in an EINVAL and raise an IOException to
the event loop selection loop.

Modifications:
Correctly calculate new timeoutTs when EINTR is detected

Result:
Fixes #9013.
2019-04-10 11:04:13 +02:00
Norman Maurer
1725504a37
Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs… (#8921)
* Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs related to GCLocker

Motivation:

GetPrimitiveArrayCritical(...) may cause multiple not-fixed bugs related to the GCLocker while there is little gain for our use-case. We should just use GetByteArrayRegion(...) and copy into a small on-stack buffer.

See also:

- https://shipilev.net/jvm/anatomy-quarks/9-jni-critical-gclocker/#_g1
- https://bugs.openjdk.java.net/browse/JDK-8048556
- https://bugs.openjdk.java.net/browse/JDK-8057573
- https://bugs.openjdk.java.net/browse/JDK-8057586

Special thanks to @jayv @shipilev @apangin for the pointers.

Modifications:

Replace GetPrimitiveArrayCritical(...) with GetByteArrayRegion(...)

Result:

Less risks hitting GCLocker related bugs.
2019-03-07 10:30:55 +01:00
田欧
a33200ca38 use checkPositive/checkPositiveOrZero (#8803)
Motivation:

We have a utility method to check for > 0 and >0 arguments. We should use it.

Modification:

use checkPositive/checkPositiveOrZero instead of if statement.

Result:

Re-use utility method.
2019-01-31 09:07:14 +01:00
kashike
6fdd7fcddb Fix minor spelling issues in javadocs (#8701)
Motivation:

Javadocs contained some spelling errors, we should fix these.

Modification:

Fix spelling

Result:

Javadoc cleanup.
2019-01-14 07:24:34 +01:00
Norman Maurer
54f565ac67
Allow to use native transports when sun.misc.Unsafe is not present on… (#8231)
* Allow to use native transports when sun.misc.Unsafe is not present on the system

Motivation:

We should be able to use the native transports (epoll / kqueue) even when sun.misc.Unsafe is not present on the system. This is especially important as Java11 will be released soon and does not allow access to it by default.

Modifications:

- Correctly disable usage of sun.misc.Unsafe when -PnoUnsafe is used while running the build
- Correctly increment metric when UnpooledDirectByteBuf is allocated. This was uncovered once -PnoUnsafe usage was fixed.
- Implement fallbacks in all our native transport code for when sun.misc.Unsafe is not present.

Result:

Fixes https://github.com/netty/netty/issues/8229.
2018-08-29 19:36:33 +02:00
Scott Mitchell
fa8f967852 netty_unix_socket free nettyClassName after using it
Motivation:
netty_unix_socket attempts to use nettyClassName in an error message, but previously freed the memory. We should wait to free the memory until after we use it.

Modifications:
- Free nettyClassName after using it in snprintf

Result:
More useful error message.
2018-07-04 09:16:13 +02:00
Norman Maurer
7f5e77484c
Fix support for shading native libraries which was broken in b818852cdb. (#8091)
Motivation:

b818852cdb broke support for shading the native libraries in netty as it missed to respect the package prefix that is used when shading.

Modifications:

Correctly respect package prefix for constructor argument and include the used classname when logging that we could not find the constructor.

Result:

Be able to shade native libraries of netty again.
2018-07-03 07:05:12 +02:00
Norman Maurer
b5fde3e0cd
Add #ifdef statements so the compilation of the native code not fails on MacOS (#7895)
Motivation:

b818852cdb added support for IP_RECVORIGDSTADDR but did not include any #ifdef statements to ensure its usable at all (which is not the case on MacOS).

Modifications:

Add #ifdef statements to check for IP_RECVORIGDSTADDR.

Result:

Compilation works again on MacOS.
2018-04-28 22:04:35 +02:00
Devrim Şahin
b818852cdb Update DatagramPacket.recipient() to return the actual destination IP (#7879)
Motivation:

DatagramPacket.recipient() doesn't return the actual destination IP, but the IP the app is bound to.

Modification:

- IP_RECVORIGDSTADDR option is enabled for UDP sockets, which allows retrieval of ancillary information containing the original recipient.
- _recvFrom(...) function from transport-native-unix-common/src/main/c/netty_unix_socket.c is modified such that if IP_RECVORIGDSTADDR is set, recvmsg is used instead of recvfrom; enabling the retrieval of the original recipient.
- DatagramSocketAddress also contains a 'local' address, representing the recipient.
- EpollDatagramChannel is updated to return the retrieved recipient address instead of the address the channel is bound to.

Result:

Fixes #4950.
2018-04-26 08:00:36 +02:00
Norman Maurer
8189399e9d Ignore EINTR on close(...) as there is nothing sane we can do.
Motivation:

If close(...) reports EINTR there is nothing sane we can do so it makes no sense to even report it. See also:

https://github.com/apple/swift-nio/pull/217

Modifications:

Just ignore EINTR when calling close(...)

Result:

Less noise in the logs.
2018-03-23 07:39:17 +01:00
Norman Maurer
f6251c8256 IovArray.add(...) should check if buffer has memory address.
Motivation:

We currently not check if the buffer has a memory address and just assume this is the case if the nioBufferCount() == 1.

Modifications:

- Check hasMemoryAddress() before trying to access it.
- Add unit case.

Result:

More correct and robust code. Related to [#7752].
2018-03-13 08:51:01 +01:00
Scott Mitchell
33ddb83dc1
IovArray#add return value resulted in more ByteBufs being added during iteration
Motivation:
IovArray implements MessageProcessor, and the processMessage method will continue to be called during iteration until it returns true. A recent commit b215794de3 changed the return value to only return true if any component of a CompositeByteBuf was added as a result of the method call. However this results in the iteration continuing, and potentially subsequent smaller buffers maybe added, which will result in out of order writes and generally corrupts data.

Modifications:
- IovArray#add should return false so that the MessageProcessor#processMessage will stop iterating.

Result:
Native transports which use IovArray will not corrupt data during gathering writes of CompositeByteBuf objects.
2018-01-04 08:04:32 -08:00
Scott Mitchell
af2f343648
FileDescriptor writev core dump
Motivation:
FileDescriptor#writev calls JNI code, and that JNI code dereferences a NULL pointer which crashes the application. This occurs when writing a single CompositeByteBuf object with more than one component.

Modifications:
- Initialize the iovec iterator properly to avoid the core dump
- Fix the array length calculation if we aren't able to fit all the ByteBuffer objects in the iovec array

Result:
No more core dump.
2017-12-14 16:47:31 -08:00
Scott Mitchell
b215794de3
Enforce writeSpinCount to limit resource consumption per socket (#7478)
Motivation:
The writeSpinCount currently loops over the same buffer, gathering
write, file write, or other write operation multiple times but will
continue writing until there is nothing left or the OS doesn't accept
any data for that specific write. However if the OS keeps accepting
writes there is no way to limit how much time we spend on a specific
socket. This can lead to unfair consumption of resources dedicated to a
single socket.
We currently don't limit the amount of bytes we attempt to write per
gathering write. If there are many more bytes pending relative to the
SO_SNDBUF size we will end up building iov arrays with more elements
than can be written, which results in extra iteration, conditionals,
and book keeping.

Modifications:
- writeSpinCount should limit the number of system calls we make to
write data, instead of applying to individual write operations
- IovArray should support a maximum number of bytes
- IovArray should support composite buffers of greater than size 1024
- We should auto-scale the amount of data that we attempt to write per
gathering write operation relative to SO_SNDBUF and how much data is
successfully written
- The non-unsafe path should also support a maximum number of bytes,
and respect the IOV_MAX limit

Result:
Write resource consumption can be bounded and gathering writes have
a limit relative to the amount of data which can actually be accepted
by the socket.
2017-12-07 16:00:52 -08:00
Norman Maurer
3f101caa4c Not call java methods from within JNI init code to prevent class loading deadlocks.
Motivation:

We used NetUtil.isIpV4StackPreferred() when loading JNI code which tries to load NetworkInterface in its static initializer. Unfortunally a lock on the NetworkInterface class init may be already hold somewhere else which may cause a loader deadlock.

Modifications:

Add a new Socket.initialize() method that will be called when init the library and pass everything needed to the JNI level so we not need to call back to java.

Result:

Fixes [#7458].
2017-12-06 14:34:15 +01:00
Carl Mastrangelo
d8cb9ce09f Throw FileNotFoundException when connecting to a missing UDS path
Motivation:
Exception handling is nicer when a more specific Exception is thrown

Modification:
Add a static reference for ENOENT, and throw FNFE if it is returned

Result:
More precise exception handling
2017-11-28 13:18:58 +01:00
Norman Maurer
b1cc5835ac Not directly call getsockopt but use exported helper function
Motivation:

To better isolate OS system calls we should not call getsockopt directly but use our netty_unix_socket_getOption0 function. See is a followup of f115bf5.

Modifications:

Export netty_unix_socket_getOption0 by declaring it in the header file and use it

Result:

Better isolation of system calls.
2017-11-22 08:40:44 +01:00
Norman Maurer
f115bf50cb EpollSocketChannelConfig.getOptions() must not throw if TCP_FASTOPEN_CONNECT is not supported
Motivation:

If a user calls EpollSocketChannelConfig.getOptions() and TCP_FASTOPEN_CONNECT is not supported we throw an exception.

Modifications:

- Just return 0 if ENOPROTOOPT is set.
- Add testcase

Result:

getOptions() works as epxected.
2017-11-16 14:07:54 +01:00
Norman Maurer
d8e187ff2c Ensure setting / getting the traffic class on an ipv4 only system works when using the native transport.
Motivation:

We tried to set IPV6 opts on an ipv4 only system and so failed to set / get the traffic opts. This resulted in a test-error when trying to compile netty on ipv4 only systems.

Modifications:

Use the correct opts depending on if the system is ipv4 only or not.

Result:

Be able to build and use on ipv4 only systems.
2017-10-24 09:03:28 +02:00
Norman Maurer
7290cbc48a More bullet-proof way of detecting if ipv6 is supported or not when using native transport
Motivation:

We should try to bind to an ipv6 only socket before we enable ipv6 support in the native transport as it may not work due setup of the platform.

Modifications:

Try to bind to ::1 use IPV6 later on if this works

Result:

Fixes [#7021].
2017-08-23 13:46:09 +02:00
Norman Maurer
19dcb15062 Use underscore in native library names for consistency.
Motivation:

At the moment we try to load the library using multiple names which includes names using - but also _ . We should just use _ all the time.

Modifications:

Replace - with _

Result:

Fixes [#7069]
2017-08-15 06:02:00 +02:00
Norman Maurer
4bb89dcc54 Correctly handle connect/disconnect in EpollDatagramChannel / KQueueDatagramChannel
Motivation:

We did not correctly handle connect() and disconnect() in EpollDatagramChannel / KQueueDatagramChannel and so the behavior was different compared to NioDatagramChannel.

Modifications:

- Correct implement connect and disconnect methods
- Share connect and related code
- Add tests

Result:

EpollDatagramChannel / KQueueDatagramChannel also supports correctly connect() and disconnect() methods.
2017-08-04 09:22:53 +02:00
Norman Maurer
f23b2fc25d Use 4 spaces and not 2 spaces (cleanup of 3d22b24244) 2017-07-20 10:24:18 +02:00
Carl Mastrangelo
3d22b24244 Allowed Netty Epoll/Kqueue to work in IPv6 Only environments
Motivation:
In some environments, IPv4 may be disabled (at a kernel level).
Google has such an environment for testing v4 -> v6 transition
paths.  This give confidence that code is v6 ready.

Modifications:
Change native socket code to ignore failures of trying to enter
dual stack mode.  This change has been made to Google's internal
JDK, and will/should be upstreamed to OpenJDK eventually.

Results:
Netty works in IPv6 only environments

Fixes: #6993
2017-07-19 18:26:37 -07:00
tonyshenkk
c8a23028ad fix UnixChannelUtil#isBufferCopyNeededForWrite
fix not execute unit test in transport-native-unix-common-tests module

Motivation:

- Commit 047da11 introduced an bug for still copy byteBuf for composed of n(n <= IOV_MAX) NIO direct buffers
- Commit 3c4dfed add UnixChannelUtilTest in transport-native-unix-common-tests module, but not execute in maven compile

as issue #6825, #6870

Modifications:

- modified UnixChannelUtil#isBufferCopyNeededForWrite(ByteBuf), and UnixChannelUtilTest
- move UnixChannelUtilTest into transport-native-unix-common module, and add packet scope method UnixChannelUtil#isBufferCopyNeededForWrite(ByteBuf, int)

Result:

- no copy byteBuf for composed of n(n <= IOV_MAX) NIO direct buffers
- auto execute unit tests in UnixChannelUtilTest and it is easier to mock IOV_MAX
2017-06-24 21:27:54 -07:00
Carl Mastrangelo
83de77fbe5 Make Native loading work better with Java 8
Motivation:
Enable static linking for Java 8.  These commits are the same as those introduced to netty tcnative.  The goal is to allow lots of JNI libraries to be statically linked together without having conflict `JNI_OnLoad` methods.

Modification:
* add JNI_OnLoad suffixes to enable static linking
* Add static names to the list of libraries that try to be loaded
* Enable compiling with JNI 1.8
* Sort includes

Result:
Enable statically linked JNI code.
2017-06-23 19:42:13 +02:00
Carl Mastrangelo
b985615522 Fix compiler warnings in netty Epoll and unix common
Motivation:
Google requires stricter compilation by adding -Werror and enabling many other warnings.

Modification:

* fix warning caused by -Wmissing-braces

* Use the address of `sendmmsg` rather than the function itself when
checking for presence.  This resovles the warning caused by
`-Wpointer-bool-conversion`.

More detail:
When compiling on Linux, `sendmmsg` is always present, so the
function is always nonnull.  When compiling elsewhere, the
function is defined as `__attribute__((weak))` which means it
may be absent at link time.  This is controlled by
`IO_NETTY_SENDMMSG_NOT_FOUND`, which is off by default.

The reason for the error is due to the risk of accidentally not
calling the function.  By adding `&` before the function, there
is no ambiguity.  (the result of the fn call cannot have its
address taken.)

* use != to check for sendmmsg

Result:
Easier compilation.
2017-06-23 07:19:46 +02:00
Norman Maurer
047da11086 Correctly handle ByteBuf implementations which have no memoryAddress when writing to native transport
Motivation:

Commit 3c4dfed08a introduced a regression in handling buffers that have no memoryAddress.

Modifications:

Fix regression and also add unit tests.

Result:

It's possible again to write buffers without memory address.
2017-06-07 19:36:05 +02:00
Norman Maurer
0b0309624a Correctly parse package-prefix if the jni lib is contained in a path that matches the library name.
Motivation:

We used strstr to find the path to the library, which fails if the library is contained in a directory that also matches the library name.

Modifications:

- Introduce netty_unix_util_strstr_last which will return a pointer which points to the last accourance and so not fails if the direct also matches the library name.

Result:

Be able to load the library in all cases.
2017-05-29 20:09:57 +02:00
louxiu
3c4dfed08a Fix handle of ByteBuf with multi nioBuffer in EpollDatagramChannel and KQueueDatagramChannel
Motivation:
1. special handling of ByteBuf with multi nioBuffer rather than type of CompositeByteBuf (eg. DuplicatedByteBuf with CompositeByteBuf)
2. EpollDatagramUnicastTest and KQueueDatagramUnicastTest passed because CompositeByteBuf is converted to DuplicatedByteBuf before write to channel
3. uninitalized struct msghdr will raise error

Modifications:
1. isBufferCopyNeededForWrite(like isSingleDirectBuffer in NioDatgramChannel) checks wether a new direct buffer is needed
2. special handling of ByteBuf with multi nioBuffer in EpollDatagramChannel, AbstractEpollStreamChannel, KQueueDatagramChannel, AbstractKQueueStreamChannel and IovArray
3. initalize struct msghdr

Result:
handle of ByteBuf with multi nioBuffer in EpollDatagramChannel and KQueueDatagramChannel are ok
2017-05-26 07:56:34 +02:00
Scott Mitchell
4c6d946fba KQueueSocket#setTrafficClass exceptions
Motivation:
MacOS will throw an error when attempting to set the IP_TOS socket option if IPv6 is available, and also when getting the value for IP_TOS.

Modifications:
- Socket#setTrafficClass and Socket#getTrafficClass should try to use IPv6 first, and check if the error code indicates the protocol is not supported before trying IPv4

Result:
Fixes https://github.com/netty/netty/issues/6741.
2017-05-18 11:26:27 -07:00
Scott Mitchell
3cc4052963 New native transport for kqueue
Motivation:
We currently don't have a native transport which supports kqueue https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2. This can be useful for BSD systems such as MacOS to take advantage of native features, and provide feature parity with the Linux native transport.

Modifications:
- Make a new transport-native-unix-common module with all the java classes and JNI code for generic unix items. This module will build a static library for each unix platform, and included in the dynamic libraries used for JNI (e.g. transport-native-epoll, and eventually kqueue).
- Make a new transport-native-unix-common-tests module where the tests for the transport-native-unix-common module will live. This is so each unix platform can inherit from these test and ensure they pass.
- Add a new transport-native-kqueue module which uses JNI to directly interact with kqueue

Result:
JNI support for kqueue.
Fixes https://github.com/netty/netty/issues/2448
Fixes https://github.com/netty/netty/issues/4231
2017-05-03 09:53:22 -07:00