Commit Graph

86 Commits

Author SHA1 Message Date
Scott Mitchell
b215794de3
Enforce writeSpinCount to limit resource consumption per socket (#7478)
Motivation:
The writeSpinCount currently loops over the same buffer, gathering
write, file write, or other write operation multiple times but will
continue writing until there is nothing left or the OS doesn't accept
any data for that specific write. However if the OS keeps accepting
writes there is no way to limit how much time we spend on a specific
socket. This can lead to unfair consumption of resources dedicated to a
single socket.
We currently don't limit the amount of bytes we attempt to write per
gathering write. If there are many more bytes pending relative to the
SO_SNDBUF size we will end up building iov arrays with more elements
than can be written, which results in extra iteration, conditionals,
and book keeping.

Modifications:
- writeSpinCount should limit the number of system calls we make to
write data, instead of applying to individual write operations
- IovArray should support a maximum number of bytes
- IovArray should support composite buffers of greater than size 1024
- We should auto-scale the amount of data that we attempt to write per
gathering write operation relative to SO_SNDBUF and how much data is
successfully written
- The non-unsafe path should also support a maximum number of bytes,
and respect the IOV_MAX limit

Result:
Write resource consumption can be bounded and gathering writes have
a limit relative to the amount of data which can actually be accepted
by the socket.
2017-12-07 16:00:52 -08:00
Norman Maurer
3f101caa4c Not call java methods from within JNI init code to prevent class loading deadlocks.
Motivation:

We used NetUtil.isIpV4StackPreferred() when loading JNI code which tries to load NetworkInterface in its static initializer. Unfortunally a lock on the NetworkInterface class init may be already hold somewhere else which may cause a loader deadlock.

Modifications:

Add a new Socket.initialize() method that will be called when init the library and pass everything needed to the JNI level so we not need to call back to java.

Result:

Fixes [#7458].
2017-12-06 14:34:15 +01:00
Tomasz Jędrzejewski
e8540c2b7a Adding stable JDK9 module names that follow reverse-DNS style
Automatic-Module-Name entry provides a stable JDK9 module name, when Netty is used in a modular JDK9 applications. More info: http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html

When Netty migrates to JDK9 in the future, the entry can be replaced by actual module-info descriptor.

Modification:

The POM-s are configured to put the correct module names to the manifest.

Result:

Fixes #7218.
2017-11-29 11:50:24 +01:00
Carl Mastrangelo
d8cb9ce09f Throw FileNotFoundException when connecting to a missing UDS path
Motivation:
Exception handling is nicer when a more specific Exception is thrown

Modification:
Add a static reference for ENOENT, and throw FNFE if it is returned

Result:
More precise exception handling
2017-11-28 13:18:58 +01:00
Norman Maurer
b1cc5835ac Not directly call getsockopt but use exported helper function
Motivation:

To better isolate OS system calls we should not call getsockopt directly but use our netty_unix_socket_getOption0 function. See is a followup of f115bf5.

Modifications:

Export netty_unix_socket_getOption0 by declaring it in the header file and use it

Result:

Better isolation of system calls.
2017-11-22 08:40:44 +01:00
Norman Maurer
f115bf50cb EpollSocketChannelConfig.getOptions() must not throw if TCP_FASTOPEN_CONNECT is not supported
Motivation:

If a user calls EpollSocketChannelConfig.getOptions() and TCP_FASTOPEN_CONNECT is not supported we throw an exception.

Modifications:

- Just return 0 if ENOPROTOOPT is set.
- Add testcase

Result:

getOptions() works as epxected.
2017-11-16 14:07:54 +01:00
Norman Maurer
188ea59c9d [maven-release-plugin] prepare for next development iteration 2017-11-08 22:36:53 +00:00
Norman Maurer
812354cf1f [maven-release-plugin] prepare release netty-4.1.17.Final 2017-11-08 22:36:33 +00:00
Norman Maurer
d8e187ff2c Ensure setting / getting the traffic class on an ipv4 only system works when using the native transport.
Motivation:

We tried to set IPV6 opts on an ipv4 only system and so failed to set / get the traffic opts. This resulted in a test-error when trying to compile netty on ipv4 only systems.

Modifications:

Use the correct opts depending on if the system is ipv4 only or not.

Result:

Be able to build and use on ipv4 only systems.
2017-10-24 09:03:28 +02:00
Norman Maurer
625a7426cd [maven-release-plugin] prepare for next development iteration 2017-09-25 06:12:32 +02:00
Norman Maurer
f57d8f00e1 [maven-release-plugin] prepare release netty-4.1.16.Final 2017-09-25 06:12:16 +02:00
Norman Maurer
b967805f32 [maven-release-plugin] prepare for next development iteration 2017-08-24 15:38:22 +02:00
Norman Maurer
da8e010a42 [maven-release-plugin] prepare release netty-4.1.15.Final 2017-08-24 15:37:59 +02:00
Norman Maurer
7290cbc48a More bullet-proof way of detecting if ipv6 is supported or not when using native transport
Motivation:

We should try to bind to an ipv6 only socket before we enable ipv6 support in the native transport as it may not work due setup of the platform.

Modifications:

Try to bind to ::1 use IPV6 later on if this works

Result:

Fixes [#7021].
2017-08-23 13:46:09 +02:00
Norman Maurer
19dcb15062 Use underscore in native library names for consistency.
Motivation:

At the moment we try to load the library using multiple names which includes names using - but also _ . We should just use _ all the time.

Modifications:

Replace - with _

Result:

Fixes [#7069]
2017-08-15 06:02:00 +02:00
Norman Maurer
4bb89dcc54 Correctly handle connect/disconnect in EpollDatagramChannel / KQueueDatagramChannel
Motivation:

We did not correctly handle connect() and disconnect() in EpollDatagramChannel / KQueueDatagramChannel and so the behavior was different compared to NioDatagramChannel.

Modifications:

- Correct implement connect and disconnect methods
- Share connect and related code
- Add tests

Result:

EpollDatagramChannel / KQueueDatagramChannel also supports correctly connect() and disconnect() methods.
2017-08-04 09:22:53 +02:00
Norman Maurer
52f384b37f [maven-release-plugin] prepare for next development iteration 2017-08-02 12:55:10 +00:00
Norman Maurer
8cc1071881 [maven-release-plugin] prepare release netty-4.1.14.Final 2017-08-02 12:54:51 +00:00
Norman Maurer
f23b2fc25d Use 4 spaces and not 2 spaces (cleanup of 3d22b24244) 2017-07-20 10:24:18 +02:00
Carl Mastrangelo
3d22b24244 Allowed Netty Epoll/Kqueue to work in IPv6 Only environments
Motivation:
In some environments, IPv4 may be disabled (at a kernel level).
Google has such an environment for testing v4 -> v6 transition
paths.  This give confidence that code is v6 ready.

Modifications:
Change native socket code to ignore failures of trying to enter
dual stack mode.  This change has been made to Google's internal
JDK, and will/should be upstreamed to OpenJDK eventually.

Results:
Netty works in IPv6 only environments

Fixes: #6993
2017-07-19 18:26:37 -07:00
Norman Maurer
2a376eeb1b [maven-release-plugin] prepare for next development iteration 2017-07-06 13:24:06 +02:00
Norman Maurer
c7f8168324 [maven-release-plugin] prepare release netty-4.1.13.Final 2017-07-06 13:23:51 +02:00
Scott Mitchell
07a641900c transport-native-kqueue libraries should not lazy link
Motivation:
We rely upon the linker being non-lazy to test compatibility the native library compatibility for kqueue, but the default mode of operation is to lazy link.

Modifications:
- We should modify the build scripts to inform the linker that this library should not be lazy linked
- Error messages changes
dyld: lazy symbol binding failed: Symbol not found: _clock_gettime

java.lang.UnsatisfiedLinkError: unsupported JNI version 0xFFFFFFFF required by .../libnetty-transport-native-kqueue.dylib

Result:
Link errors are detected upon library load time.
2017-06-28 11:47:16 -04:00
tonyshenkk
c8a23028ad fix UnixChannelUtil#isBufferCopyNeededForWrite
fix not execute unit test in transport-native-unix-common-tests module

Motivation:

- Commit 047da11 introduced an bug for still copy byteBuf for composed of n(n <= IOV_MAX) NIO direct buffers
- Commit 3c4dfed add UnixChannelUtilTest in transport-native-unix-common-tests module, but not execute in maven compile

as issue #6825, #6870

Modifications:

- modified UnixChannelUtil#isBufferCopyNeededForWrite(ByteBuf), and UnixChannelUtilTest
- move UnixChannelUtilTest into transport-native-unix-common module, and add packet scope method UnixChannelUtil#isBufferCopyNeededForWrite(ByteBuf, int)

Result:

- no copy byteBuf for composed of n(n <= IOV_MAX) NIO direct buffers
- auto execute unit tests in UnixChannelUtilTest and it is easier to mock IOV_MAX
2017-06-24 21:27:54 -07:00
Carl Mastrangelo
83de77fbe5 Make Native loading work better with Java 8
Motivation:
Enable static linking for Java 8.  These commits are the same as those introduced to netty tcnative.  The goal is to allow lots of JNI libraries to be statically linked together without having conflict `JNI_OnLoad` methods.

Modification:
* add JNI_OnLoad suffixes to enable static linking
* Add static names to the list of libraries that try to be loaded
* Enable compiling with JNI 1.8
* Sort includes

Result:
Enable statically linked JNI code.
2017-06-23 19:42:13 +02:00
Carl Mastrangelo
b985615522 Fix compiler warnings in netty Epoll and unix common
Motivation:
Google requires stricter compilation by adding -Werror and enabling many other warnings.

Modification:

* fix warning caused by -Wmissing-braces

* Use the address of `sendmmsg` rather than the function itself when
checking for presence.  This resovles the warning caused by
`-Wpointer-bool-conversion`.

More detail:
When compiling on Linux, `sendmmsg` is always present, so the
function is always nonnull.  When compiling elsewhere, the
function is defined as `__attribute__((weak))` which means it
may be absent at link time.  This is controlled by
`IO_NETTY_SENDMMSG_NOT_FOUND`, which is off by default.

The reason for the error is due to the risk of accidentally not
calling the function.  By adding `&` before the function, there
is no ambiguity.  (the result of the fn call cannot have its
address taken.)

* use != to check for sendmmsg

Result:
Easier compilation.
2017-06-23 07:19:46 +02:00
Norman Maurer
fd67a2354d [maven-release-plugin] prepare for next development iteration 2017-06-08 21:06:24 +02:00
Norman Maurer
3acd5c68ea [maven-release-plugin] prepare release netty-4.1.12.Final 2017-06-08 21:06:01 +02:00
Norman Maurer
047da11086 Correctly handle ByteBuf implementations which have no memoryAddress when writing to native transport
Motivation:

Commit 3c4dfed08a introduced a regression in handling buffers that have no memoryAddress.

Modifications:

Fix regression and also add unit tests.

Result:

It's possible again to write buffers without memory address.
2017-06-07 19:36:05 +02:00
Norman Maurer
0b0309624a Correctly parse package-prefix if the jni lib is contained in a path that matches the library name.
Motivation:

We used strstr to find the path to the library, which fails if the library is contained in a directory that also matches the library name.

Modifications:

- Introduce netty_unix_util_strstr_last which will return a pointer which points to the last accourance and so not fails if the direct also matches the library name.

Result:

Be able to load the library in all cases.
2017-05-29 20:09:57 +02:00
louxiu
3c4dfed08a Fix handle of ByteBuf with multi nioBuffer in EpollDatagramChannel and KQueueDatagramChannel
Motivation:
1. special handling of ByteBuf with multi nioBuffer rather than type of CompositeByteBuf (eg. DuplicatedByteBuf with CompositeByteBuf)
2. EpollDatagramUnicastTest and KQueueDatagramUnicastTest passed because CompositeByteBuf is converted to DuplicatedByteBuf before write to channel
3. uninitalized struct msghdr will raise error

Modifications:
1. isBufferCopyNeededForWrite(like isSingleDirectBuffer in NioDatgramChannel) checks wether a new direct buffer is needed
2. special handling of ByteBuf with multi nioBuffer in EpollDatagramChannel, AbstractEpollStreamChannel, KQueueDatagramChannel, AbstractKQueueStreamChannel and IovArray
3. initalize struct msghdr

Result:
handle of ByteBuf with multi nioBuffer in EpollDatagramChannel and KQueueDatagramChannel are ok
2017-05-26 07:56:34 +02:00
Scott Mitchell
4c6d946fba KQueueSocket#setTrafficClass exceptions
Motivation:
MacOS will throw an error when attempting to set the IP_TOS socket option if IPv6 is available, and also when getting the value for IP_TOS.

Modifications:
- Socket#setTrafficClass and Socket#getTrafficClass should try to use IPv6 first, and check if the error code indicates the protocol is not supported before trying IPv4

Result:
Fixes https://github.com/netty/netty/issues/6741.
2017-05-18 11:26:27 -07:00
Norman Maurer
0db2901f4d [maven-release-plugin] prepare for next development iteration 2017-05-11 16:00:55 +02:00
Norman Maurer
f7a19d330c [maven-release-plugin] prepare release netty-4.1.11.Final 2017-05-11 16:00:16 +02:00
Norman Maurer
9e62c79574 Correctly include all modules during build
Motivation:

To ensure the release plugin works correctly we need to ensure all modules are included during build.

Modification:

- Include all modules
- Skip compilation and tests for native code when not supported but still include the module and build the jar

Result:

Build and release works again
2017-05-11 16:57:14 +02:00
Scott Mitchell
3cc4052963 New native transport for kqueue
Motivation:
We currently don't have a native transport which supports kqueue https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2. This can be useful for BSD systems such as MacOS to take advantage of native features, and provide feature parity with the Linux native transport.

Modifications:
- Make a new transport-native-unix-common module with all the java classes and JNI code for generic unix items. This module will build a static library for each unix platform, and included in the dynamic libraries used for JNI (e.g. transport-native-epoll, and eventually kqueue).
- Make a new transport-native-unix-common-tests module where the tests for the transport-native-unix-common module will live. This is so each unix platform can inherit from these test and ensure they pass.
- Add a new transport-native-kqueue module which uses JNI to directly interact with kqueue

Result:
JNI support for kqueue.
Fixes https://github.com/netty/netty/issues/2448
Fixes https://github.com/netty/netty/issues/4231
2017-05-03 09:53:22 -07:00