Commit Graph

526 Commits

Author SHA1 Message Date
Norman Maurer
cd3254df88
Update to new checkstyle plugin (#8777) (#8780)
Motivation:

We need to update to a new checkstyle plugin to allow the usage of lambdas.

Modifications:

- Update to new plugin version.
- Fix checkstyle problems.

Result:

Be able to use checkstyle plugin which supports new Java syntax.
2019-01-25 11:58:42 +01:00
root
cf03ed0478 [maven-release-plugin] prepare for next development iteration 2019-01-21 12:26:44 +00:00
root
37484635cb [maven-release-plugin] prepare release netty-4.1.33.Final 2019-01-21 12:26:12 +00:00
yulianoifa-mobius
1e4481e551 Allowed IP_FREEBIND option for UDP epoll (#8728)
Motivation:

While using Load Balancers or HA support is needed there are cases when UDP channel need to bind to IP Address which is not available on network interfaces locally.

Modification:

Modified EpollDatagramChannelConfig to allow IP_FREEBIND option

Result:

Fixes ##8727.
2019-01-21 07:42:05 +01:00
Dmitriy Dumanskiy
165912365a Clenaup: simplify EpollEventLoop.closeAll() (#8719)
Motivation:

Avoid unnecessary iteration and `ArrayList` allocation.

Modification:

```
for (AbstractEpollChannel channel: channels.values()) {
     array.add(channel);
}
```
replaced with 

`array.addAll(channels.values())`

and

```
Collection<AbstractEpollChannel> array = new ArrayList<AbstractEpollChannel>(channels.size());
array.addAll(channels.values())
```

replaced with:

`AbstractEpollChannel[] localChannels = channels.values().toArray(new AbstractEpollChannel[0]);`

Result:

Simpler code in `EpollEventLoop.closeAll();`
2019-01-16 11:00:25 +01:00
Feri73
5df235c083 Correcting Maven Dependencies (#8622)
Motivation:

Most of the maven modules do not explicitly declare their
dependencies and rely on transitivity, which is not always correct.

Modifications:

For all maven modules, add all of their dependencies to pom.xml

Result:

All of the (essentially non-transitive) depepdencies of the modules are explicitly declared in pom.xml
2018-12-06 09:01:14 +01:00
root
8eb313072e [maven-release-plugin] prepare for next development iteration 2018-11-29 11:15:09 +00:00
root
afcb4a37d3 [maven-release-plugin] prepare release netty-4.1.32.Final 2018-11-29 11:14:20 +00:00
Norman Maurer
d4b1202e62
Add testcase for epollWait(...) with negative timerfd values. (#8447)
Motivation:

https://github.com/netty/netty/issues/8444 reports that there is some issue with negative values passed to timerfd_settime. This test verifies that everything is working as expected.

Modifications:

Add testcase.

Result:

Test to verify expected behaviour.
2018-10-30 19:38:02 +01:00
root
3e7ddb36c7 [maven-release-plugin] prepare for next development iteration 2018-10-29 15:38:51 +00:00
root
9e50739601 [maven-release-plugin] prepare release netty-4.1.31.Final 2018-10-29 15:37:47 +00:00
Johno Crawford
5b3b8db07f epoll_wait produces an EINVAL error since 4.1.30 (#8350)
Motivation:

epoll_wait should work in 4.1.30 like it did in 4.1.29.

Modifications:

Revert Integer.MAX_VALUE back to MAX_SCHEDULED_TIMERFD_NS (999,999,999).
Add unit test.

Result:

epoll_wait will no longer throw EINVAL.
2018-10-12 05:02:41 +02:00
Matteo Merli
3a96e7373b Added option to do busy-wait on epoll (#8267)
Motivation:

Add an option (through a SelectStrategy return code) to have the Netty event loop thread to do busy-wait on the epoll.

The reason for this change is to avoid the context switch cost that comes when the event loop thread is blocked on the epoll_wait() call.

On average, the context switch has a penalty of ~13usec.

This benefits both:

The latency when reading from a socket
Scheduling tasks to be executed on the event loop thread.
The tradeoff, when enabling this feature, is that the event loop thread will be using 100% cpu, even when inactive.

Modification:

Added SelectStrategy option to return BUSY_WAIT
Epoll loop will do a epoll_wait() with no timeout
Use pause instruction to hint to processor that we're in a busy loop
Result:

When enabled, minimizes impact of context switch in the critical path
2018-09-28 22:52:00 +02:00
Roger
6138541033 Avoid repeating the same field and hiding it (#8335)
Motivation

The EpollChannelConfig (same for KQueues) and its subclasses repeatetly declare their own channel field which leads to a 3x repetition for each config instance. Given the fields are protected or package-private it's exposing the code code to "field hiding" bugs.

Modifications

Use the the existing protected channel field from the DefaultChannelConfig class and simply cast it when needed.

Result

Fixes #8331
2018-09-28 17:37:14 +02:00
root
2d7cb47edd [maven-release-plugin] prepare for next development iteration 2018-09-27 19:00:45 +00:00
root
3a9ac829d5 [maven-release-plugin] prepare release netty-4.1.30.Final 2018-09-27 18:56:12 +00:00
Carl Mastrangelo
1dff107de1 Don't re-arm timerfd each epoll_wait (#7816)
Motivation:
The Epoll transport checks to see if there are any scheduled tasks
before entering epoll_wait, and resets the timerfd just before.
This causes an extra syscall to timerfd_settime before doing any
actual work.   When scheduled tasks aren't added frequently, or
tasks are added with later deadlines, this is unnecessary.

Modification:
Check the *deadline* of the peeked task in EpollEventLoop, rather
than the *delay*.  If it hasn't changed since last time, don't
re-arm the timer

Result:
About 2us faster on gRPC RTT 50pct latency benchmarks.

Before (2 runs for 5 minutes, 1 minute of warmup):

```
50.0%ile Latency (in nanos):		64267
90.0%ile Latency (in nanos):		72851
95.0%ile Latency (in nanos):		78903
99.0%ile Latency (in nanos):		92327
99.9%ile Latency (in nanos):		119691
100.0%ile Latency (in nanos):		13347327
QPS:                           14933

50.0%ile Latency (in nanos):		63907
90.0%ile Latency (in nanos):		73055
95.0%ile Latency (in nanos):		79443
99.0%ile Latency (in nanos):		93739
99.9%ile Latency (in nanos):		123583
100.0%ile Latency (in nanos):		14028287
QPS:                           14936
```

After:
```
50.0%ile Latency (in nanos):		62123
90.0%ile Latency (in nanos):		70795
95.0%ile Latency (in nanos):		76895
99.0%ile Latency (in nanos):		90887
99.9%ile Latency (in nanos):		117819
100.0%ile Latency (in nanos):		14126591
QPS:                           15387

50.0%ile Latency (in nanos):		61021
90.0%ile Latency (in nanos):		70311
95.0%ile Latency (in nanos):		76687
99.0%ile Latency (in nanos):		90887
99.9%ile Latency (in nanos):		119527
100.0%ile Latency (in nanos):		6351615
QPS:                           15571
```
2018-09-11 13:38:38 +02:00
Matteo Merli
2a1596a4e9 Allow to configure socket option SO_BUSY_POLL (#8268)
Motivation:

When using Epoll based transport, allow applications to configure SO_BUSY_POLL socket option:

       SO_BUSY_POLL (since Linux 3.11)
              Sets the approximate time in microseconds to busy poll on a
              blocking receive when there is no data.  Increasing this value
              requires CAP_NET_ADMIN.  The default for this option is con‐
              trolled by the /proc/sys/net/core/busy_read file.

              The value in the /proc/sys/net/core/busy_poll file determines
              how long select(2) and poll(2) will busy poll when they oper‐
              ate on sockets with SO_BUSY_POLL set and no events to report
              are found.

              In both cases, busy polling will only be done when the socket
              last received data from a network device that supports this
              option.

              While busy polling may improve latency of some applications,
              care must be taken when using it since this will increase both
              CPU utilization and power usage.

Modification:

Added SO_BUSY_POLL socket option
Result:

Able to configure SO_BUSY_POLL from Netty
2018-09-07 20:50:51 +02:00
Norman Maurer
b73f785631
We should call the UnLoad methods when we detect an error during calling OnLoad (#8237)
Motivation:

We should ensure we call *UnLoad when we detect an error during calling *OnLoad and previous *OnLoad calls were succesfull.

Modifications:

Correctly call *UnLoad when needed.

Result:

More correct code and no leaks when an error happens during loading the native lib.
2018-08-30 06:56:42 +02:00
Norman Maurer
54f565ac67
Allow to use native transports when sun.misc.Unsafe is not present on… (#8231)
* Allow to use native transports when sun.misc.Unsafe is not present on the system

Motivation:

We should be able to use the native transports (epoll / kqueue) even when sun.misc.Unsafe is not present on the system. This is especially important as Java11 will be released soon and does not allow access to it by default.

Modifications:

- Correctly disable usage of sun.misc.Unsafe when -PnoUnsafe is used while running the build
- Correctly increment metric when UnpooledDirectByteBuf is allocated. This was uncovered once -PnoUnsafe usage was fixed.
- Implement fallbacks in all our native transport code for when sun.misc.Unsafe is not present.

Result:

Fixes https://github.com/netty/netty/issues/8229.
2018-08-29 19:36:33 +02:00
root
a580dc7585 [maven-release-plugin] prepare for next development iteration 2018-08-24 06:36:33 +00:00
root
3fc789e83f [maven-release-plugin] prepare release netty-4.1.29.Final 2018-08-24 06:36:06 +00:00
Norman Maurer
ea4c315b45
Ensure multiple shaded version of the same netty artifact can be loaded as long as the shaded prefix is different (#8207)
Motivation:

We should support to load multiple shaded versions of the same netty artifact as netty is often used in multiple dependencies.

This is related to https://github.com/netty/netty/issues/7272.

Modifications:

- Use -fvisibility=hidden when compiling and use JNIEXPORT for things we really want to have exported
- Ensure fields are declared as static so these are not exported
- Adjust testsuite-shading to use install_name_tool on MacOS to change the id of the lib. Otherwise the wrong may be used.

Result:

Be able to use multiple shaded versions of the same netty artifact.
2018-08-21 07:53:45 +02:00
Ziyan Mo
785473788f (Nio|Epoll)EventLoop.pendingTasks does not need to dispatch to the EventLoop (#8197)
Motivation:

EventLoop.pendingTasks should be (reasonably) cheap to invoke so it can be used within observability. 

Modifications:

Remove code that dispatch access to the internal taskqueue to the EventLoop when invoked as this is not needed anymore with the current MPSC queues we are using. 

See https://github.com/netty/netty/issues/8196#issuecomment-413653286.

Result:

Fixes https://github.com/netty/netty/issues/8196
2018-08-18 07:28:31 +02:00
Nick Hill
630c82717d Lazy initialize NativeDatagramPacketArray and IovArray in EpollEventLoop (#8160)
Motivation:

Avoid unnecessary native memory allocation if UDP / TCP isn't being
used.

Modifications:

Create the reused NativeDatagramPacketArray and IovArray upon first use
instead of EpollEventLoop construction.

Also correct related comment in NativeDatagramPacketArray.

Result:

Reduced native memory use when using epoll in many cases
2018-07-29 18:22:27 +08:00
root
fcb19cb589 [maven-release-plugin] prepare for next development iteration 2018-07-27 04:59:28 +00:00
root
ff785fbe39 [maven-release-plugin] prepare release netty-4.1.28.Final 2018-07-27 04:59:06 +00:00
root
b4dbdc2036 [maven-release-plugin] prepare for next development iteration 2018-07-11 15:37:40 +00:00
root
1c16519ac8 [maven-release-plugin] prepare release netty-4.1.27.Final 2018-07-11 15:37:21 +00:00
root
7bb9e7eafe [maven-release-plugin] prepare for next development iteration 2018-07-10 05:21:24 +00:00
root
8ca5421bd2 [maven-release-plugin] prepare release netty-4.1.26.Final 2018-07-10 05:18:13 +00:00
Norman Maurer
05e5ab1ecb Store NativeDatagramPacketArray directly in the EpollEventLoop
Motivation:

We can store the NativeDatagramPacketArray directly in the EpollEventLoop. This removes the need of using FastThreadLocal.

Modifications:

- Store NativeDatagramPacketArray directly in the EpollEventLoop (just as we do with IovArray as well).

Result:

Less FastThreadLocal usage and more consistent code.
2018-06-27 20:52:28 +02:00
Scott Mitchell
12f6500a4f Epoll and Kqueue shouldn't read by default (#8024)
Motivation:
Epoll and Kqueue channels have internal state which forces
a single read operation after channel construction. This
violates the Channel#read() interface which indicates that
data shouldn't be delivered until this method is called.
The behavior is also inconsistent with the NIO transport.

Modifications:
- Epoll and Kqueue shouldn't unconditionally read upon
initialization, and instead should rely upon Channel#read()
or auto_read.

Result:
Epoll and Kqueue are more consistent with NIO.
2018-06-15 10:28:50 +02:00
Norman Maurer
d133bf06a4
Allow to schedule tasks up to Long.MAX_VALUE (#7972)
Motivation:

We should allow to schedule tasks with a delay up to Long.MAX_VALUE as we did pre 4.1.25.Final.

Modifications:

Just ensure we not overflow and put the correct max limits in place when schedule a timer. At worse we will get a wakeup to early and then schedule a new timeout.

Result:

Fixes https://github.com/netty/netty/issues/7970.
2018-05-30 11:11:42 +02:00
Norman Maurer
030318e53c
Read until all data is consumed when EOF is detected even if readPend… (#7961)
* Read until all data is consumed when EOF is detected even if readPending is false and auto-read is disabled.

Motivation:

We should better always notify the user of EOF even if the user did not request any data as otherwise we may never be notified when the remote peer closes the connection. This should be ok as the amount of extra data we may read and so fire through the pipeline is limited by SO_RECVBUF.

Modifications:

- Always drain the socket when EOF is detected.
- Add testcase

Result:

No risk for the user to be not notified of EOF.
2018-05-24 20:29:29 +02:00
Norman Maurer
64bb279f47 [maven-release-plugin] prepare for next development iteration 2018-05-14 11:11:45 +00:00
Norman Maurer
c67a3b0507 [maven-release-plugin] prepare release netty-4.1.25.Final 2018-05-14 11:11:24 +00:00
Norman Maurer
358249e5c9
Allow to disable native transport and native ssl support via system property. (#7903)
Motivation:

Sometimes it's useful to disable native transports / native ssl to debug a problem. We should allow to do so with a system property so people not need to adjust code for this.

Modifications:

Add system properties which allow to disable native transport and native ssl.

Result:

Easier to disable native code usage without code changes.
2018-05-04 14:44:44 +02:00
Devrim Şahin
b818852cdb Update DatagramPacket.recipient() to return the actual destination IP (#7879)
Motivation:

DatagramPacket.recipient() doesn't return the actual destination IP, but the IP the app is bound to.

Modification:

- IP_RECVORIGDSTADDR option is enabled for UDP sockets, which allows retrieval of ancillary information containing the original recipient.
- _recvFrom(...) function from transport-native-unix-common/src/main/c/netty_unix_socket.c is modified such that if IP_RECVORIGDSTADDR is set, recvmsg is used instead of recvfrom; enabling the retrieval of the original recipient.
- DatagramSocketAddress also contains a 'local' address, representing the recipient.
- EpollDatagramChannel is updated to return the retrieved recipient address instead of the address the channel is bound to.

Result:

Fixes #4950.
2018-04-26 08:00:36 +02:00
Norman Maurer
b47fb81799
EventLoop.schedule with big delay fails (#7402)
Motivation:

Using a very huge delay when calling schedule(...) may cause an Selector error when calling select(...) later on. We should gaurd against such a big value.

Modifications:

- Add guard against a very huge value.
- Added tests.

Result:

Fixes [#7365]
2018-04-24 11:15:20 +02:00
Norman Maurer
b75f44db9a [maven-release-plugin] prepare for next development iteration 2018-04-19 11:56:07 +00:00
Norman Maurer
04fac00c8c [maven-release-plugin] prepare release netty-4.1.24.Final 2018-04-19 11:55:47 +00:00
Devrim Şahin
0b690a991f Added IP_TRANSPARENT option for UDP (#7872)
Motivation:

This allows netty to operate in 'transparent proxy' mode for UDP, intercepting connections
to other addresses by means of Linux firewalling rules, as per

https://www.kernel.org/doc/Documentation/networking/tproxy.txt


Modification:

Add IP_TRANSPARENT option.

Result:

Allows setting and getting of the IP_TRANSPARENT option, which allows retrieval of the ultimate socket address originally requested.
2018-04-17 09:07:02 +02:00
root
0a61f055f5 [maven-release-plugin] prepare for next development iteration 2018-04-04 10:44:46 +00:00
root
8c549bad38 [maven-release-plugin] prepare release netty-4.1.23.Final 2018-04-04 10:44:15 +00:00
Scott Mitchell
ed0668384b NIO read spin event loop spin when half closed (#7801)
Motivation:
AbstractNioByteChannel will detect that the remote end of the socket has
been closed and propagate a user event through the pipeline. However if
the user has auto read on, or calls read again, we may propagate the
same user events again. If the underlying transport continuously
notifies us that there is read activity this will happen in a spin loop
which consumes unnecessary CPU.

Modifications:
- AbstractNioByteChannel's unsafe read() should check if the input side
of the socket has been shutdown before processing the event. This is
consistent with EPOLL and KQUEUE transports.
- add unit test with @normanmaurer's help, and make transports consistent with respect to user events

Result:
No more read spin loop in NIO when the channel is half closed.
2018-03-28 20:02:57 +02:00
Norman Maurer
0a8e1aaf19 Flush task should not flush messages that were written since last flush attempt.
Motivation:

The flush task is currently using flush() which will have the affect of have the flush traverse the whole ChannelPipeline and also flush messages that were written since we gave up flushing. This is not really correct as we should only continue to flush messages that were flushed at the point in time when the flush task was submitted for execution if the user not explicit call flush() by him/herself.

Modification:

Call *Unsafe.flush0() via the flush task which will only continue flushing messages that were marked as flushed before.

Result:

More correct behaviour when the flush task is used.
2018-03-02 10:09:40 +09:00
Norman Maurer
69582c0b6c [maven-release-plugin] prepare for next development iteration 2018-02-21 12:52:33 +00:00
Norman Maurer
786f35c6c9 [maven-release-plugin] prepare release netty-4.1.22.Final 2018-02-21 12:52:19 +00:00
Scott Mitchell
ce241bd11e Epoll flush/writabilityChange deadlock
Motivation:
b215794de3 recently introduced a change in behavior where writeSpinCount provided a limit for how many write operations were attempted per flush operation. However when the write quantum was meet the selector write flag was not cleared, and the channel unsafe flush0 method has an optimization which prematurely exits if the write flag is set. This may lead to no write progress being made under the following scenario:
- flush is called, but the socket can't accept all data, we set the write flag
- the selector wakes us up because the socket is writable, we write data and use the writeSpinCount quantum
- we then schedule a flush() on the EventLoop to execute later, however it the flush0 optimization prematurely exits because the write flag is still set

In this scenario the socket is still writable so the EventLoop may never notify us that the socket is writable, and therefore we may never attempt to flush data to the OS.

Modifications:
- When the writeSpinCount quantum is exceeded we should clear the selector write flag

Result:
Fixes https://github.com/netty/netty/issues/7729
2018-02-20 11:40:58 +01:00
Norman Maurer
e71fa1e7b6 [maven-release-plugin] prepare for next development iteration 2018-02-05 12:02:35 +00:00
Norman Maurer
41ebb5fcca [maven-release-plugin] prepare release netty-4.1.21.Final 2018-02-05 12:02:19 +00:00
Norman Maurer
ea58dc7ac7 [maven-release-plugin] prepare for next development iteration 2018-01-21 12:53:51 +00:00
Norman Maurer
96c7132dee [maven-release-plugin] prepare release netty-4.1.20.Final 2018-01-21 12:53:34 +00:00
Scott Mitchell
33ddb83dc1
IovArray#add return value resulted in more ByteBufs being added during iteration
Motivation:
IovArray implements MessageProcessor, and the processMessage method will continue to be called during iteration until it returns true. A recent commit b215794de3 changed the return value to only return true if any component of a CompositeByteBuf was added as a result of the method call. However this results in the iteration continuing, and potentially subsequent smaller buffers maybe added, which will result in out of order writes and generally corrupts data.

Modifications:
- IovArray#add should return false so that the MessageProcessor#processMessage will stop iterating.

Result:
Native transports which use IovArray will not corrupt data during gathering writes of CompositeByteBuf objects.
2018-01-04 08:04:32 -08:00
Norman Maurer
264a5daa41 [maven-release-plugin] prepare for next development iteration 2017-12-15 13:10:54 +00:00
Norman Maurer
0786c4c8d9 [maven-release-plugin] prepare release netty-4.1.19.Final 2017-12-15 13:09:30 +00:00
Scott Mitchell
af2f343648
FileDescriptor writev core dump
Motivation:
FileDescriptor#writev calls JNI code, and that JNI code dereferences a NULL pointer which crashes the application. This occurs when writing a single CompositeByteBuf object with more than one component.

Modifications:
- Initialize the iovec iterator properly to avoid the core dump
- Fix the array length calculation if we aren't able to fit all the ByteBuffer objects in the iovec array

Result:
No more core dump.
2017-12-14 16:47:31 -08:00
Dmitriy Dumanskiy
8b3e98163d Added Automatic-Module-Name for native modules to support Java 9 modules
Motivation:

Latest netty missing predefined Automatic-Module-Name module name entry in MANIFEST.MF for native transports.

Modification:

Added Automatic-Module-Name entry to manifest during native transports build.

Fixes [#7501]
2017-12-14 19:41:45 +00:00
Norman Maurer
b2bc6407ab [maven-release-plugin] prepare for next development iteration 2017-12-08 09:26:15 +00:00
Norman Maurer
96732f47d8 [maven-release-plugin] prepare release netty-4.1.18.Final 2017-12-08 09:25:56 +00:00
Scott Mitchell
b215794de3
Enforce writeSpinCount to limit resource consumption per socket (#7478)
Motivation:
The writeSpinCount currently loops over the same buffer, gathering
write, file write, or other write operation multiple times but will
continue writing until there is nothing left or the OS doesn't accept
any data for that specific write. However if the OS keeps accepting
writes there is no way to limit how much time we spend on a specific
socket. This can lead to unfair consumption of resources dedicated to a
single socket.
We currently don't limit the amount of bytes we attempt to write per
gathering write. If there are many more bytes pending relative to the
SO_SNDBUF size we will end up building iov arrays with more elements
than can be written, which results in extra iteration, conditionals,
and book keeping.

Modifications:
- writeSpinCount should limit the number of system calls we make to
write data, instead of applying to individual write operations
- IovArray should support a maximum number of bytes
- IovArray should support composite buffers of greater than size 1024
- We should auto-scale the amount of data that we attempt to write per
gathering write operation relative to SO_SNDBUF and how much data is
successfully written
- The non-unsafe path should also support a maximum number of bytes,
and respect the IOV_MAX limit

Result:
Write resource consumption can be bounded and gathering writes have
a limit relative to the amount of data which can actually be accepted
by the socket.
2017-12-07 16:00:52 -08:00
Norman Maurer
3f101caa4c Not call java methods from within JNI init code to prevent class loading deadlocks.
Motivation:

We used NetUtil.isIpV4StackPreferred() when loading JNI code which tries to load NetworkInterface in its static initializer. Unfortunally a lock on the NetworkInterface class init may be already hold somewhere else which may cause a loader deadlock.

Modifications:

Add a new Socket.initialize() method that will be called when init the library and pass everything needed to the JNI level so we not need to call back to java.

Result:

Fixes [#7458].
2017-12-06 14:34:15 +01:00
Norman Maurer
251bb1a739 Not use safeRelease(...) but release(...) to release non-readable holders to ensure we not mask errors.
Motivation:

AbstractChannel attempts to "filter" messages which are written [1]. A goal of this process is to copy from heap to direct if necessary. However implementations of this method [2][3] may translate a buffer with 0 readable bytes to EMPTY_BUFFER. This may mask a user error where an empty buffer is written but already released.

Modifications:

Replace safeRelease(...) with release(...) to ensure we propagate reference count issues.

Result:

Fixes [#7383]
2017-12-04 20:38:35 +01:00
Tomasz Jędrzejewski
e8540c2b7a Adding stable JDK9 module names that follow reverse-DNS style
Automatic-Module-Name entry provides a stable JDK9 module name, when Netty is used in a modular JDK9 applications. More info: http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html

When Netty migrates to JDK9 in the future, the entry can be replaced by actual module-info descriptor.

Modification:

The POM-s are configured to put the correct module names to the manifest.

Result:

Fixes #7218.
2017-11-29 11:50:24 +01:00
Norman Maurer
b1cc5835ac Not directly call getsockopt but use exported helper function
Motivation:

To better isolate OS system calls we should not call getsockopt directly but use our netty_unix_socket_getOption0 function. See is a followup of f115bf5.

Modifications:

Export netty_unix_socket_getOption0 by declaring it in the header file and use it

Result:

Better isolation of system calls.
2017-11-22 08:40:44 +01:00
Norman Maurer
f115bf50cb EpollSocketChannelConfig.getOptions() must not throw if TCP_FASTOPEN_CONNECT is not supported
Motivation:

If a user calls EpollSocketChannelConfig.getOptions() and TCP_FASTOPEN_CONNECT is not supported we throw an exception.

Modifications:

- Just return 0 if ENOPROTOOPT is set.
- Add testcase

Result:

getOptions() works as epxected.
2017-11-16 14:07:54 +01:00
Norman Maurer
188ea59c9d [maven-release-plugin] prepare for next development iteration 2017-11-08 22:36:53 +00:00
Norman Maurer
812354cf1f [maven-release-plugin] prepare release netty-4.1.17.Final 2017-11-08 22:36:33 +00:00
Norman Maurer
e7f02b1dc0 Set readPending to false when EOF is detected while issue an read
Motivation:

We need to set readPending to false when we detect a EOF while issue a read as otherwise we may not unregister from the Selector / Epoll / KQueue and so keep on receving wakeups.

The important bit is that we may even get a wakeup for a read event but will still will only be able to read 0 bytes from the socket, so we need to be very careful when we clear the readPending. This can happen because we generally using edge-triggered mode for our native transports and because of the nature of edge-triggered we may schedule an read event just to find out there is nothing left to read atm (because we completely drained the socket on the previous read).

Modifications:

Set readPending to false when EOF is detected.

Result:

Fixes [#7255].
2017-11-06 15:44:36 -08:00
Norman Maurer
bcad9dbf97 Revert "Set readPending to false when ever a read is done"
This reverts commit 413c7c2cd8 as it introduced an regression when edge-triggered mode is used which is true for our native transports by default. With 413c7c2cd8 included it was possible that we set readPending to false by mistake even if we would be interested in read more.
2017-11-06 09:21:42 -08:00
Janusz Dziemidowicz
cdb2a27857 Add TCP_FASTOPEN_CONNECT epoll option
Motivation:

Linux kernel 4.11 introduced a new socket option,
TCP_FASTOPEN_CONNECT, that greatly simplifies making TCP Fast Open
connections on client side. Usually simply setting the flag before
connect() call is enough, no more changes are required.

Details can be found in kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=19f6d3f3

Modifications:

TCP_FASTOPEN_CONNECT socket option was added to EpollChannelOption
class.

Result:

Netty clients can easily make TCP Fast Open connections. Simply
calling option(EpollChannelOption.TCP_FASTOPEN_CONNECT, true) in
client bootstrap is enough (given recent enough kernel).
2017-10-29 13:42:15 +01:00
Scott Mitchell
413c7c2cd8 Set readPending to false when ever a read is done
Motivation:
readPending is currently only set to false if data is delivered to the application, however this may result in duplicate events being received from the selector in the event that the socket was closed.

Modifications:
- We should set readPending to false before each read attempt for all
transports besides NIO.
- Based upon the Javadocs it is possible that NIO may have spurious
wakeups [1]. In this case we should be more cautious and only set
readPending to false if data was actually read.

[1] https://docs.oracle.com/javase/7/docs/api/java/nio/channels/SelectionKey.html
That a selection key's ready set indicates that its channel is ready for some operation category is a hint, but not a guarantee, that an operation in such a category may be performed by a thread without causing the thread to block.

Result:
Notification from the selector (or simulated events from kqueue/epoll ET) in the event of socket closure.
Fixes https://github.com/netty/netty/issues/7255
2017-10-25 08:25:54 -07:00
Idel Pivnitskiy
50a067a8f7 Make methods 'static' where it possible
Motivation:

Even if it's a super micro-optimization (most JVM could optimize such
 cases in runtime), in theory (and according to some perf tests) it
 may help a bit. It also makes a code more clear and allows you to
 access such methods in the test scope directly, without instance of
 the class.

Modifications:

Add 'static' modifier for all methods, where it possible. Mostly in
test scope.

Result:

Cleaner code with proper 'static' modifiers.
2017-10-21 14:59:26 +02:00
Carl Mastrangelo
d3ca087f6b Propagate all exceptions when loading native code
Motivation:
There are 2 motivations, the first depends on the second:

Loading Netty Epoll statically stopped working in 4.1.16, due to
`Native` always loading the arch specific shared object.  In a
static binary, there is no arch specific SO.

Second, there are a ton of exceptions that can happen when loading
a native library.  When loading native code, Netty tries a bunch of
different paths but a failure in any given may not be fatal.

Additionally: turning on debug logging is not always feasible so
exceptions get silently swallowed.

Modifications:

* Change Epoll and Kqueue to try the static load second
* Modify NativeLibraryLoader to record all the locations where
  exceptions occur.
* Attempt to use `addSuppressed` from Java 7 if available.

Alternatives Considered:

An alternative would be to record log messages at each failure.  If
all load attempts fail, the log messages are printed as warning,
else as debug. The problem with this is there is no `LogRecord` to
create like in java.util.logging.  Buffering the args to
logger.log() at the end of the method loses the call site, and
changes the order of events to be confusing.

Another alternative is to teach NativeLibraryLoader about loading
the SO first, and then the static version.  This would consolidate
the code fore Epoll, Kqueue, and TCNative.   I think this is the
long term better option, but this PR is changing a lot already.
Someone else can take a crack at it later

Results:
Epoll Still Loads and easier debugging.
2017-10-04 08:45:27 +02:00
Norman Maurer
625a7426cd [maven-release-plugin] prepare for next development iteration 2017-09-25 06:12:32 +02:00
Norman Maurer
f57d8f00e1 [maven-release-plugin] prepare release netty-4.1.16.Final 2017-09-25 06:12:16 +02:00
Norman Maurer
9d5daa37cc Check for latest centos release version when doing a release
Motivation:

We used to check for version 6.8 but the latest is 6.9

Modifications:

Update version to 6.9 in the check.

Result:

Be able to cut a release on latest centos version
2017-09-21 12:23:37 -07:00
Norman Maurer
aa8bdb5d6b Fix assertion error when closing / shutdown native channel and SO_LINGER is set.
Motivation:

When SO_LINGER is used we run doClose() on the GlobalEventExecutor by default so we need to ensure we schedule all code that needs to be run on the EventLoop on the EventLoop in doClose. Beside this there are also threading issues when calling shutdownOutput(...)

Modifications:

- Schedule removal from EventLoop to the EventLoop
- Correctly handle shutdownOutput and shutdown in respect with threading-model
- Add unit tests

Result:

Fixes [#7159].
2017-09-18 14:46:37 -07:00
Norman Maurer
426938307d Use the correct osname in the Bundle-NativeCode declaration.
Motivation:

We need to ensure we use the correct osname in the Bundle-NativeCode declaration as declared in:

https://www.osgi.org/developer/specifications/reference/

Modifications:

Update osname to match the spec.

Result:

Correct Bundle-NativeCode entry in the MANIFEST
2017-09-14 08:26:49 -07:00
Norman Maurer
0fffc844d6 Only load native transport if running architecture match the compiled library architecture.
Motivation:

We should only try to load the native artifacts if the architecture we are currently running on is the same as the one the native libraries were compiled for.

Modifications:

Include architecture in native lib name and append the current arch when trying to load these. This will fail then if its not the same as the arch of the compiled arch.

Result:

Fixes [#7150].
2017-09-04 13:34:55 +02:00
Carl Mastrangelo
c891c9c13f Include more detail why Unsafe is not available
Motivation:
PD and PD0 Both try to find and use Unsafe.  If unavailable, they
try to log why and continue on.  However, it is not always east to
enable this logging.  Chaining exceptions together is much easier
to reach, and the original exception is relevant when Unsafe is
needed.

Modifications:
* Make PD log why PD0 could not be loaded with a trace level log
* Make PD0 remember why Unsafe wasn't available
* Expose unavailability cause through PD for higher level use.
* Make Epoll and KQueue include the reason when failing

Result:
Easier debugging in hard to reconfigure environments
2017-08-29 22:02:06 +02:00
Scott Mitchell
89ecb4b4a4 AutoClose behavior may infinite loop
Motivation:
If AutoClose is false and there is a IoException then AbstractChannel will not close the channel but instead just fail flushed element in the ChannelOutboundBuffer. AbstractChannel also notifies of writability changes, which may lead to an infinite loop if the peer has closed its read side of the socket because we will keep accepting more data but continuously fail because the peer isn't accepting writes.

Modifications:
- If the transport throws on a write we should acknowledge that the output side of the channel has been shutdown and cleanup. If the channel can't accept more data because it is full, and still healthy it is not expected to throw. However if the channel is not healthy it will throw and is not expected to accept any more writes. In this case we should shutdown the output for Channels that support this feature and otherwise just close.
- Connection-less protocols like UDP can remain the same because the channel may disconnected temporarily.
- Make sure AbstractUnsafe#shutdownOutput is called because the shutdown on the socket may throw an exception.

Result:
More correct handling of write failure when AutoClose is false.
2017-08-25 21:01:41 -07:00
Norman Maurer
b967805f32 [maven-release-plugin] prepare for next development iteration 2017-08-24 15:38:22 +02:00
Norman Maurer
da8e010a42 [maven-release-plugin] prepare release netty-4.1.15.Final 2017-08-24 15:37:59 +02:00
Carl Mastrangelo
7f1051b6ca Include JNIEXPORT on exported symbols
Motivation:
As noticed in https://stackoverflow.com/questions/45700277/
compilation can fail if the definition of a method doesn't
match the declaration.  It's easy enough to add this in, and make
it easy to compile.

Modifications:
Add JNIEXPORT to the entry points.

* On Windows this adds: `__declspec(dllexport)`
* On Mac this adds: `__attribute__((visibility("default")))`
* On Linux (GCC 4.2+) this adds: ` __attribute__((visibility("default")))`
* On other it doesn't add anything.

Result:
Easier compilation
2017-08-18 17:34:48 -07:00
Scott Mitchell
fe2dd973e9 Unify KQueue and Epoll wait timeout approach
Motivation:
KQueueEventLoop and EpollEventLoop implement different approaches to applying a timeout of their respective poll calls. Epoll attempts to ensure the desired timeout is satisfied at the java layer and at the JNI layer, but it should be sufficient to account for spurious wakups at the JNI layer. Epoll timeout granularity is also limited to milliseconds which may be too large for some latency sensitive applications.

Modifications:
- Make EpollEventLoop wait method look like KQueueEventLoop
- Epoll should support a finer timeout granularity via timerfd_create. We can hide most of these details behind the epollWait0 JNI call to avoid crossing additional JNI boundaries.

Result:
More consistent timeout approach between KQueue and Epoll.
2017-08-18 13:09:02 -07:00
Scott Mitchell
1d7c3fb7ee KQueue detect peer close without EVFILT_READ
Motivation:
The EPOLL transport uses EPOLLRDHUP to detect when the peer closes the write side of the socket. Currently KQueue is not able to mimic this behavior and the only way to detect if the peer has closed is to read. It may not always be appropriate to read for backpressure and other reasons at the application level.

Modifications:
- Support EVFILT_SOCK filter which provides notification when the peer closes the socket

Result:
KQueue transport has more consistent behavior with Epoll transport for detecting peer closure.
2017-08-18 11:00:18 -07:00
Carl Mastrangelo
e4af881bdb Do not define JNI_OnLoad when not dynamic
Motivation:
Due to an oversight (by myself), linking two JNI modules with
duplicate symbols fails in linking.  This only seems to happen
some of the time (the behavior seems to be different between GCC
and Clang toolchains).  For instance, including both netty tcnative
and netty epoll fails to link because of duplicate JNI_OnLoad
symobols.

Modification:
Do not define the JNI_OnLoad and JNI_OnUnload symbols when
compiling for static linkage, as indicated by the NETTY_BUILD_STATIC
preprocessor define.  They are never directly called when
statically linked.

Result:
Able to statically compile epoll and tcnative code into a single
binary.
2017-08-18 09:20:58 +02:00
Norman Maurer
19dcb15062 Use underscore in native library names for consistency.
Motivation:

At the moment we try to load the library using multiple names which includes names using - but also _ . We should just use _ all the time.

Modifications:

Replace - with _

Result:

Fixes [#7069]
2017-08-15 06:02:00 +02:00
Scott Mitchell
237a4da1b7 Shutting down the outbound side of the channel should not accept future writes
Motivation:
Implementations of DuplexChannel delegate the shutdownOutput to the underlying transport, but do not take any action on the ChannelOutboundBuffer. In the event of a write failure due to the underlying transport failing and application may attempt to shutdown the output and allow the read side the transport to finish and detect the close. However this may result in an issue where writes are failed, this generates a writability change, we continue to write more data, and this may lead to another writability change, and this loop may continue. Shutting down the output should fail all pending writes and not allow any future writes to avoid this scenario.

Modifications:
- Implementations of DuplexChannel should null out the ChannelOutboundBuffer and fail all pending writes

Result:
More controlled sequencing for shutting down the output side of a channel.
2017-08-04 10:59:57 -07:00
Norman Maurer
4bb89dcc54 Correctly handle connect/disconnect in EpollDatagramChannel / KQueueDatagramChannel
Motivation:

We did not correctly handle connect() and disconnect() in EpollDatagramChannel / KQueueDatagramChannel and so the behavior was different compared to NioDatagramChannel.

Modifications:

- Correct implement connect and disconnect methods
- Share connect and related code
- Add tests

Result:

EpollDatagramChannel / KQueueDatagramChannel also supports correctly connect() and disconnect() methods.
2017-08-04 09:22:53 +02:00
Norman Maurer
52f384b37f [maven-release-plugin] prepare for next development iteration 2017-08-02 12:55:10 +00:00
Norman Maurer
8cc1071881 [maven-release-plugin] prepare release netty-4.1.14.Final 2017-08-02 12:54:51 +00:00
Alberto K
21b7ab1f25 Allows IP_TRANSPARENT to be set on a redirecting socket
Motivation:

IP_TRANSPARENT support is not complete, the option can currently only be set on EpollServerSocket. Setting the option on an EpollSocket is also requires so as to be able to bind a socket to a non-local address as described in ip(7)
http://man7.org/linux/man-pages/man7/ip.7.html

"TProxy redirection with the iptables TPROXY target also
requires that this option be set on the redirected socket."

Modifications:

Added IP_TRANSPARENT socket option to EpollSocketChannelConfig

Result:

A redirecting socket can be created with a non-local IP address as required for TPROXY
2017-07-30 19:51:55 +02:00
Norman Maurer
efb2d141c1 Correctly handle unsigned int values returned from TCP_INFO
Motivation:

We used an int[] to store all values that are returned in the struct for TCP_INFO which is not good enough as it uses usigned int values.

Modifications:

- Change int[] to long[] and correctly cast values.

Result:

No more truncated values.
2017-07-25 16:18:52 +02:00
Carl Mastrangelo
d8f4547f5c Unify {Epoll,KQueue}EventLoopGroup initialization.
Motivation:
`Epoll.ensureAvailability()` is called multiple times, once in
static initialization and in a couple of the constructors.  This is
redundant and confusing to read.

Modifications:
Move `Epoll.ensureAvailability()` call into an instance initializer
and remove all other references.  This ensures that every EELG
checks availability, while still delaying the check until
construction.  This pattern is used when there are multiple ctors,
as in this class.

Result:
Easier to read code.
2017-07-24 20:14:54 +02:00
Norman Maurer
3cdff36821 Update tests to not use TestUtils.getFreePort() and so ensure we not try to use a port that is used by the system in the meantime.
Motivation:

We should not try to detect a free port in tests put just use 0 when bind so there is no race in which the system my bind something to the port we choosen before.

Modifications:

- Remove the usage of TestUtils.getFreePort() in the testsuite
- Remove hack to workaround bind errors which will not happen anymore now

Result:

Less flacky tests.
2017-07-20 08:25:37 +02:00
Norman Maurer
e6a399a778 Explicit specify hostaddress during tests to ensure testsuite pass on docker (mac)
Motivation:

When run the current testsuite on docker (mac) it will fail a few tests with:

io.netty.channel.AbstractChannel$AnnotatedConnectException: connect(..) failed: Cannot assign requested address: /0:0:0:0:0:0:0:0%0:46607
Caused by: java.net.ConnectException: connect(..) failed: Cannot assign requested address

Modifications:

Specify host explicit as done in other tests to only use ipv6 when really supported.

Result:

Build pass on docker as well
2017-07-18 07:24:04 +02:00
Scott Mitchell
7cfe416182 Use unbounded queues from JCTools 2.0.2
Motivation:
JCTools 2.0.2 provides an unbounded MPSC linked queue. Before we shaded JCTools we had our own unbounded MPSC linked queue and used it in various places but gave this up because there was no public equivalent available in JCTools at the time.

Modifications:
- Use JCTool's MPSC linked queue when no upper bound is specified

Result:
Fixes https://github.com/netty/netty/issues/5951
2017-07-10 12:32:15 -07:00
Norman Maurer
2a376eeb1b [maven-release-plugin] prepare for next development iteration 2017-07-06 13:24:06 +02:00
Norman Maurer
c7f8168324 [maven-release-plugin] prepare release netty-4.1.13.Final 2017-07-06 13:23:51 +02:00
Nikolay Fedorovskikh
016e00bf7a Clarify the appointment of the intermediate collection
Motivation:

An intermediate list is creating in the `EpollEventLoop#closeAll` to prevent ConcurrentModificationException. But this is not the obvious purpose has no comment.

Modifications:

Add comment to clarify the appointment of the intermediate collection.

Result:

More clear code.
2017-07-05 20:44:11 -04:00
Norman Maurer
c318fc7cea Remove not needed intermediate collection while reading DatagramPackets in native transports
Motivation:

We used an intermediate collection to store the read DatagramPackets and only fired these through the pipeline once wewere done with the reading loop. This is not needed and can also increase memory usage.

Modifications:

Remove intermediate collection

Result:

Less overhead and possible less memory usage during read loop.
2017-07-05 18:20:05 +02:00
Norman Maurer
773757f2dd Introduce EpollSocketChannelConfig.setTcpKeepCnt as replacement for setTcpKeepCntl.
Motivation:

We had a typo in the method name of the EpollSocketChannelConfig.

Modifications:

Deprecate old method and introduce a new one.

Result:

Fixes [#6909]
2017-06-28 07:43:59 +02:00
Carl Mastrangelo
83de77fbe5 Make Native loading work better with Java 8
Motivation:
Enable static linking for Java 8.  These commits are the same as those introduced to netty tcnative.  The goal is to allow lots of JNI libraries to be statically linked together without having conflict `JNI_OnLoad` methods.

Modification:
* add JNI_OnLoad suffixes to enable static linking
* Add static names to the list of libraries that try to be loaded
* Enable compiling with JNI 1.8
* Sort includes

Result:
Enable statically linked JNI code.
2017-06-23 19:42:13 +02:00
Carl Mastrangelo
7baef4fbe8 Move "fallthrough" statement to where fall actually happens Motivation: Static analysis looks for error prone switch case statements. Accidental fall through is one such case, but it is sometimes intentional. To indicate this, the "//fallthrough" comment can be added before the fall.
The code in question has this comment, but it is *after* the fall
so the static analysis flags it.

This is described in http://errorprone.info/bugpattern/FallThrough

Modifications:
Move fall through comment to where the fall actually occurs

Result:
More compatible with Error Prone tools
2017-06-23 07:22:47 +02:00
Carl Mastrangelo
b985615522 Fix compiler warnings in netty Epoll and unix common
Motivation:
Google requires stricter compilation by adding -Werror and enabling many other warnings.

Modification:

* fix warning caused by -Wmissing-braces

* Use the address of `sendmmsg` rather than the function itself when
checking for presence.  This resovles the warning caused by
`-Wpointer-bool-conversion`.

More detail:
When compiling on Linux, `sendmmsg` is always present, so the
function is always nonnull.  When compiling elsewhere, the
function is defined as `__attribute__((weak))` which means it
may be absent at link time.  This is controlled by
`IO_NETTY_SENDMMSG_NOT_FOUND`, which is off by default.

The reason for the error is due to the risk of accidentally not
calling the function.  By adding `&` before the function, there
is no ambiguity.  (the result of the fn call cannot have its
address taken.)

* use != to check for sendmmsg

Result:
Easier compilation.
2017-06-23 07:19:46 +02:00
Rogan Dawes
051e0ad4be Add support for IP_TRANSPARENT socket option
Motivation:

This allows netty to operate in 'transparent proxy' mode, intercepting connections
to other addresses by means of Linux firewalling rules, as per

https://www.kernel.org/doc/Documentation/networking/tproxy.txt

The original destination address can be obtained by referencing
ch.localAddress().

Modification:

Add methods similar to those for ipFreeBind, to set the IP_TRANSPARENT option.

Result:

Allows setting and getting of the IP_TRANSPARENT option, which allows retrieval of the ultimate socket address originally requested.
2017-06-12 08:03:39 +02:00
Norman Maurer
b6c27b9f6b Not force to run autoconf and compile multiple times
Motivation:

We should not force autoconf and compile as this will result in multiple executions and so slow down the build.

Modifications:

Remove force declarations

Result:

Faster build of native modules
2017-06-09 20:34:28 +02:00
Norman Maurer
fd67a2354d [maven-release-plugin] prepare for next development iteration 2017-06-08 21:06:24 +02:00
Norman Maurer
3acd5c68ea [maven-release-plugin] prepare release netty-4.1.12.Final 2017-06-08 21:06:01 +02:00
louxiu
3c4dfed08a Fix handle of ByteBuf with multi nioBuffer in EpollDatagramChannel and KQueueDatagramChannel
Motivation:
1. special handling of ByteBuf with multi nioBuffer rather than type of CompositeByteBuf (eg. DuplicatedByteBuf with CompositeByteBuf)
2. EpollDatagramUnicastTest and KQueueDatagramUnicastTest passed because CompositeByteBuf is converted to DuplicatedByteBuf before write to channel
3. uninitalized struct msghdr will raise error

Modifications:
1. isBufferCopyNeededForWrite(like isSingleDirectBuffer in NioDatgramChannel) checks wether a new direct buffer is needed
2. special handling of ByteBuf with multi nioBuffer in EpollDatagramChannel, AbstractEpollStreamChannel, KQueueDatagramChannel, AbstractKQueueStreamChannel and IovArray
3. initalize struct msghdr

Result:
handle of ByteBuf with multi nioBuffer in EpollDatagramChannel and KQueueDatagramChannel are ok
2017-05-26 07:56:34 +02:00
Norman Maurer
2f8fe2af01 Only try to deregister from EventLoop when the native Channel was registered before.
Motivation:

We only can call eventLoop() if we are registered on an EventLoop yet. As we just did this without checking we spammed the log with an error that was harmless.

Modifications:

Check if registered on eventLoop before try to deregister on close.

Result:

Fixes [#6770]
2017-05-24 13:19:18 +02:00
Norman Maurer
201d9b6536 Share code that is needed to support shaded native libraries.
Motivation:

For our native libraries in netty we support shading, to have this work on runtime the user needs to set a system property. This code should shared.

Modifications:

Move logic to NativeLbiraryLoader and so share for all native libs.

Result:

Less code duplication and also will work for netty-tcnative out of the box once it support shading
2017-05-19 19:33:21 +02:00
Scott Mitchell
4c6d946fba KQueueSocket#setTrafficClass exceptions
Motivation:
MacOS will throw an error when attempting to set the IP_TOS socket option if IPv6 is available, and also when getting the value for IP_TOS.

Modifications:
- Socket#setTrafficClass and Socket#getTrafficClass should try to use IPv6 first, and check if the error code indicates the protocol is not supported before trying IPv4

Result:
Fixes https://github.com/netty/netty/issues/6741.
2017-05-18 11:26:27 -07:00
Norman Maurer
0db2901f4d [maven-release-plugin] prepare for next development iteration 2017-05-11 16:00:55 +02:00
Norman Maurer
f7a19d330c [maven-release-plugin] prepare release netty-4.1.11.Final 2017-05-11 16:00:16 +02:00
Norman Maurer
9e62c79574 Correctly include all modules during build
Motivation:

To ensure the release plugin works correctly we need to ensure all modules are included during build.

Modification:

- Include all modules
- Skip compilation and tests for native code when not supported but still include the module and build the jar

Result:

Build and release works again
2017-05-11 16:57:14 +02:00
Scott Mitchell
3cc4052963 New native transport for kqueue
Motivation:
We currently don't have a native transport which supports kqueue https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2. This can be useful for BSD systems such as MacOS to take advantage of native features, and provide feature parity with the Linux native transport.

Modifications:
- Make a new transport-native-unix-common module with all the java classes and JNI code for generic unix items. This module will build a static library for each unix platform, and included in the dynamic libraries used for JNI (e.g. transport-native-epoll, and eventually kqueue).
- Make a new transport-native-unix-common-tests module where the tests for the transport-native-unix-common module will live. This is so each unix platform can inherit from these test and ensure they pass.
- Add a new transport-native-kqueue module which uses JNI to directly interact with kqueue

Result:
JNI support for kqueue.
Fixes https://github.com/netty/netty/issues/2448
Fixes https://github.com/netty/netty/issues/4231
2017-05-03 09:53:22 -07:00
Norman Maurer
6915ec3bb9 [maven-release-plugin] prepare for next development iteration 2017-04-29 14:10:00 +02:00
Norman Maurer
f30f242fee [maven-release-plugin] prepare release netty-4.1.10.Final 2017-04-29 14:09:32 +02:00
Norman Maurer
2b8c8e0805 [maven-release-plugin] prepare for next development iteration 2017-03-10 07:46:17 +01:00
Norman Maurer
1db58ea980 [maven-release-plugin] prepare release netty-4.1.9.Final 2017-03-10 07:45:28 +01:00
Jason Brown
3861b7de2b Motivation:
Make the FileRegion comments about which transports are supported more accurate.
Also, eleminate any outstanding references to FileRegion.transfered as the method was renamed for spelling.

Modifications:

Class-level comment on FileRegion, can call renamed method.

Result:

More accurate documentation and less calls to deprecated methods.
2017-03-06 14:43:52 -08:00
Scott Mitchell
93f5f62a20 Cleanup EPOLL native exceptions
Motivation:
Exceptions generated from transport-native-epoll may include duplicate error string description or inconsistent usage of the method name in the string description.

Modifications:
- Ensure the method name from static exceptions and dynamic exceptions is of the same format
- Remove duplicate string rational from the exception messages

Result:
More consistent error messages with no duplicate error description.
2017-03-01 05:42:48 -08:00
Norman Maurer
c5324ea48c Make netty build work on Java9
Motivation:

We missed some stuff in 5728e0eb2c and so the build failed on java9

Modifications:

- Add extra cmdline args when needed
- skip the autobahntestsuite as jython not works with java9
- skip the osgi testsuite as the maven plugin not works with java9

Result:

Build finally passed on java9
2017-02-16 20:26:30 +01:00
Scott Mitchell
0befcd8431 EpollDatagramChannel avoid getOption
Motivation:
EpollDatagramChannel uses getOption in the isActive method. getOption is backed by a relatively large conditional if/else if block and this conditional checking can be avoided in the epoll transport.

Modifications:
- Add EpollDatagramChannelConfig#getActiveOnOpen and use this in EpollDatagramChannel

Result:
Conditional checking due to getOption is removed from EpollDatagramChannel.
2017-02-16 11:04:37 -08:00
Scott Mitchell
f3dd410142 EPOLLRDHUP prematurely marking input as shutdown
Motivation:
When the EPOLLRDHUP event is received we assume that the read side of the FD is no longer functional and force the input state to be shutdown. However if the channel is still active we should rely upon EPOLLIN and read to indicate there is no more data before we update the shutdown state. If we do not do this we may not read all pending data in the FD if the RecvByteBufAllocator doesn't want to consume it all in a single read operation.

Modifications:
- AbstractEpollChannel#epollRdHupReady() shouldn't force shutdown the input if the channel is active

Result:
All data can be read even if the RecvByteBufAllocator doesn't read it all in the current read loop.
Fixes https://github.com/netty/netty/issues/6303
2017-02-16 08:40:32 -08:00
Norman Maurer
5728e0eb2c Use the correct arguments when run with jdk9
Motivation:

We need to pass special arguments to run with jdk9 as otherwise some tests will not be able to run.

Modifications:

Allow to define extra arguments when running with jdk9

Result:

Tests pass with jdk9
2017-02-15 10:15:00 +01:00
Scott Mitchell
413d6eba53 EPOLL include error description and cause in exceptions
Motivation:
EPOLL annotates some exceptions to provide the remote address, but the original exception is not preserved. This may make determining a root cause more difficult. The static EPOLL exceptions references the native method that failed, but does not provide a description of the actual error number. Without the description users have to know intimate details about the native calls and how they may fail to debug issues.

Modifications:
- annotated exceptions should preserve the original exception
- static exceptions should include the string description of the expected errno

Result:
EPOLL exceptions provide more context and are more useful to end users.
2017-02-13 18:37:16 -08:00
Scott Mitchell
a1b5b5dcca EpollRecvByteAllocatorHandle doesn't inform delegate of more data
Motivation:
EpollRecvByteAllocatorHandle intends to override the meaning of "maybe more data to read" which is a concept also used in all existing implementations of RecvByteBufAllocator$Handle but the interface doesn't support overriding. Because the interfaces lack the ability to propagate this computation EpollRecvByteAllocatorHandle attempts to implement a heuristic on top of the delegate which may lead to reading when we shouldn't or not reading data.

Modifications:
- Create a new interface ExtendedRecvByteBufAllocator and ExtendedHandle which allows the "maybe more data to read" between interfaces
- Deprecate RecvByteBufAllocator and change all existing implementations to extend ExtendedRecvByteBufAllocator
- transport-native-epoll should require ExtendedRecvByteBufAllocator so the "maybe more data to read" can be propagated to the ExtendedHandle

Result:
Fixes https://github.com/netty/netty/issues/6303.
2017-02-13 17:42:24 -08:00
Dmitriy Dumanskiy
b9abd3c9fc Cleanup : for loops for arrays to make code easier to read and removed unnecessary toLowerCase() 2017-02-06 07:47:59 +01:00
Norman Maurer
735d6dd636 [maven-release-plugin] prepare for next development iteration 2017-01-30 15:14:02 +01:00
Norman Maurer
76e22e63f3 [maven-release-plugin] prepare release netty-4.1.8.Final 2017-01-30 15:12:36 +01:00
Norman Maurer
81735c2535 Check if Epoll is avaible when construct EpollEventLoopGroup
Motivation:

We should call Epoll.ensureAvailability() when init EpollEventLoopGroup to fail fast and with a proper exception.

Modifications:

Call Epoll.ensureAvailability() during EpollEventLoopGroup init.

Result:

Fail fast if epoll is not availability (for whatever reason).
2017-01-27 08:19:45 +01:00
Norman Maurer
7f01da8d0f [maven-release-plugin] prepare for next development iteration 2017-01-12 11:36:51 +01:00
Norman Maurer
7a21eb1178 [maven-release-plugin] prepare release netty-4.1.7.Final 2017-01-12 11:35:58 +01:00
Scott Mitchell
631077c793 EPOLL RDHUP processing
Motivation:
EpollRecvByteAllocatorHandle will read unconditionally if EPOLLRDHUP has been received. However we can just treat this the same was we do as data maybe pending in ET mode, and let LT mode notify us if we haven't read all data.

Modifications:
- EpollRecvByteAllocatorHandle should not always force a read just because EPOLLRDHUP has been received, but just treated as an indicator that there maybe more data to read in ET mode

Result:
Fixes https://github.com/netty/netty/issues/6173.
2017-01-10 13:11:18 -08:00
Norman Maurer
89e93968ac Remove usage of own Atomic*FieldUpdater in favor of JDKs
Motivation:

In later Java8 versions our Atomic*FieldUpdater are slower then the JDK implementations so we should not use ours anymore. Even worse the JDK implementations provide for example an optimized version of addAndGet(...) using intrinsics which makes it a lot faster for this use-case.

Modifications:

- Remove methods that return our own Atomic*FieldUpdaters.
- Use the JDK implementations everywhere.

Result:

Faster code.
2016-12-15 08:09:06 +00:00
Norman Maurer
5fec897ef7 Fail build when unused c code is detected
Motivation:

To keep our code clean we should fail the build when unused c code is detected.

Modifications:

- Add '-Wunused-variable' to build flags

Result:

Cleaner code.
2016-12-15 08:04:25 +00:00
T Jake Luciani
8d664fa0fd Include frame-pointer to improve epoll flamegraph tracing
Motivation:

When attempting to flamegraph netty w/ epoll it was noticed the stacks are lost going from
java to epoll lib.

Modifications:

added the -fno-omit-framepointer flag to compiler flags to ensure the fp is kept intact

Result:

Flamegraphs will now show native code in the same stack as java code using perf-java-flames
2016-12-07 08:06:26 +01:00
Norman Maurer
2de644224c Correctly mark EpollServerDomainSocketChannel.isActive() as true after bind is complete.]
Motivation:

We missed to set active = true in EpollServerDomainSocketChannel.doBind(...) which also means that channelActive(...) was never triggered.

Modifications:

Correct set active = true in doBind(...)

Result:

EpollServerDomainSocketChannel is correctly set to active when bound.
2016-12-01 21:26:11 +01:00
Demetrius
b3fa976028 Added support to fetch the UID, GID, and PID of the connected unix domain socket (EG: SO_PEERCREDS)
Motivation:

I had a need to know the user credentials of a connected unix domain socket.

Modifications:

Added a class to encapsulate user credentials (UID, GID, and the PID).
Augemented the Socket class to provide the JNI native interface to return this new class
Augemented the c code to call getSockOpts passing <a href=http://man7.org/linux/man-pages/man7/socket.7.html>SO_PEERCRED</a>
Then surfaced the ability to get user credentials in the EpollDomainSocketChannel

Result:

The EpollDomainSocketChannel now has a the following function signature:
public PeerCredentials peerCredentials() throws IOException allowing a caller to get the UID, GID, and PID of the linux process
connected to the unix domain socket.
2016-11-04 07:27:30 +01:00
Christoph Ingwersen
b6099bc193 Fix unbalanced quotes in Bundle-NativeCode manifest header
Motivation:

The previously generated manifest causes a parse exception when loaded into an Apache Felix OSGI container.

Modifications:

Fix parameter delimiter and unbalanced quotes in manifest entry. Suffixed with asterisk so the bundle is resolved on other architectures as well even if native libs won't be loaded.

Result:

Bundle will load properly in OSGI containers.
2016-11-01 10:24:42 +01:00
Norman Maurer
5f533b7358 [maven-release-plugin] prepare for next development iteration 2016-10-14 13:20:41 +02:00
Norman Maurer
35fb0babe2 [maven-release-plugin] prepare release netty-4.1.6.Final 2016-10-14 12:47:19 +02:00
Norman Maurer
a09e56850e [#5882] Ensure we even process tasks if processing of ready channels throws an Exception in event loop.
Motivation:

If an exception is thrown while processing the ready channels in the EventLoop we should still run all tasks as this may allow to recover. For example a OutOfMemoryError may be thrown and runAllTasks() will free up memory again. Beside this we should also ensure we always allow to shutdown even if an exception was thrown.

Modifications:

- Call runAllTasks() in a finally block
- Ensure shutdown is always handles.

Result:

More robust EventLoop implementations for NIO and Epoll.
2016-10-10 07:49:57 +02:00
Norman Maurer
2c4a7a2539 [#5800] Support any FileRegion implementation when using epoll transport
Motivation:

At the moment only DefaultFileRegion is supported when using the native epoll transport.

Modification:

- Add support for any FileRegion implementation
- Add test case

Result:

Also custom FileRegion implementation are supported when using the epoll transport.
2016-09-15 23:03:37 -07:00
Norman Maurer
54b1a100f4 [maven-release-plugin] prepare for next development iteration 2016-08-26 10:06:32 +02:00