Commit Graph

463 Commits

Author SHA1 Message Date
Carl Mastrangelo
262ced7ce4 Unconditionally initialize sockaddrs in epoll linuxsocket (#9299)
Motivation:

Compiling with -Werror,-Wuninitialized complains about the sockaddrs being uninitialized.
I believe this is because the init function netty_unix_socket_initSockaddr is in a
separate compilation unit.  Since this code isn't on the criticial path, it's easy
to just memset the variables rather than suppress the warning.

Modification:
Always clear the sockaddrs, even if they will be initialized later.

Result:
Able to compile with warnings turned on
2019-06-29 12:16:58 +02:00
root
5b58b8e6b5 [maven-release-plugin] prepare for next development iteration 2019-06-28 05:57:21 +00:00
root
35e0843376 [maven-release-plugin] prepare release netty-4.1.37.Final 2019-06-28 05:56:28 +00:00
jimin
856f1185e1 All override methods must be added @override (#9285)
Motivation:

Some methods that either override others or are implemented as part of implementation an interface did miss the `@Override` annotation

Modifications:

Add missing `@Override`s

Result:

Code cleanup
2019-06-27 13:51:26 +02:00
jimin
9621a5b981 remove unused imports (#9287)
Motivation:

Some imports are not used

Modification:

remove unused imports

Result:

Code cleanup
2019-06-26 21:08:31 +02:00
Norman Maurer
517a93d87d Make EventLoopTaskQueueFactory a top-level interface
Motivation:

c9aaa93d83 added the ability to specify an EventLoopTaskQueueFactory but did place it under MultithreadEventLoopGroup while not really belongs there.

Modifications:

Make EventLoopTaskQueueFactory a top-level interface

Result:

More logical code layout.
2019-06-22 07:38:03 +02:00
Norman Maurer
c9aaa93d83
Allow to specify a EventLoopTaskQueueFactory for various EventLoopGroup implementations (#9247)
Motivation:

Sometimes it is desirable to be able to use a different Queue implementation for the EventLoop of a Channel. This is currently not possible without resort to reflection.

Modifications:

- Add a new constructor to Nio|Epoll|KQueueEventLoopGroup which allows to specify a factory which is used to create the task queue. This was the user can override the default implementation.
- Add test

Result:

Be able to change Queue that is used for the EventLoop.
2019-06-21 09:05:19 +02:00
Norman Maurer
165229658b
Add support for loopbackmode and accessing the configured interface when using epoll native transport with multicast (#9218)
Motivation:

We did not have support for enable / disable loopback mode in our native epoll transport and also missed the implemention to access the configured interface.

Modifications:

Add implementation and adjust test to cover it

Result:

More complete multicast support with native epoll transport
2019-06-07 13:44:06 -07:00
Carl Mastrangelo
9abeaf16fd Properly debounce wakeups (#9191)
Motivation:
The wakeup logic in EpollEventLoop is overly complex

Modification:
* Simplify the race to wakeup the loop
* Dont let the event loop wake up itself (it's already awake!)
* Make event loop check if there are any more tasks after preparing to
sleep.  There is small window where the non-eventloop writers can issue
eventfd writes here, but that is okay.

Result:
Cleaner wakeup logic.

Benchmarks:

```
BEFORE
Benchmark                                   Mode  Cnt       Score      Error  Units
EpollSocketChannelBenchmark.executeMulti   thrpt   20  408381.411 ± 2857.498  ops/s
EpollSocketChannelBenchmark.executeSingle  thrpt   20  157022.360 ± 1240.573  ops/s
EpollSocketChannelBenchmark.pingPong       thrpt   20   60571.704 ±  331.125  ops/s

Benchmark                                   Mode  Cnt       Score      Error  Units
EpollSocketChannelBenchmark.executeMulti   thrpt   20  440546.953 ± 1652.823  ops/s
EpollSocketChannelBenchmark.executeSingle  thrpt   20  168114.751 ± 1176.609  ops/s
EpollSocketChannelBenchmark.pingPong       thrpt   20   61231.878 ±  520.108  ops/s
```
2019-06-04 05:17:23 -07:00
Norman Maurer
f6cf681f90
Don't read from timerfd and eventfd on each EventLoop tick (#9192)
Motivation:

We do not need to issue a read on timerfd and eventfd when the EventLoop wakes up if we register these as Edge-Triggered. This removes the overhead of 2 syscalls and so helps to reduce latency.

Modifications:

- Ensure we register the timerfd and eventfd with EPOLLET flag
- If eventfd_write fails with EAGAIN, call eventfd_read and try eventfd_write again as we only use it as wake-up mechanism.

Result:

Less syscalls and so reducing overhead.

Co-authored-by: Carl Mastrangelo <carl@carlmastrangelo.com>
2019-05-31 06:59:39 +02:00
Norman Maurer
e17ce934da
Correctly detect InternetProtocolFamily when EpollDatagramChannel is created with existing FileDescriptor (#9185)
Motivation:

When EpollDatagramChannel is created with an existing FileDescriptor we should detect the correct InternetProtocolFamily.

Modifications:

Obtain the InternetProtocolFamily from the given FD

Result:

Use correct InternetProtocolFamily when EpollDatagramChannel is created via existing FileDescriptor
2019-05-26 20:22:55 +02:00
Steve Buzzard
70731bfa7e Added UDP multicast (with caveats: getInterface, getNetworkInterface, block or loopback-mode-disabled operations).
Motivation:

Provide epoll/native multicast to support high load multicast users (we are using it for a high load telecomm app at my day job).

Modification:

Added support for source specific and any source multicast for epoll transport. Some caveats: no support for disabling loop back mode, retrieval of interface and block operation, all of which tend to be less frequently used.

Result:

Provides epoll transport multicast for common use cases.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2019-05-25 08:00:16 +02:00
Julien Viet
e348bd9217 KQueueEventLoop | EpollEventLoop may incorrectly update registration when FD is reused.
Motivation:

The current KQueueEventLoop implementation does not process concurrent domain socket channel registration/unregistration in the order they actual
happen since unregistration are delated by an event loop task scheduling. When a domain socket is closed, it's file descriptor might be reused
quickly and therefore trigger a new channel registration using the same descriptor.

Consequently the KQueueEventLoop#add(AbstractKQueueChannel) method will overwrite the current inactive channels having the same descriptor
and the delayed KQueueEventLoop#remove(AbstractKQueueChannel) will remove the active channel that replaced the inactive one.

As active channels are registered, events for this file descriptor won't be processed anymore and the channels will never be closed.

The same problem can also happen in EpollEventLoop. Beside this we also may never remove the AbstractEpollChannel from the internal map
when it is unregistered which will prevent it from be GC'ed

Modifications:

- Change logic of native KQueue and Epoll implementations to ensure we correctly handle the case of FD reuse
- Only try to update kevent / epoll if the Channel is still open (as otherwise it will be handled by kqueue / epoll itself)
- Correctly remove AbstractEpollChannel from internal map in all cases
- Make implementation of closeAll() consistent for Epoll and KQueueEventLoop

Result:

KQueue and Epoll native transports correctly handle FD reuse

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2019-05-22 09:23:09 +02:00
Norman Maurer
f17bfd0f64
Only use static Exception instances when we can ensure addSuppressed … (#9152)
Motivation:

OOME is occurred by increasing suppressedExceptions because other libraries call Throwable#addSuppressed. As we have no control over what other libraries do we need to ensure this can not lead to OOME.

Modifications:

Only use static instances of the Exceptions if we can either dissable addSuppressed or we run on java6.

Result:

Not possible to OOME because of addSuppressed. Fixes https://github.com/netty/netty/issues/9151.
2019-05-17 22:23:02 +02:00
root
ba06eafa1c [maven-release-plugin] prepare for next development iteration 2019-04-30 16:42:29 +00:00
root
49a451101c [maven-release-plugin] prepare release netty-4.1.36.Final 2019-04-30 16:41:28 +00:00
Nick Hill
6248b2492b Remove static wildcard imports in EpollDomainSocketChannelConfig (#9066)
Motivation

These aren't needed, only one field from each class is used. It also showed as an ambiguous identifier compilation error in my IDE even though javac is obviously fine with it.

Modifications

Static-import explicit ChannelOption fields in EpollDomainSocketChannelConfig instead of using .* wildcard.

Result

Cleaner / more consistent code.
2019-04-18 07:33:44 +02:00
root
baab215f66 [maven-release-plugin] prepare for next development iteration 2019-04-17 07:26:24 +00:00
root
dfe657e2d4 [maven-release-plugin] prepare release netty-4.1.35.Final 2019-04-17 07:25:40 +00:00
Nick Hill
9ed41db1d7 Have (Epoll|KQueue)RecvByteAllocatorHandle extend DelegatingHandle (#9060)
Motivation

These implementations delegate most of their methods to an existing Handle and previously extended RecvByteBufAllocator.DelegatingHandle. This was reverted in #6322 with the introduction of ExtendedHandle but it's not clear to me why it needed to be - the code looks a lot cleaner.

Modifications

Have (Epoll|KQueue)RecvByteAllocatorHandle extend DelegatingHandle again, while still implementing ExtendedHandle.

Result

Less code.
2019-04-16 09:14:09 +02:00
Norman Maurer
86dd388637 Revert "Added UDP multicast (with caveats: no ipv6, getInterface, getNetworkI… (#9006)"
This reverts commit a3e8c86741 as there are some issues that need to be fixed first.
2019-04-12 21:32:22 +02:00
Norman Maurer
fcfa9eb9a8
Throw IOException (not ChannelException) if netty_epoll_linuxsocket_setTcpMd5Sig fails (#9039)
Motivation:

At the moment we throw a ChannelException if netty_epoll_linuxsocket_setTcpMd5Sig fails. This is inconsistent with other methods which throw a IOException.

Modifications:

Throw IOException

Result:

More correct and consistent exception usage in epoll transport
2019-04-12 15:15:27 +02:00
Norman Maurer
778ff2057e
Add IPv6 multicast test to testsuite (#9037)
Motivation:

We currently only cover ipv4 multicast in the testsuite but we should also have tests for ipv6.

Modifications:

- Add test for ipv6
- Ensure we only try to run multicast test for ipv4 / ipv6 if the loopback interface supports it.

Result:

Better test coverage
2019-04-12 12:29:08 +02:00
Norman Maurer
45b0daf9e6
netty_epoll_linuxsocket_setTcpMd5Sig should throw ChannelException when not able to init sockaddr (#9029)
Motivation:

When netty_epoll_linuxsocket_setTcpMd5Sig fails to init the sockaddr we should throw an exception and not silently return.

Modifications:

Throw exception if init of sockaddr fails.

Result:

Correctly report back error to user.
2019-04-11 18:50:16 +02:00
Steve Buzzard
a3e8c86741 Added UDP multicast (with caveats: no ipv6, getInterface, getNetworkI… (#9006)
…nterface, block or loopback-mode-disabled operations).


Motivation:

Provide epoll/native multicast to support high load multicast users (we are using it for a high load telecomm app at my day job).

Modification:

Added support for (ipv4 only) source specific and any source multicast for epoll transport. Some caveats (beyond no ipv6 support initially - there’s a bit of work to add in join and leave group specifically around SSM, as ipv6 uses different data structures for this): no support for disabling loop back mode, retrieval of interface and block operation, all of which tend to be less frequently used.

Result:

Provides epoll transport multicast for IPv4 for common use cases. Understand if you’d prefer to hold off until ipv6 is included but not sure when I’ll be able to get to that.
2019-04-08 20:13:39 +02:00
Vladimir Kostyukov
0a0da67f43 Introduce SingleThreadEventLoop.registeredChannels (#8428)
Motivation:

Systems depending on Netty may benefit (telemetry, alternative even loop scheduling algorithms) from knowing the number of channels assigned to each EventLoop.

Modification:

Expose the number of channels registered in the EventLoop via SingleThreadEventLoop.registeredChannels.

Result:

Fixes #8276.
2019-03-28 11:33:12 +00:00
Norman Maurer
8206604003
Upgrade to new netty-build and com.puppycrawl.tools 8.18 (#8980)
Motivation:

com.puppycrawl.tools checkstyle < 8.18 was reported to contain a possible security flaw. We should upgrade.

Modifications:

- Upgrade netty-build and checkstyle.
- Fix checkstyle errors

Result:

Fixes https://github.com/netty/netty/issues/8968.
2019-03-26 14:21:34 +01:00
Lunfu Zhong
e7b3195570 Support ALLOW_HALF_CLOSURE channel option on Unix domain socket. (#8932)
Motivation:

Since DomainSocketChannel is a DuplexChannel,  which be able to shutdown input or output individually on demands, but ALLOW_HALF_CLOSURE channel option has not been supported yet.

I thought this could be a missing feature of Unix domain socket, so here the PR for it.

Modifications:

1. Added allHalfClosure property both in  EpollDomainSocketChannelConfig and KQueueDomainSocketChannelConfig,
2. Enabled isAllowHalfClosure method of native channel to support domain channel config,
3. Created EpollDomainSocketShutdownOutputByPeerTest and KQueueDomainSocketShutdownOutputByPeerTest to verify the change.

Result:

ALLOW_HALF_CLOSURE channel option can be set with DomainSocketChannel, and no more warning of Unknown channel option 'ALLOW_HALF_CLOSURE'.
2019-03-19 11:24:07 +01:00
Norman Maurer
5eb91d9ca1
Remove --add-opens=java.base/java.nio=ALL-UNNAMED when running tests as it is not needed anymore since a long time (#8934)
Motivation:

At some point we needed --add-opens=java.base/java.nio=ALL-UNNAMED to run our native tests but this is not true anymore.

Modifications:

Remove --add-opens=java.base/java.nio=ALL-UNNAMED when running native tests.

Result:

Remove obsolate jvm arg.
2019-03-13 08:25:10 +01:00
root
92b19cfedd [maven-release-plugin] prepare for next development iteration 2019-03-08 08:55:45 +00:00
root
ff7a9fa091 [maven-release-plugin] prepare release netty-4.1.34.Final 2019-03-08 08:51:34 +00:00
Norman Maurer
81e43d5088
DefaultFileRegion.transferTo with invalid count may cause busy-spin (#8885)
Motivation:

`DefaultFileRegion.transferTo` will return 0 all the time when we request more data then the actual file size. This may result in a busy spin while processing the fileregion during writes.

Modifications:

- If we wrote 0 bytes check if the underlying file size is smaller then the requested count and if so throw an IOException
- Add DefaultFileRegionTest
- Add a test to the testsuite

Result:

Fixes https://github.com/netty/netty/issues/8868.
2019-02-26 11:08:09 +01:00
Norman Maurer
fa6a8cb09c
Support using an Executor to offload blocking / long-running tasks wh… (#8847)
Motivation:

The SSLEngine does provide a way to signal to the caller that it may need to execute a blocking / long-running task which then can be offloaded to an Executor to ensure the I/O thread is not blocked. Currently how we handle this in SslHandler is not really optimal as while we offload to the Executor we still block the I/O Thread.

Modifications:

- Correctly support offloading the task to the Executor while suspending processing of SSL in the I/O Thread
- Add new methods to SslContext to specify the Executor when creating a SslHandler
- Remove @deprecated annotations from SslHandler constructor that takes an Executor
- Adjust tests to also run with the Executor to ensure all works as expected.

Result:

Be able to offload long running tasks to an Executor when using SslHandler. Partly fixes https://github.com/netty/netty/issues/7862 and https://github.com/netty/netty/issues/7020.
2019-02-11 09:47:44 +01:00
Norman Maurer
c6a90d90a6
Add more tests to KQueue and Epoll testsuites. (#8851)
Motivation:

We missed to extend a few tests from the testsuite and so also run these with our native KQueue and Epoll transport.

Modifications:

Extend tests and so run these for our native transports as well.

Result:

More tests.
2019-02-08 20:08:34 +01:00
Norman Maurer
7f61055cbd
Reduce direct memory overhead per EpollEventLoop when using EpollDatagramChannel (#8825)
Motivation:

When using a linux distribution that supports sendmmsg(...) we allocated enough direct memory per EpollEventLoop to be able to write IOV_MAX number of iovecs per message that can be written per sendmmsg.
The number of messages that can be written per sendmmsg(...) call is limited by UIO_MAX_IOV.

In practice this resulted in an allocation of 16MB direct memory per EpollEventLoop instance that stayed allocated until the EpollEventLoop was shutdown which happens as part of the shutdown of the enclosing EpollEVentLoopGroup.

This resulted in quite some heavy direct memory usage in practice even when in practice we have very slim changes to ever need all of the memory.

Modification:

Adjust NativeDatagramPacketArray to share one IovArray instance across all NativeDatagramPacket instances it holds. This limits the max number of iovecs we can write across all messages to IOV_MAX per sendmmsg(...) call.
This in practice will still be enough to allow us to write multiple messages with one syscall while keep the memory overhead to a minimum.

Result:

Smaller direct memory footprint per EpollEventLoop when using EpollDatagramChannel on distributions that support sendmmsg(...).
Fixes https://github.com/netty/netty/issues/8814
2019-02-02 07:10:02 +01:00
Norman Maurer
7bba4f49cf
Reduce GC produced by native DatagramChannel implementations when in connected mode. (#8806)
Motivation:

In the native code EpollDatagramChannel / KQueueDatagramChannel creates a DatagramSocketAddress object for each received UDP datagram even when in connected mode as it uses the recvfrom(...) / recvmsg(...)  method. Creating these is quite heavy in terms of allocations as internally, char[], String, Inet4Address, InetAddressHolder, InetSocketAddressHolder, InetAddress[], byte[] objects are getting generated when constructing the object. When in connected mode we can just use regular read(...) calls which do not need to allocate all of these.

Modifications:

- When in connected mode use read(...) and NOT recvfrom(..) / readmsg(...) to reduce allocations when possible.
- Adjust tests to ensure read works as expected when in connected mode.

Result:

Less allocations and GC when using native datagram channels in connected mode. Fixes https://github.com/netty/netty/issues/8770.
2019-02-01 10:29:36 +01:00
田欧
a33200ca38 use checkPositive/checkPositiveOrZero (#8803)
Motivation:

We have a utility method to check for > 0 and >0 arguments. We should use it.

Modification:

use checkPositive/checkPositiveOrZero instead of if statement.

Result:

Re-use utility method.
2019-01-31 09:07:14 +01:00
Norman Maurer
cd3254df88
Update to new checkstyle plugin (#8777) (#8780)
Motivation:

We need to update to a new checkstyle plugin to allow the usage of lambdas.

Modifications:

- Update to new plugin version.
- Fix checkstyle problems.

Result:

Be able to use checkstyle plugin which supports new Java syntax.
2019-01-25 11:58:42 +01:00
root
cf03ed0478 [maven-release-plugin] prepare for next development iteration 2019-01-21 12:26:44 +00:00
root
37484635cb [maven-release-plugin] prepare release netty-4.1.33.Final 2019-01-21 12:26:12 +00:00
yulianoifa-mobius
1e4481e551 Allowed IP_FREEBIND option for UDP epoll (#8728)
Motivation:

While using Load Balancers or HA support is needed there are cases when UDP channel need to bind to IP Address which is not available on network interfaces locally.

Modification:

Modified EpollDatagramChannelConfig to allow IP_FREEBIND option

Result:

Fixes ##8727.
2019-01-21 07:42:05 +01:00
Dmitriy Dumanskiy
165912365a Clenaup: simplify EpollEventLoop.closeAll() (#8719)
Motivation:

Avoid unnecessary iteration and `ArrayList` allocation.

Modification:

```
for (AbstractEpollChannel channel: channels.values()) {
     array.add(channel);
}
```
replaced with 

`array.addAll(channels.values())`

and

```
Collection<AbstractEpollChannel> array = new ArrayList<AbstractEpollChannel>(channels.size());
array.addAll(channels.values())
```

replaced with:

`AbstractEpollChannel[] localChannels = channels.values().toArray(new AbstractEpollChannel[0]);`

Result:

Simpler code in `EpollEventLoop.closeAll();`
2019-01-16 11:00:25 +01:00
Feri73
5df235c083 Correcting Maven Dependencies (#8622)
Motivation:

Most of the maven modules do not explicitly declare their
dependencies and rely on transitivity, which is not always correct.

Modifications:

For all maven modules, add all of their dependencies to pom.xml

Result:

All of the (essentially non-transitive) depepdencies of the modules are explicitly declared in pom.xml
2018-12-06 09:01:14 +01:00
root
8eb313072e [maven-release-plugin] prepare for next development iteration 2018-11-29 11:15:09 +00:00
root
afcb4a37d3 [maven-release-plugin] prepare release netty-4.1.32.Final 2018-11-29 11:14:20 +00:00
Norman Maurer
d4b1202e62
Add testcase for epollWait(...) with negative timerfd values. (#8447)
Motivation:

https://github.com/netty/netty/issues/8444 reports that there is some issue with negative values passed to timerfd_settime. This test verifies that everything is working as expected.

Modifications:

Add testcase.

Result:

Test to verify expected behaviour.
2018-10-30 19:38:02 +01:00
root
3e7ddb36c7 [maven-release-plugin] prepare for next development iteration 2018-10-29 15:38:51 +00:00
root
9e50739601 [maven-release-plugin] prepare release netty-4.1.31.Final 2018-10-29 15:37:47 +00:00
Johno Crawford
5b3b8db07f epoll_wait produces an EINVAL error since 4.1.30 (#8350)
Motivation:

epoll_wait should work in 4.1.30 like it did in 4.1.29.

Modifications:

Revert Integer.MAX_VALUE back to MAX_SCHEDULED_TIMERFD_NS (999,999,999).
Add unit test.

Result:

epoll_wait will no longer throw EINVAL.
2018-10-12 05:02:41 +02:00
Matteo Merli
3a96e7373b Added option to do busy-wait on epoll (#8267)
Motivation:

Add an option (through a SelectStrategy return code) to have the Netty event loop thread to do busy-wait on the epoll.

The reason for this change is to avoid the context switch cost that comes when the event loop thread is blocked on the epoll_wait() call.

On average, the context switch has a penalty of ~13usec.

This benefits both:

The latency when reading from a socket
Scheduling tasks to be executed on the event loop thread.
The tradeoff, when enabling this feature, is that the event loop thread will be using 100% cpu, even when inactive.

Modification:

Added SelectStrategy option to return BUSY_WAIT
Epoll loop will do a epoll_wait() with no timeout
Use pause instruction to hint to processor that we're in a busy loop
Result:

When enabled, minimizes impact of context switch in the critical path
2018-09-28 22:52:00 +02:00