Commit Graph

1706 Commits

Author SHA1 Message Date
Scott Mitchell
eb1dfb8f5e SingleThreadEventLoopTest failures
Motivation:
Some unit tests in SingleThreadEventLoopTest rely upon Thread.sleep for sequencing events between threads. This can be unreliable and result in spurious test failures if thread scheduling does not occur in a fair predictable manner.

Modifications:
- Reduce the reliance on Thread.sleep in SingleThreadEventLoopTest

Result:
Fixes https://github.com/netty/netty/issues/5851
2016-10-11 09:09:53 +02:00
Norman Maurer
6c8cd023bf [#5770] Use heapbuffers by default when using LocalChannel and LocalServerChannel.
Motivation:

The local transport is used to communicate in the same JVM so we should use heap buffers.

Modifications:

Use heapbuffers by default if not requested otherwise.

Result:

No allocating of direct buffers by default when using local transport
2016-10-10 11:16:17 +02:00
Norman Maurer
e102a008b6 [#5893] Ensure we not close NioDatagramChannel when SocketException is received.
Motivation:

When using java.nio.DatagramChannel we should not close the channel when a SocketException was thrown as we can still use the channel.

Modifications:

Not close the Channel when SocketException is thrown

Result:

More robust and correct handling of exceptions when using NioDatagramChannel.
2016-10-10 10:24:28 +02:00
Norman Maurer
a09e56850e [#5882] Ensure we even process tasks if processing of ready channels throws an Exception in event loop.
Motivation:

If an exception is thrown while processing the ready channels in the EventLoop we should still run all tasks as this may allow to recover. For example a OutOfMemoryError may be thrown and runAllTasks() will free up memory again. Beside this we should also ensure we always allow to shutdown even if an exception was thrown.

Modifications:

- Call runAllTasks() in a finally block
- Ensure shutdown is always handles.

Result:

More robust EventLoop implementations for NIO and Epoll.
2016-10-10 07:49:57 +02:00
Norman Maurer
3cf7ccbd3c Process OP_WRITE before OP_READ to free memory faster
Motivation:

We should better first process OP_WRITE before OP_READ as this may allow us to free memory in a faster fashion for previous queued writes.

Modifications:

Process OP_WRITE before OP_READ

Result:

Free memory faster for queued writes.
2016-10-10 07:42:39 +02:00
radai-rosenblatt
15ac6c4a1f Clean-up unused imports
Motivation:

the build doesnt seem to enforce this, so they piled up

Modifications:

removed unused import lines

Result:

less unused imports

Signed-off-by: radai-rosenblatt <radai.rosenblatt@gmail.com>
2016-09-30 09:08:50 +02:00
Roger Kapsi
149916d052 Adding ability omit the implicit #flush() call in EmbeddedChannel#writeOutbound() and
the implicit #fireChannelReadComplete() in EmbeddedChannel#writeInbound().

Motivation

We use EmbeddedChannels to implement a ProxyChannel of some sorts that shovels
messages between a source and a destination Channel. The latter are real network
channels (such as Epoll) and they may or may not be managed in a ChannelPool. We
could fuse both ends directly together but the EmbeddedChannel provides a nice
disposable section of a ChannelPipeline that can be used to instrument the messages
that are passing through the proxy portion.

The ideal flow looks abount like this:

source#channelRead() -> proxy#writeOutbound() -> destination#write()
source#channelReadComplete() -> proxy#flushOutbound() -> destination#flush()

destination#channelRead() -> proxy#writeInbound() -> source#write()
destination#channelReadComplete() -> proxy#flushInbound() -> source#flush()

The problem is that #writeOutbound() and #writeInbound() emit surplus #flush()
and #fireChannelReadComplete() events which in turn yield to surplus #flush()
calls on both ends of the pipeline.

Modifications

Introduce a new set of write methods that reain the same sematics as the #write()
method and #flushOutbound() and #flushInbound().

Result

It's possible to implement the above ideal flow.

Fix for EmbeddedChannel#ensureOpen() and Unit Tests for it

Some PR stuff.
2016-09-24 12:33:04 -07:00
ChuntaoLu
9008e72c2b Fix javadoc
Removes unmatched brace
2016-09-12 06:43:10 -07:00
Norman Maurer
6bbf32134a Log more details if notification of promise fails in PromiseNotifier and AbstractChannelHandlerContext
Motivation:

To make it easier to debug why notification of a promise failed we should log extra info and make it consistent.

Modifications:

- Create a new PromiseNotificationUtil that has static methods that can be used to try notify a promise and log.
- Reuse this in AbstractChannelHandlerContext, ChannelOutboundBuffer and PromiseNotifier

Result:

Easier to debug why a promise could not be notified.
2016-09-07 06:55:38 +02:00
Norman Maurer
dfa3bbbf00 Add support for Client Subnet in DNS Queries (RFC7871)
Motivation:

RFC7871 defines an extension which allows to request responses for a given subset.

Modifications:

- Add DnsOptPseudoRrRecord which can act as base class for extensions based on EDNS(0) as defined in RFC6891
- Add DnsOptEcsRecord to support the Client Subnet in DNS Queries extension
- Add tests

Result:

Client Subnet in DNS Queries extension is now supported.
2016-09-06 07:16:57 +02:00
buchgr
4af6855695 Remove @Deprecated for primitive WriteWaterMark getters and setters
Motivation:

For use cases that demand frequent updates of the write watermarks, an
API that requires immutable WriteWaterMark objects is not ideal, as it
implies a lot of object allocation.

For example, the HTTP/2 child channel API uses write watermarks for outbound
flow control and updates the write watermarks on every DATA frame write.

Modifications:

Remote @Deprecated tag from primitive getters and setters, however the corresponding
channel options remain deprecated.

Result:

Primitive getters and setters for write watermarks are no longer marked @Deprecated.
2016-09-05 10:26:05 +02:00
Norman Maurer
30fe2e868f Call finishConnect() before try to call read(...) / write(...) when using NIO
Motivation:

The JDK implementation of SocketChannel has an internal state that is tracked for its operations. Because of this we need to ensure we call finishConnect() before try to call read(...) / write(...) as otherwise it may produce a NotYetConnectedException.

Modifications:

First process OP_CONNECT flag.

Result:

No more possibility of NotYetConnectedException because OP_CONNECT is handled not early enough when processing interestedOps for a Channel.
2016-09-01 08:55:02 +02:00
Norman Maurer
54b1a100f4 [maven-release-plugin] prepare for next development iteration 2016-08-26 10:06:32 +02:00
Norman Maurer
6fd8bb8c63 [#5763] DefaultEventLoopGroup doesn't expose ctor variant that accepts custom Executor
Motivation:

The DefaultEventLoopGroup class extends MultithreadEventExecutorGroup but doesn't expose the ctor variants that accept a custom Executor like NioEventLoopGroup and EpollEventLoopGroup do.

Modifications:

Add missing constructor.

Result:

Be able to use custom Executor with DefaultEventLoopGroup.
2016-09-01 08:20:05 +02:00
Jason Tedor
00c0664ef8 Avoid inaccessible object exception replacing set
Motivation:

When attempting to set the selectedKeys fields on the selector
implementation, JDK 9 can throw an inaccessible object exception.

Modications:

Catch and log this exception as an possible course of action if the
sun.nio.ch package is not exported from java.base.

Result:

The selector replacement will fail gracefully as an expected course of
action if the sun.nio.ch package is not exported from java.base.
2016-08-31 13:59:48 +02:00
Norman Maurer
1208b90f57 [maven-release-plugin] prepare release netty-4.1.5.Final 2016-08-26 04:59:35 +02:00
Norman Maurer
d3cb95ef00 Make NIO and EPOLL transport connect errors more consistent with the JDK
Motivation:

The NIO transport used an IllegalStateException if a user tried to issue another connect(...) while the connect was still in process. For this case the JDK specified a ConnectPendingException which we should use. The same issues exists in the EPOLL transport. Beside this the EPOLL transport also does not throw the right exceptions for ENETUNREACH and EISCONN errno codes.

Modifications:

- Replace IllegalStateException with ConnectPendingException in NIO and EPOLL transport
- throw correct exceptions for ENETUNREACH and EISCONN in EPOLL transport
- Add test case

Result:

More correct error handling for connect attempts when using NIO and EPOLL transport
2016-08-27 20:57:36 +02:00
buchgr
4accd300e5 Fix write watermarks comparison to use less than and greater than.
Motivation:

The API documentation in ChannelConfig states that a a channel is writable,
if the number of pending bytes is below the low watermark and a
channel is not writable, if the number of pending bytes exceeds the high
watermark.

Therefore, we should use < operators instead of <= as well as > instead of >=.

Using <= and >= is also problematic, if the low watermark is equal to the high watermark,
as then a channel could be both writable and unwritable with the same number of pending
bytes (depending on whether remove() or addMessage() is called first).

The use of <= and >= was introduced in PR https://github.com/netty/netty/pull/3036, but
I don't understand why, as there doesn't seem to have been any discussion around that.

Modifications:

Use < and > operators instead of <= and >=.

Result:

High and low watermarks are treated as stated in the API docs.
2016-08-24 15:58:02 +02:00
Norman Maurer
5e148d5670 [#5639] Ensure fireChannelActive() is also called if Channel is closed in connect promise.
Motivation:

We need to ensure we also call fireChannelActive() if the Channel is directly closed in a ChannelFutureListener that is belongs to the promise for the connect. Otherwise we will see missing active events.

Modifications:

Ensure we always call fireChannelActive() if the Channel was active.

Result:

No missing events.
2016-08-24 08:47:49 +02:00
Norman Maurer
43626ac6ea Use NIO methods when using Java7+ in the NIO transport
Motivation:

We use often javachannel().socket().* in NIO as these methods exists in java6. The problem is that these will throw often very general Exceptions (Like SocketException) while it is more expected to throw the Exceptions listed in the nio interfaces. When possible we should use the new methods available in java7+ which throw the correct exceptions.

Modifications:

Check for java version and depending on it using the socket or the javachannel.

Result:

Throw expected Exceptions.
2016-08-24 07:36:14 +02:00
Norman Maurer
8ce7e73e78 Prevent extra peformance hit by fillInStackTrace() when create a new annotated connect exception.
Motivation:

To make it easier to debug connect exceptions we create new exceptions which also contain the remote address. For this we basically created a new instance and call setStackTrace(...). When doing this we pay an extra penality because it calls fillInStackTrace() when calling the super constructor.

Modifications:

Create special sub-classes of Exceptions that override the fillInStackTrace() method and so eliminate the overhead.

Result:

Less overhead when "annotate" connect exceptions.
2016-08-24 07:34:43 +02:00
buchgr
8d1e46ffd1 Remove reference to 5.0 release.
Motivation:

Comments stating that AUTO_CLOSE will be removed in Netty 5.0 are wrong,
as there is no Netty 5.0.

Modifications:

Removed comment.

Result:

No more references to Netty 5.0
2016-08-23 09:46:57 +02:00
Norman Maurer
6d70f4b38a Guard against re-entrance in PendingWriteQueue.removeAndWriteAll()
Motivation:

PendingWriteQueue should guard against re-entrant writes once removeAndWriteAll() is run.

Modifications:

Continue writing until queue is empty.

Result:

Correctly guard against re-entrance.
2016-08-18 07:13:19 +02:00
tbcs
95ee705e17 fix typo in javadoc 2016-08-12 20:42:37 +02:00
Jason Tedor
a6dfd08812 Mark initialization of selector as privileged
Motivation:

Instrumenting the NIO selector implementation requires special
permissions. Yet, the code for performing this instrumentation is
executed in a manner that would require all code leading up to the
initialization to have the requisite permissions. In a restrictive
environment (e.g., under a security policy that only grants the
requisite permissions the Netty transport jar but not to application
code triggering the Netty initialization), then instrumeting the
selector will not succeed even if the security policy would otherwise
permit it.

Modifications:

This commit marks the necessary blocks as privileged. This enables
access to the necessary resources for instrumenting the selector. The
idea is that we are saying the Netty code is trusted, and as long as the
Netty code has been granted the necessary permissions, then we will
allow the caller access to these resources even though the caller itself
might not have the requisite permissions.

Result:

The selector can be instrumented in a restrictive security environment.
2016-08-05 19:01:57 +02:00
Jason Tedor
32629078a2 Mark setting of sun.nio.ch.bugLevel as privileged
Motivation:

Writing to a system property requires permissions. Yet the code for
setting sun.nio.ch.bugLevel is not marked as privileged. In a
restrictive environment (e.g., under a security policy that only grants
the requisite permissions the Netty transport jar but not to application
code triggering the Netty initialization), writing to this system
property will not succeed even if the security policy would otherwise
permit it.

Modifications:

This commt marks the necessary code block as privileged. This enables
writing to this system property. The idea is that we are saying the
Netty code is trusted, and as long as the Netty code has been granted
the necessary permissions, then we will allow the caller access to these
resources even though the caller itself might not have the requisite
permissions.

Result:

The system property sun.nio.ch.bugLevel can be written to in a
restrictive security environment.
2016-08-05 18:58:34 +02:00
Norman Maurer
aa6e6ae307 [#4241] Ensure NioEventLoopGroup.shutdownGracefully(...) with no quiet period shutdown as fast as expected.
Motivation:

If the user uses 0 as quiet period we should shutdown without any delay if possible.

Modifications:

Ensure we not introduce extra delay when a shutdown quit period of 0 is used.

Result:

EventLoop shutdown as fast as expected.
2016-08-05 07:21:17 +02:00
Norman Maurer
26aa34853a Ensure correct ordering if a ChannelInitializer adds another ChannelInitializer
Motivation:

At the moment we call initChannel(...) in the channelRegistered(...) method which has the effect that if another ChannelInitializer is added within the initChannel(...) method the ordering of the added handlers is not correct and surprising. This is as the whole initChannel(...) method block is executed before the initChannel(...) block of the added ChannelInitializer is handled.

Modifications:

Call initChannel(...) from within handlerAdded(...) if the Channel is registered already. This is true in all cases for our DefaultChannelPipeline implementation. This way the ordering is always as expected. We still keep the old behaviour as well to not break code for other ChannelPipeline implementations (if someone ever wrote one).

Result:

Correct and expected ordering of ChannelHandlers.
2016-08-03 07:50:46 +02:00
Norman Maurer
f585806a74 [#5598] Ensure SslHandler not log false-positives when try to close the channel due timeout.
Motivation:

When we try to close the Channel due a timeout we need to ensure we not log if the notification of the promise fails as it may be completed in the meantime.

Modifications:

Add another constructor to ChannelPromiseNotifier and PromiseNotifier which allows to log on notification failure.

Result:

No more miss-leading logs.
2016-07-30 21:15:09 +02:00
Norman Maurer
cb7cf4491c [maven-release-plugin] prepare for next development iteration 2016-07-27 13:29:56 +02:00
Norman Maurer
9466b32d05 [maven-release-plugin] prepare release netty-4.1.4.Final 2016-07-27 13:16:59 +02:00
Norman Maurer
b97a36a10f Add test to verify that its possible to add another ChannelInitializer in the initChannel(...) method.
Motivation:

I received a report the its not possible to add another ChannelInitialiter in the initChannel(...) method, so we should add a test case for it.

Modifications:

Added testcase.

Result:

Validate that all works as expected.
2016-07-27 09:25:13 +02:00
Norman Maurer
4638df2062 [#5566] Ensure using a ChannelInitializer via ServerBootstrap.handler(...) produce correct ordering.
Motivation:

When a ChannelInitializer is used via ServerBootstrap.handler(...) the users handlers may be added after the internal ServerBootstrapAcceptor. This should not happen.

Modifications:

Delay the adding of the ServerBootstrapAcceptor until the initChannel(....) method returns.

Result:

Correct order of handlers in the ServerChannels ChannelPipeline.
2016-07-27 08:34:31 +02:00
Norman Maurer
dc6c6d956a [#5541] Ensure failing a Promise in SimpleChannelPool will not result in stack overflow.
Motivation:

We used Promise.setFailure(...) when fail a Promise in SimpleChannelPool. As this happens in multiple levels this can result in stackoverflow as setFailure(...) may throw an IllegalStateException which then again is propergated.

Modifications:

Use tryFailure(...)

Result:

No more possibility to cause a stack overflow when failing the promise.
2016-07-27 07:57:58 +02:00
Norman Maurer
b8400f9628 [#5553] SimpleChannelPool#notifyConnect() may leak Channels
Motivation:

The SimpleChannelPool#notifyConnect() method will leak Channels if the user cancelled the Promise in between.

Modifications:

Release the channel if the Promise was complete before.

Result:

No more channel leaks.
2016-07-20 20:17:19 +02:00
Jason Tedor
d262f7c189 Reduce permissions needed for process ID
Motiviation:

DefaultChannelId attempts to acquire a default process ID by determining
the process PID. However, to do this it attempts to punch through to the
system classloader, a permission that in the face of a restrictive
security manager is unlikely to be granted. Looking past this, it then
attempts to load a declared method off a reflectively loaded class,
another permission that is not likely to be granted in the face of a
restrictive security manager. However, neither of these permissions are
necessary as the punching through to the system security manager is
completely unneeded, and there is no need to load a public method as a
declared method.

Modifications:

Instead of punching through to the system classloader requiring
restricted permissions, we can just use current classloader. To address
the access declared method permission, we instead just reflectively
obtain the desired public method via Class#getMethod.

Result:

Acquiring the default process ID from the PID will succeed without
requiring the runtime permissions "getClassLoader" and
"accessDeclaredMembers".
2016-07-20 19:47:56 +02:00
Scott Mitchell
3d7ae97359 Make Epoll ChannelMetadata more consistent with NIO
Motivation:
In 4.0 AbstractNioByteChannel has a default of 16 max messages per read. However in 4.1 that constraint was applied at the NioSocketChannel which is not equivalent. In 4.1 AbstractEpollStreamChannel also did not have the default of 16 max messages per read applied.

Modifications:
- Make Nio consistent with 4.0
- Make Epoll consistent with Nio

Result:
Nio and Epoll both have consistent ChannelMetadata and are consistent with 4.0.
2016-07-18 13:26:05 +02:00
Norman Maurer
047f6aed28 [maven-release-plugin] prepare for next development iteration 2016-07-15 09:09:13 +02:00
Norman Maurer
b2adea87a0 [maven-release-plugin] prepare release netty-4.1.3.Final 2016-07-15 09:08:53 +02:00
Nitesh Kant
77770374fb Ability to run a task at the end of an eventloop iteration.
Motivation:

This change is part of the change done in PR #5395 to provide an `AUTO_FLUSH` capability.
Splitting this change will enable to try other ways of implementing `AUTO_FLUSH`.

Modifications:



Two methods:

```java
void executeAfterEventLoopIteration(Runnable task);


boolean removeAfterEventLoopIterationTask(Runnable task);
```
are added to `SingleThreadEventLoop` class for adding/removing a task to be executed at the end of current/next iteration of this `eventloop`.

In order to support the above, a few methods are added to `SingleThreadEventExecutor`

```java
protected void afterRunningAllTasks() { }
```

This is invoked after all tasks are run for this executor OR if the passed timeout value for `runAllTasks(long timeoutNanos)` is expired.

Added a queue of `tailTasks` to `SingleThreadEventLoop` to hold all tasks to be executed at the end of every iteration.


Result:



`SingleThreadEventLoop` now has the ability to execute tasks at the end of an eventloop iteration.
2016-07-12 10:22:15 +02:00
Norman Maurer
b37a41a535 Allow to get the number of bytes queued in PendingWriteQueue
Motivation:

For some use-cases it would be useful to know the number of bytes queued in the PendingWriteQueue without the need to dequeue them.

Modifications:

Add PendingWriteQueue.bytes().

Result:

Be able to get the number of bytes queued.
2016-07-11 09:05:20 +02:00
Norman Maurer
50a74e95f2 Ensure ChannelHandler.handlerAdded(...) callback is executed directly when added from ChannelFutureListener added to the registration future.
Motivation:

Commit 4c048d069d moved the logic of calling handlerAdded(...) to the channelRegistered(...) callback of the head of the DefaultChannelPipeline. Unfortunatlly this may execute the callbacks to late as a user may add handlers to the pipeline in the ChannelFutureListener attached to the registration future. This can lead to incorrect ordering.

Modifications:

Ensure we always invoke ChannelHandler.handlerAdded(...) for all handlers before the registration promise is notified.

Result:

Not possible of incorrect ordering or missed events.
2016-07-09 08:04:15 +02:00
Norman Maurer
061899f2a7 Allow to remove pinning of EventExecutor for EventExecutorGroup
Motivation:

We pinned the EventExecutor for a Channel in DefaultChannelPipeline. Which means if the user added multiple handlers with the same EventExecutorGroup to the ChannelPipeline it will use the same EventExecutor for all of these handlers. This may be unexpected and even not what the user wants. If the user want to use the same one for all of them it can be done by obtain an EventExecutor and pass the same instance to the add methods. Because of this we should allow to not pin.

Modifications:

Allow to disable pinning of EventExecutor for Channel based on EventExecutorGroup via ChannelOption.

Result:

Less confusing and more flexible usage of EventExecutorGroup when adding ChannelHandlers to the ChannelPipeline.
2016-07-08 20:09:16 +02:00
Norman Maurer
64bf167423 Fix flacky test introducd by 29fdb160f3 2016-07-07 22:53:08 +02:00
Roger Kapsi
02850da480 Rename ChannelHandlerContext#fireUserEventTriggered() argument from event to evt so it matches the ChannelInboundHandler#userEventTriggered() argument's name.
Motivation

When I override ChannelHandler methods I usually (always) refire events myself via
ChannelHandlerContext instead of relieing on calling the super method (say
`super.write(ctx, ...)`). This works great and the IDE actually auto completes/generates
the right code for it except `#fireUserEventTriggered()` and `#userEventTriggered()`
which have a mismatching argument names and I have to manually "intervene".

Modification

Rename `ChannelHandlerContext#fireUserEventTriggered()` argument from `event` to `evt`
to match its handler counterpart.

Result

The IDE's auto generated code will reference the correct variable.
2016-07-07 17:01:19 +02:00
Norman Maurer
29fdb160f3 [#5486] Not operate on serial execution assumption when using EventExecutor in the DefaultChannelPipeline.
Motivation:

In commit f984870ccc I made a change which operated under invalide assumption that tasks executed by an EventExecutor will always be processed in a serial fashion. This is true for SingleThreadEventExecutor sub-classes but not part of the EventExecutor interface contract.

Because of this change implementations of EventExecutor which not strictly execute tasks in a serial fashion may miss events before handlerAdded(...) is called. This is strictly speaking not correct as there is not guarantee in this case that handlerAdded(...) will be called as first task (as there is no ordering guarentee).

Cassandra itself ships such an EventExecutor implementation which has no strict ordering to spread load across multiple threads.

Modifications:

- Add new OrderedEventExecutor interface and let SingleThreadEventExecutor / EventLoop implement / extend it.
- Only expose "restriction" of skipping events until handlerAdded(...) is called for OrderedEventExecutor implementations
- Add ThreadPoolEventExecutor implementation which executes tasks in an unordered fashion. This is used in added unit test but can also be used for protocols which not expose an strict ordering.
- Add unit test.

Result:

Resurrect the possibility to implement an EventExecutor which does not enforce serial execution of events and be able to use it with the DefaultChannelPipeline.
2016-07-07 15:01:56 +02:00
Norman Maurer
c393374cf5 [#5455] Clarify ChannelPool javadocs
Motivation:

We should make it clear that each acquired Channel needs to be released in all cases.

Modifications:

More clear javadocs.

Result:

Harder for users to leak Channel.
2016-07-07 06:46:28 +02:00
Norman Maurer
195d7476d5 Deprecate methods in AbstractChannel that have no real usage.
Motivation:

We should deprecate methods that are not used.

Modifications:

Add @Deprecated to methods

Result:

Be able to cleanup stuff sooner.
2016-07-07 06:41:43 +02:00
buchgr
0fbb791ad6 Make AbstractChannel.outboundBuffer volatile.
Motivation:

The field can be read from arbitrary threads via Channel.(isWritable()|bytesBeforeWritable()|bytesBeforeUnwritable()), WriteAndFlushTask.newInstance(), PendingWriteQueue, etc.

Modifications:

Make AbstractChannel.outboundBuffer volatile.

Result:

More correct in a concurrent use case.
2016-07-06 21:25:33 +02:00
Norman Maurer
44b942e507 Rename future to promise in the ChannelDuplexHandler method arguments.
Motivation:

We used future in many method of ChannelDuplexHandler as argument name of ChannelPromise. We should make it more consistent and correct.

Modifications:

Replace future with promise.

Result:

More correct and consistent naming.
2016-07-06 08:55:29 +02:00