Commit Graph

1917 Commits

Author SHA1 Message Date
Norman Maurer
c3637ff42c
AUTO_CLOSE should not be marked as deprecated. (#7967)
Motivation:

A long time ago we deprecated AUTO_CLOSE but it turned out this feature is still useful because if a write error is detected there still maybe data to read, and if we close the channel automatically we will lose data

Modifications:

- Remove `@Deprecated` tag for AUTO_CLOSE, setAutoClose(...) and isAutoClose(...)
- Fix javadocs on ChannelConfig to correctly tell the default value of AUTO_CLOSE.

Result:

Less warnings.
2018-05-24 21:25:30 +02:00
Norman Maurer
583fc272f2
Fixed|SimpleChannelPool.close() should only return after complete. (#7927)
Motivation:

We need to ensure we only return from close() after all work is done as otherwise we may close the EventExecutor before we dispatched everything.

Modifications:

Correctly wait on operations to complete before return.

Result:

Fixes https://github.com/netty/netty/issues/7901.
2018-05-21 19:22:31 +02:00
Norman Maurer
64bb279f47 [maven-release-plugin] prepare for next development iteration 2018-05-14 11:11:45 +00:00
Norman Maurer
c67a3b0507 [maven-release-plugin] prepare release netty-4.1.25.Final 2018-05-14 11:11:24 +00:00
Norman Maurer
cbe9ed8cc1
Add test that we handle thread.interrupt() in NioEventLoop (#7917)
Motivation:

We added some code to guard against thread.interrupt() in NioEventLoop but did not added a test.

Modifications:

Add testcase.

Result:

Verify that we correctly handle interrupt().
2018-05-09 08:57:53 +02:00
Konstantin Lutovich
437849bd93 Fix tiny typo in FixedChannelPool error message (#7918)
Motivation:

Closed `FixedChannelPool` fails acquire and release operations with
`IllegalStateException`s. These exceptions had message
"FixedChannelPooled was closed". Here "FixedChannelPooled" looks like
a typo and should probably be "FixedChannelPool".

Modifications:

Changed exception message to "FixedChannelPool was closed".

Result:

A tiny bit cleaner exception message.
2018-05-08 15:58:23 +02:00
Norman Maurer
095dadcbec
Fix race in ChannelReadHandler used during LocalChannel testing. (#7904)
Motivation:

ChannelReadHandler is used in tests added via f4d7e8de14. In the handler we verify the number of messages we receive per read() call but missed to sometimes reset the counter which resulted in exceptions.

Modifications:

Correctly reset read counter in all cases.

Result:

No more unexpected exceptions when running LocalChannel tests.
2018-05-04 07:36:33 +02:00
Norman Maurer
f4d7e8de14
Respect MAX_MESSAGES_PER_READ in LocalChannel / LocalServerChannel. (#7885)
Motivation:

LocalChannel / LocalServerChannel did not respect read limits and just always read all of the messages.

Modifications:

- Correct respect MAX_MESSAGES_PER_READ settings
- Add unit tests

Result:

Fixes https://github.com/netty/netty/issues/7880.
2018-04-26 07:58:56 +02:00
Norman Maurer
b47fb81799
EventLoop.schedule with big delay fails (#7402)
Motivation:

Using a very huge delay when calling schedule(...) may cause an Selector error when calling select(...) later on. We should gaurd against such a big value.

Modifications:

- Add guard against a very huge value.
- Added tests.

Result:

Fixes [#7365]
2018-04-24 11:15:20 +02:00
Norman Maurer
b75f44db9a [maven-release-plugin] prepare for next development iteration 2018-04-19 11:56:07 +00:00
Norman Maurer
04fac00c8c [maven-release-plugin] prepare release netty-4.1.24.Final 2018-04-19 11:55:47 +00:00
Norman Maurer
0f34d887b7
Only reset readIsPending if outboundBuffer is not empty. (#7874)
Motivation:

We need to ensure we only reset readInProgress if the outboundBuffer is not empty as otherwise we may miss to call fireChannelRead(...) later on when using the LocalChannel.

Modifications:

Also check if the outboundBuffer is not empty before setting readInProgress to false again

Result:

Fixes https://github.com/netty/netty/issues/7855
2018-04-19 09:39:16 +02:00
Nikolay Fedorovskikh
401b196623 Extract common parts from if statements (#7831)
Motivation:
Some `if` statements contains common parts that can be extracted.

Modifications:
Extract common parts from `if` statements.

Result:
Less code and bytecode. The code is simpler and more clear.
2018-04-11 14:36:56 +02:00
root
0a61f055f5 [maven-release-plugin] prepare for next development iteration 2018-04-04 10:44:46 +00:00
root
8c549bad38 [maven-release-plugin] prepare release netty-4.1.23.Final 2018-04-04 10:44:15 +00:00
Scott Mitchell
ed0668384b NIO read spin event loop spin when half closed (#7801)
Motivation:
AbstractNioByteChannel will detect that the remote end of the socket has
been closed and propagate a user event through the pipeline. However if
the user has auto read on, or calls read again, we may propagate the
same user events again. If the underlying transport continuously
notifies us that there is read activity this will happen in a spin loop
which consumes unnecessary CPU.

Modifications:
- AbstractNioByteChannel's unsafe read() should check if the input side
of the socket has been shutdown before processing the event. This is
consistent with EPOLL and KQUEUE transports.
- add unit test with @normanmaurer's help, and make transports consistent with respect to user events

Result:
No more read spin loop in NIO when the channel is half closed.
2018-03-28 20:02:57 +02:00
Dmitriy Dumanskiy
62d8a5e9d2 Add removeIfExists() method to DefaultChannelPipeline
Motivation:

Sometimes it is very convenient to remove the handler from pipeline without throwing the exception in case those handler doesn't exist in the pipeline.

Modification:

Added 3 overloaded methods to DefaultChannelPipeline, but not added to ChannelHandler due to back compatibility.

Result:

Fixes #7662
2018-03-27 09:48:52 +02:00
Norman Maurer
fc3b145cbb Correctly handle non IOException during read in NioServerSocketChannel
Motivation:

Our code was not correct in AbstractNioMessageChannel.closeOnReadError(....) which lead to the situation that we always tried to continue reading no matter what exception was thrown when using the NioServerSocketChannel. Also even on an IOException we should check if the Channel itself is still active or not and if not stop reading.

Modifications:

Fix closeOnReadError impl and added test.

Result:

Correctly stop reading on NioServerSocketChannel when error happens during read.
2018-03-25 17:31:59 +02:00
Norman Maurer
0adccfdb50 Simplify DefaultChannelGroup.contains(...) and so remove one instanceof check.
Motivation:

DefaultChannelGroup.contains(...) did one more instanceof check then needed.

Modifications:

Simplify contains(...) and remove one instanceof check.

Result:

Simplier and cheaper implementation.
2018-03-17 09:45:24 +01:00
Alexey Kachayev
b0823761f4 PendingWriteQueue to handle write operations with void future
Motivation:

Right now PendingWriteQueue.removeAndWriteAll collects all promises to
PromiseCombiner instance which sets listener to each given promise throwing
IllegalStateException on VoidChannelPromise which breaks while loop
and "reports" operation as failed (when in fact part of writes might be
actually written).

Modifications:

Check if the promise is not void before adding it to the PromiseCombiner
instance.

Result:

PendingWriteQueue.removeAndWriteAll succesfully writes all pendings
even in case void promise was used.
2018-03-16 08:23:40 +01:00
Norman Maurer
0a8e1aaf19 Flush task should not flush messages that were written since last flush attempt.
Motivation:

The flush task is currently using flush() which will have the affect of have the flush traverse the whole ChannelPipeline and also flush messages that were written since we gave up flushing. This is not really correct as we should only continue to flush messages that were flushed at the point in time when the flush task was submitted for execution if the user not explicit call flush() by him/herself.

Modification:

Call *Unsafe.flush0() via the flush task which will only continue flushing messages that were marked as flushed before.

Result:

More correct behaviour when the flush task is used.
2018-03-02 10:09:40 +09:00
Norman Maurer
69582c0b6c [maven-release-plugin] prepare for next development iteration 2018-02-21 12:52:33 +00:00
Norman Maurer
786f35c6c9 [maven-release-plugin] prepare release netty-4.1.22.Final 2018-02-21 12:52:19 +00:00
Scott Mitchell
ce241bd11e Epoll flush/writabilityChange deadlock
Motivation:
b215794de3 recently introduced a change in behavior where writeSpinCount provided a limit for how many write operations were attempted per flush operation. However when the write quantum was meet the selector write flag was not cleared, and the channel unsafe flush0 method has an optimization which prematurely exits if the write flag is set. This may lead to no write progress being made under the following scenario:
- flush is called, but the socket can't accept all data, we set the write flag
- the selector wakes us up because the socket is writable, we write data and use the writeSpinCount quantum
- we then schedule a flush() on the EventLoop to execute later, however it the flush0 optimization prematurely exits because the write flag is still set

In this scenario the socket is still writable so the EventLoop may never notify us that the socket is writable, and therefore we may never attempt to flush data to the OS.

Modifications:
- When the writeSpinCount quantum is exceeded we should clear the selector write flag

Result:
Fixes https://github.com/netty/netty/issues/7729
2018-02-20 11:40:58 +01:00
Shohei Kamimori
73f23c5faa Fix typos in docs.
Motivation:

There are same typos in the docs.

Modifications:

Fix typos. Docs only changing.

Result:

More correct docs.
2018-02-14 08:44:07 +01:00
Scott Mitchell
7928a6fef3
NioDatagramChannel invalid usage of internalNioBuffer
Motivation:
NioDatagramChannel attempts to unpack a AddressedEnvelope and unconditionally uses internalNioBuffer. However if the ByteBuf is a CompositeByteBuf with more than 1 components, the write will fail and throw an exception.

Modifications:
- NioDatagramChannel should check the nioBufferCount before attempting
to use internalNioBuffer

Result:
No more failure to write UDP packets on NIO when a CompositeByteBuf is
used.
2018-02-12 09:31:36 -08:00
Norman Maurer
e71fa1e7b6 [maven-release-plugin] prepare for next development iteration 2018-02-05 12:02:35 +00:00
Norman Maurer
41ebb5fcca [maven-release-plugin] prepare release netty-4.1.21.Final 2018-02-05 12:02:19 +00:00
Norman Maurer
e72c197aa3 Reflective setAccessible(true) will produce scary warnings on the console when using java9+, dont do it
Motivation:

Reflective setAccessible(true) will produce scary warnings on the console when using java9+, while netty still works. That said users may feel uncomfortable with these warnings, we should not try to do it by default when using java9+.

Modifications:

Add io.netty.tryReflectionSetAccessible  system property which controls if setAccessible(...) will be used. By default it will bet set to false when using java9+.

Result:

Fixes [#7254].
2018-01-30 12:18:34 +01:00
Jason
9dd5c928f3 Add java-doc for implemented methods of io.netty.util.concurrent.Future#cancel(boolean mayInterruptIfRunning)
Motivation:

The methods implement io.netty.util.concurrent.Future#cancel(boolean mayInterruptIfRunning) which actually ignored the param mayInterruptIfRunning.We need to add comments for the `mayInterruptIfRunning` param.

Modifications:

Add comments for the `mayInterruptIfRunning` param.

Result:

People who call the `cancel` method will be more clear about the effect of `mayInterruptIfRunning` param.
2018-01-29 11:19:52 +01:00
Norman Maurer
36304e1f05 Reduce object allocation by using same ChannelFutureListener instance.
Motivation:

When VoidChannelPromise.unvoid() was called we created a new ChannelFutureListener everytime. This is not needed as its stateless.

Modifications:

Reuse the ChannelFutureListener.

Result:

Less object allocations
2018-01-26 08:42:39 +01:00
Scott Mitchell
2d815fa752 DefaultChannelPipeline will not invoke handler if events are fired from handlerAdded
Motiviation:
DefaultChannelPipeline and AbstractChannelHandlerContext maintain state
which indicates if a ChannelHandler should be invoked or not. However
the state is updated to allow the handler to be invoked only after the
handlerAdded method completes. If the handlerAdded method generates
events which may result in other methods being invoked on that handler
they will be missed.

Modifications:
- DefaultChannelPipeline should set the state before calling
handlerAdded

Result:
DefaultChannelPipeline will allow events to be processed during the
handlerAdded process.
2018-01-24 10:32:48 +01:00
Norman Maurer
ea58dc7ac7 [maven-release-plugin] prepare for next development iteration 2018-01-21 12:53:51 +00:00
Norman Maurer
96c7132dee [maven-release-plugin] prepare release netty-4.1.20.Final 2018-01-21 12:53:34 +00:00
Norman Maurer
1740f366eb Fail fast when DefaultChannelPromise is constructed with null as Channel.
Motivation:

We should fail fast when DefaultChannelPromise is constructed with null as Channel as otherwise it will fail with a NPE once we call setSuccess / setFailure.

Modifications:

Add null check and test.

Result:

Fail fast.
2018-01-18 18:57:42 +00:00
Dmitriy Dumanskiy
e6c9ac968d Cleanup: replaced deprecated ctx.attr() and ctx.hasAttr() methods usage with ch.attr() and ch.hasAttr().
Motivation:

Will allow easy removal of deprecated methods in future.

Modification:

Replaced ctx.attr(), ctx.hasAttr() with ctx.channel().attr(), ctx.channel().hasAttr().

Result:

No deprecated ctx.attr(), ctx.hasAttr() methods usage.
2018-01-18 15:00:41 +00:00
Norman Maurer
0c5014b105 Add a hint of ownership transfer when calling EmbeddedChannel.read*() methods.
Motivation:

As shown in issues it is sometimes hard to understand why a leak was reported when the user just calles EmbeddedChannel.readInbound() / EmbeddedChannel.readOutbound() and drop the message on the floor.

Modifications:

Add a hint before handover the message to the user and transfer the ownership.

Result:

Easier debugging of leaks caused by EmbeddedChannel.read*().
2017-12-20 07:44:25 +01:00
Dmitriy Dumanskiy
f9888acfdd added overloaded method to the DefaultChannelPipeline in order to avoid unnecessary allocation
Motivation :

Avoid unnecessary array allocation when using the function with varargs in the DefaultChannelPipeline class.

Modifications :

Added addLast and addFirst overloaded methods with 1 handler instead of varargs.

Result :

No array allocation when using simple construction like pipeline.addLast(new Handler());
2017-12-18 09:34:05 +01:00
Roger Kapsi
79ed1c6871 Ability to scoop up events that reach the tail of the ChannelPipeline.
Motivation

There is currently no way to enforce the position of a handler in a ChannelPipeline and assume you wanted to write something like a custom Channel type that acts as a proxy between two other Channels.

ProxyChannel(Channel client, Channel server) {
  client calls write(msg) -> server.write(msg)
  client calls flush() -> server.flush()
  server calls fireChannelRead(msg) -> client.write(msg)
  server calls fireChannelReadComplete() -> client.flush()
}

In order to make it work reliably one needs to be able to scoop up the various events at the head and tail of the pipeline. The head side of the pipeline is covered by Unsafe and it's also relatively safe to count on the user to not use the addFirst() method to manipulate the pipeline. The tail side is always at a risk of getting broken because addLast() is the goto method to add handlers.

Modifications

Adding a few extra methods to DefaultChannelPipeline that expose some of the events that reach the pipeline's TailContext.

Result

Fixes #7484
2017-12-15 21:19:16 +01:00
Norman Maurer
264a5daa41 [maven-release-plugin] prepare for next development iteration 2017-12-15 13:10:54 +00:00
Norman Maurer
0786c4c8d9 [maven-release-plugin] prepare release netty-4.1.19.Final 2017-12-15 13:09:30 +00:00
Norman Maurer
b2bc6407ab [maven-release-plugin] prepare for next development iteration 2017-12-08 09:26:15 +00:00
Norman Maurer
96732f47d8 [maven-release-plugin] prepare release netty-4.1.18.Final 2017-12-08 09:25:56 +00:00
louxiu
805ac002e6 FIX: force a read operation for peer instead of self (#7454)
* FIX: force a read operation for peer instead of self

Motivation:
When A is in `writeInProgress` and call self close, A should
`finishPeerRead` for B(A' peer).

Modifications:
Call `finishPeerRead` with peer in `LocalChannel#doClose`

Result:
Clear confuse of code logic

* FIX: preserves order of close after write in same event loop

Motivation:
If client and server(client's peer channel) are in same event loop, client writes data to
server in `ChannelActive`. Server receives the data and write it
back. The client's read can't be triggered becasue client's
`ChannelActive` is not finished at this point and its `readInProgress`
is false. Then server closes itself, it will also close the client's
channel. And client has no chance to receive the data.

Modifications:
1. Add a test case to demonstrate the problem
2. When `doClose` peer, we always call
`peer.eventLoop().execute()` and `registerInProgress` is not needed.
3. Remove test case
`testClosePeerInWritePromiseCompleteSameEventLoopPreservesOrder`. This
test case can't pass becasue of this commit. IMHO, I think it is OK,
becasue it is reasonable that the client flushes the data to socket,
then server close the channel without received the data.
4. For mismatch test in SniClientTest, the client should receive server's alert before closed(caused by server's close)

Result:
The problem is gone.
2017-12-07 17:05:57 -08:00
Scott Mitchell
b215794de3
Enforce writeSpinCount to limit resource consumption per socket (#7478)
Motivation:
The writeSpinCount currently loops over the same buffer, gathering
write, file write, or other write operation multiple times but will
continue writing until there is nothing left or the OS doesn't accept
any data for that specific write. However if the OS keeps accepting
writes there is no way to limit how much time we spend on a specific
socket. This can lead to unfair consumption of resources dedicated to a
single socket.
We currently don't limit the amount of bytes we attempt to write per
gathering write. If there are many more bytes pending relative to the
SO_SNDBUF size we will end up building iov arrays with more elements
than can be written, which results in extra iteration, conditionals,
and book keeping.

Modifications:
- writeSpinCount should limit the number of system calls we make to
write data, instead of applying to individual write operations
- IovArray should support a maximum number of bytes
- IovArray should support composite buffers of greater than size 1024
- We should auto-scale the amount of data that we attempt to write per
gathering write operation relative to SO_SNDBUF and how much data is
successfully written
- The non-unsafe path should also support a maximum number of bytes,
and respect the IOV_MAX limit

Result:
Write resource consumption can be bounded and gathering writes have
a limit relative to the amount of data which can actually be accepted
by the socket.
2017-12-07 16:00:52 -08:00
Scott Mitchell
95b02e49ac DefaultMaxMessagesRecvByteBufAllocator support an option to ignore maybeMoreData
Motivation:
If large amounts of data is being transferred it is difficult to correlate the amount we attempt to read vs the maximum amount that the OS will actually buffer and deliver to the application. For exmaple some OSes may dynicamlly update the SO_RCVBUF size or otherwise dynamically adjust how much data is delieved to the application. In these circumstances it can reduce latency to just call read() on the socket another time to see if there is really any data remaining instead of giving up the maxMessagesPerRead quantum and going back to the selector to read later.

Motifications:
- Add DefaultMaxMessagesRecvByteBufAllocator#respectMaybeMoreData which provides a way to ignore the maybeMoreData function which may not account for the current data pending, and if it does this maybe racy.

Result:
Option to always use the full maxMessagesPerRead quantum before going back to the selector.
2017-12-04 11:03:58 -08:00
Scott Mitchell
d7c977dd71 SslHandler aggregation prefer copy over CompositeByteBuf
Motivation:
SslHandler will do aggregation of writes by default in an attempt to improve goodput and reduce the number of discrete buffers which must be accumulated. However if aggregation is not possible then a CompositeByteBuf is used to accumulate multiple buffers. Using a CompositeByteBuf doesn't provide any of the benefits of better goodput and in the case of small + large writes (e.g. http/2 frame header + data) this can reduce the amount of data that can be passed to writev by about half. This has the impact of increasing latency as well as reducing goodput.

Modifications:
- SslHandler should prefer copying instead of using a CompositeByteBuf

Result:
Better goodput (and potentially improved latency) at the cost of copy operations.
2017-12-04 11:02:33 -08:00
Scott Mitchell
ec368fea47 AdaptiveRecvByteBufAllocator should ramp up while reading
Motivation:
AdaptiveRecvByteBufAllocator currently adjusts the ByteBuf allocation size guess when readComplete is called. However the default configuration for number of reads before readComplete is called is 16. This means that there will be 16 reads done before any adjustment is done. If there is a large amount of data pending AdaptiveRecvByteBufAllocator will be slow to adjust the allocation size guess. In addition to being slow the result of only updating the guess in readComplete means that we must go back to the selector and wait to be woken up again when data is ready to read. Going back to the selector is an expensive operations and can add significant latency if there is large amount of data pending to read.

Modifications:
- AdaptiveRecvByteBufAllocator should check on each read if a step up is necessary. The step down process is left unchanged and can be more gradual at the cost of potentially over allocating.

Result:
AdaptiveRecvByteBufAllocator increases the guess size during the read loop to reduce latency when large amounts of data is being read.
2017-12-03 17:25:31 -08:00
Tomasz Jędrzejewski
e8540c2b7a Adding stable JDK9 module names that follow reverse-DNS style
Automatic-Module-Name entry provides a stable JDK9 module name, when Netty is used in a modular JDK9 applications. More info: http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html

When Netty migrates to JDK9 in the future, the entry can be replaced by actual module-info descriptor.

Modification:

The POM-s are configured to put the correct module names to the manifest.

Result:

Fixes #7218.
2017-11-29 11:50:24 +01:00
lutovich
25d146038c Improved error message in FixedChannelPool
Motivation:

`FixedChannelPool` allows users to configure `acquireTimeoutMillis`
and expects given value to be greater or equal to zero when timeout
action is supplied. However, validation error message said that
value is expected to be greater or equal to one. Code performs
check against zero.

Modifications:

Changed error message to say that value greater or equal to
zero is expected. Added test to check that zero is an acceptable
value.

Result:

Exception with right error message is thrown.
2017-11-13 20:29:07 +01:00
Norman Maurer
188ea59c9d [maven-release-plugin] prepare for next development iteration 2017-11-08 22:36:53 +00:00
Norman Maurer
812354cf1f [maven-release-plugin] prepare release netty-4.1.17.Final 2017-11-08 22:36:33 +00:00
Scott Mitchell
7511c15187 AbstractCoalescingBufferQueue addFirst void promise handling
Motivation:
AbstractCoalescingBufferQueue#add accounts for void promises, but AbstractCoalescingBufferQueue#addFirst does not. These methods should be consistent.

Modifications:
- AbstractCoalescingBufferQueue#addFirst should account for void promises and share code with AbstractCoalescingBufferQueue#add

Result:
More correct void promise handling in AbstractCoalescingBufferQueue.
2017-11-07 11:33:53 -08:00
Scott Mitchell
8c5eeb581e SslHandler promise completion incorrect if write doesn't immediately
complete

Motivation:
SslHandler removes a Buffer/Promise pair from
AbstractCoalescingBufferQueue when wrapping data. However it is possible
the SSLEngine will not consume the entire buffer. In this case
SslHandler adds the Buffer back to the queue, but doesn't add the
Promise back to the queue. This may result in the promise completing
immediately in finishFlush, and generally not correlating to the
completion of writing the corresponding Buffer

Modifications:
- AbstractCoalescingBufferQueue#addFirst should also support adding the
ChannelPromise
- In the event of a handshake timeout we should immediately fail pending
writes immediately to get a more accurate exception

Result:
Fixes https://github.com/netty/netty/issues/7378.
2017-11-07 09:24:40 -08:00
Norman Maurer
e7f02b1dc0 Set readPending to false when EOF is detected while issue an read
Motivation:

We need to set readPending to false when we detect a EOF while issue a read as otherwise we may not unregister from the Selector / Epoll / KQueue and so keep on receving wakeups.

The important bit is that we may even get a wakeup for a read event but will still will only be able to read 0 bytes from the socket, so we need to be very careful when we clear the readPending. This can happen because we generally using edge-triggered mode for our native transports and because of the nature of edge-triggered we may schedule an read event just to find out there is nothing left to read atm (because we completely drained the socket on the previous read).

Modifications:

Set readPending to false when EOF is detected.

Result:

Fixes [#7255].
2017-11-06 15:44:36 -08:00
Scott Mitchell
35b0cd58fb HTTP/2 write of released buffer should not write and should fail the promise
Motivation:
HTTP/2 allows writes of 0 length data frames. However in some cases EMPTY_BUFFER is used instead of the actual buffer that was written. This may mask writes of released buffers or otherwise invalid buffer objects. It is also possible that if the buffer is invalid AbstractCoalescingBufferQueue will not release the aggregated buffer nor fail the associated promise.

Modifications:
- DefaultHttp2FrameCodec should take care to fail the promise, even if releasing the data throws
- AbstractCoalescingBufferQueue should release any aggregated data and fail the associated promise if something goes wrong during aggregation

Result:
More correct handling of invalid buffers in HTTP/2 code.
2017-11-06 14:38:58 -08:00
Norman Maurer
bcad9dbf97 Revert "Set readPending to false when ever a read is done"
This reverts commit 413c7c2cd8 as it introduced an regression when edge-triggered mode is used which is true for our native transports by default. With 413c7c2cd8 included it was possible that we set readPending to false by mistake even if we would be interested in read more.
2017-11-06 09:21:42 -08:00
Scott Mitchell
413c7c2cd8 Set readPending to false when ever a read is done
Motivation:
readPending is currently only set to false if data is delivered to the application, however this may result in duplicate events being received from the selector in the event that the socket was closed.

Modifications:
- We should set readPending to false before each read attempt for all
transports besides NIO.
- Based upon the Javadocs it is possible that NIO may have spurious
wakeups [1]. In this case we should be more cautious and only set
readPending to false if data was actually read.

[1] https://docs.oracle.com/javase/7/docs/api/java/nio/channels/SelectionKey.html
That a selection key's ready set indicates that its channel is ready for some operation category is a hint, but not a guarantee, that an operation in such a category may be performed by a thread without causing the thread to block.

Result:
Notification from the selector (or simulated events from kqueue/epoll ET) in the event of socket closure.
Fixes https://github.com/netty/netty/issues/7255
2017-10-25 08:25:54 -07:00
Norman Maurer
55b501d0d4 Correctly update Channel writability when queueing data in SslHandler.
Motivation:

A regression was introduced in 86e653e which had the effect that the writability was not updated for a Channel while queueing data in the SslHandler.

Modifications:

- Factor out code that will increment / decrement pending bytes and use it in AbstractCoalescingBufferQueue and PendingWriteQueue
- Add test-case

Result:

Channel writability changes are triggered again.
2017-10-24 09:13:15 +02:00
Idel Pivnitskiy
558097449c Add missed 'serialVersionUID' field for Serializable classes
Motivation:

Without a 'serialVersionUID' field, any change to a class will make
previously serialized versions unreadable.

Modifications:

Add missed 'serialVersionUID' field for all Serializable
classes.

Result:

Proper deserialization of previously serialized objects.
2017-10-21 14:41:18 +02:00
Sky Ao
47f016bf56 extract self() method to avoid too many unchecked @SuppressWarnings
Motivation:

There are many @SuppressWarnings("unchecked") in the code for the same purpose that we want to do this return:

@SuppressWarnings("unchecked")
public B someMethod() {
    ......
    return (B) this;
}
Modification:

Add a method self() and reuse in all these return lines:

@SuppressWarnings("unchecked")
    private B self() {
        return (B) this;
    }
Result:

Then only one @SuppressWarnings("unchecked") left in the code.
2017-09-27 13:25:16 +02:00
Norman Maurer
625a7426cd [maven-release-plugin] prepare for next development iteration 2017-09-25 06:12:32 +02:00
Norman Maurer
f57d8f00e1 [maven-release-plugin] prepare release netty-4.1.16.Final 2017-09-25 06:12:16 +02:00
Norman Maurer
aa8bdb5d6b Fix assertion error when closing / shutdown native channel and SO_LINGER is set.
Motivation:

When SO_LINGER is used we run doClose() on the GlobalEventExecutor by default so we need to ensure we schedule all code that needs to be run on the EventLoop on the EventLoop in doClose. Beside this there are also threading issues when calling shutdownOutput(...)

Modifications:

- Schedule removal from EventLoop to the EventLoop
- Correctly handle shutdownOutput and shutdown in respect with threading-model
- Add unit tests

Result:

Fixes [#7159].
2017-09-18 14:46:37 -07:00
Nikolay Fedorovskikh
e404690574 Optimize DefaultChannelId.equals
Motivation:
A `DefaultChannelId` has final `hashCode` field calculated in the constructor. We can use it in `equals` to the fast return for different objects.

Modifications:
Use `hashCode` field in `DefaultChannelId.equals()`.

Result:
Fast `equals` on negative scenarios.
2017-09-08 10:36:48 +02:00
Norman Maurer
a739d89792 Not log notify failure for DelegatingChannelPromiseNotifier when promise is VoidChannelPromise
Motivation:

We should not log by default if the promise is a VoidChannelPromise as its try* methods will always return false.

Modifications:

Do an instanceof check to determine if we should log or not by default

Result:

No more noise in the logs when using a VoidChannelPromise.
2017-09-07 08:58:39 +02:00
Scott Mitchell
89ecb4b4a4 AutoClose behavior may infinite loop
Motivation:
If AutoClose is false and there is a IoException then AbstractChannel will not close the channel but instead just fail flushed element in the ChannelOutboundBuffer. AbstractChannel also notifies of writability changes, which may lead to an infinite loop if the peer has closed its read side of the socket because we will keep accepting more data but continuously fail because the peer isn't accepting writes.

Modifications:
- If the transport throws on a write we should acknowledge that the output side of the channel has been shutdown and cleanup. If the channel can't accept more data because it is full, and still healthy it is not expected to throw. However if the channel is not healthy it will throw and is not expected to accept any more writes. In this case we should shutdown the output for Channels that support this feature and otherwise just close.
- Connection-less protocols like UDP can remain the same because the channel may disconnected temporarily.
- Make sure AbstractUnsafe#shutdownOutput is called because the shutdown on the socket may throw an exception.

Result:
More correct handling of write failure when AutoClose is false.
2017-08-25 21:01:41 -07:00
Norman Maurer
b967805f32 [maven-release-plugin] prepare for next development iteration 2017-08-24 15:38:22 +02:00
Norman Maurer
da8e010a42 [maven-release-plugin] prepare release netty-4.1.15.Final 2017-08-24 15:37:59 +02:00
Norman Maurer
dd9ad15b12 Add Unit test for [#7143] 2017-08-24 13:29:38 +02:00
Scott Mitchell
c80fdc8241 AbstractChannel should call doClose even after shutdownOutput
Motivation:
ShutdownOutput now fails all pending writes in the ChannelOutboundBuffer and sets it to null. However the Close code path uses the ChannelOutboundBuffer as an indication that the close operation is in progress and exits early and will not call doClose. This will lead to the Channel not actually being fully closed.

Bug introduced by 237a4da1b7

Modifications:
- AbstractChannel#close shouldn't exit early just because outboundBuffer is null, and instead should use additional state closeInitiated to avoid duplicate close operations

Result:
AbstractChannel#close(..) after AbstractChannel#shutdownOutbound() will still invoke doClose and cleanup Channel state.
2017-08-24 12:30:34 +02:00
Derek Perez
b18a201d02 various errorprone fixes.
Motivation:

Continuing to make netty happy when compiling through errorprone.

Modification:

Mostly comments, some minor switch statement changes.

Result:

No more compiler errors!
2017-08-23 12:49:58 +02:00
Carl Mastrangelo
03d89c2222 Use Constructor for reflective class instantiation.
Motivation:
Calling `newInstance()` on a Class object can bypass compile time
checked Exception propagation.  This is noted in Java Puzzlers,
as well as in ErrorProne:
http://errorprone.info/bugpattern/ClassNewInstance

Modifications:
Use the niladic constructor to create a new instance.

Result:
Compile time safety for checked exceptions
2017-08-18 09:42:36 +02:00
Norman Maurer
74f24a5c19 Finish work on http2 child channel implementation and http2 frame api.
Motivation:

Our http2 child channel implementation was not 100 % complete and had a few bugs. Beside this the performance overhead was non-trivial.

Modifications:

There are a lot of modifications, the most important....
  * Http2FrameCodec extends Http2ConnectionHandler and Http2MultiplexCodec extends Http2FrameCodec to reduce performance heads and inter-dependencies on handlers in the pipeline
  * Correctly handle outbound flow control for child channels
  * Support unknow frame types in Http2FrameCodec and Http2MultiplexCodec
  * Use a consistent way how to create Http2ConnectionHandler, Http2FrameCodec and Http2MultiplexCodec (via a builder)
  * Remove Http2Codec and Http2CodecBuilder as the user should just use Http2MultipleCodec and Http2MultiplexCodecBuilder now
  * Smart handling of flushes from child channels to reduce overhead
  * Reduce object allocations
  * child channels always use the same EventLoop as the parent Channel to reduce overhead and simplify implementation.
  * Not extend AbstractChannel for the child channel implementation to reduce overhead in terms of performance and memory usage
  * Remove Http2FrameStream.managedState(...) as the user of the child channel api should just use Channel.attr(...)

Result:

Http2MultiplexCodec (and so child channels) and Http2FrameCodec are more correct, faster and more feature complete.
2017-08-11 12:41:28 +02:00
buchgr
5380c7c3e3 HTTP/2 Child Channel and FrameCodec Feature Parity.
Motivation:

This PR (unfortunately) does 4 things:
1) Add outbound flow control to the Http2MultiplexCodec:
   The HTTP/2 child channel API should interact with HTTP/2 outbound/remote flow control. That is,
   if a H2 stream used up all its flow control window, the corresponding child channel should be
   marked unwritable and a writability-changed event should be fired. Similarly, a unwritable
   child channel should be marked writable and a writability-event should be fired, once a
   WINDOW_UPDATE frame has been received. The changes are (mostly) contained in ChannelOutboundBuffer,
   AbstractHttp2StreamChannel and Http2MultiplexCodec.

2) Introduce a Http2Stream2 object, that is used instead of stream identifiers on stream frames. A
   Http2Stream2 object allows an application to attach state to it, and so a application handler
   no longer needs to maintain stream state (i.e. in a map(id -> state)) himself.

3) Remove stream state events, which are no longer necessary due to the introduction of Http2Stream2.
   Also those stream state events have been found hard and complex to work with, when porting gRPC
   to the Http2FrameCodec.

4) Add support for HTTP/2 frames that have not yet been implemented, like PING and SETTINGS. Also add
   a Http2FrameCodecBuilder that exposes options from the Http2ConnectionHandler API that couldn't else
   be used with the frame codec, like buffering outbound streams, window update ratio, frame logger, etc.

Modifications:

1) A child channel's writability and a H2 stream's outbound flow control window interact, as described
   in the motivation. A channel handler is free to ignore the channel's writability, in which case the
   parent channel is reponsible for buffering writes until a WINDOW_UPDATE is received.

   The connection-level flow control window is ignored for now. That is, a child channel's writability
   is only affected by the stream-level flow control window. So a child channel could be marked writable,
   even though the connection-level flow control window is zero.

2) Modify Http2StreamFrame and the Http2FrameCodec to take a Http2Stream2 object intstead of a primitive
   integer. Introduce a special Http2ChannelDuplexHandler that has newStream() and forEachActiveStream()
   methods. It's recommended for a user to extend from this handler, to use those advanced features.

3) As explained in the documentation, a new inbound stream active can be detected by checking if the
   Http2Stream2.managedState() of a Http2HeadersFrame is null. An outbound stream active can be detected
   by adding a listener to the ChannelPromise of the write of the first Http2HeadersFrame. A stream
   closed event can be listened to by adding a listener to the Http2Stream2.closeFuture().

4) Add a simple Http2FrameCodecBuilder and implement the missing frame types.

Result:

1) The Http2MultiplexCodec supports outbound flow control.
2) The Http2FrameCodec API makes it easy for a user to manage custom stream specific state and to create
   new outbound streams.
3) The Http2FrameCodec API is much cleaner and easier to work with. Hacks like the ChannelCarryingHeadersFrame
   are no longer necessary.
4) The Http2FrameCodec now also supports PING and SETTINGS frames. The Http2FrameCodecBuilder allows the Http2FrameCodec
   to use some of the rich features of the Http2ConnectionHandler API.
2017-08-11 12:41:28 +02:00
Norman Maurer
d0c43c9e42 We should prefer heap buffers when using the OIO transport to reduce memory copies.
Motivation:

When using the OIO transport we need to act on byte[] when writing and reading from / to the underyling Socket. So we should ensure we use heap buffers by default to reduce memory copies.

Modifications:

Ensure we prefer heap buffers by default for the OIO transport.

Result:

Possible less memory copies.
2017-08-11 08:48:04 +02:00
Norman Maurer
85f5d6bf05 Ensure we null out the previous set InetAddress on java.net.DatagramPacket when using OioDatagramChannel.
Motivation:

We need to ensure we always null out (or set) the address on the java.net.DatagramPacket when doing read or write operation as the same instance is used across different calls.

Modifications:

Null out the address if needed.

Result:

Ensure the correct remote address is used when connect / disconnect between calls and also mix these with calls that directly specify the remote address for adatagram packets.
2017-08-09 07:31:59 +02:00
Norman Maurer
f8b495b2f2 Correctly support SO_TIMEOUT for OioDatagramChannel
Motivation:

We need to support SO_TIMEOUT for the OioDatagramChannel but we miss this atm as we not have special handling for it in the DatagramChannelConfig impl that we use. Because of this the following log lines showed up when running the testsuite:

20:31:26.299 [main] WARN  io.netty.bootstrap.Bootstrap - Unknown channel option 'SO_TIMEOUT' for channel '[id: 0x7cb9183c]'

Modifications:

- Add OioDatagramChannelConfig and impl
- Correctly set SO_TIMEOUT in testsuite

Result:

Support SO_TIMEOUT for OioDatagramChannel and so faster execution of datagram related tests in the testsuite
2017-08-08 09:08:33 +02:00
Scott Mitchell
237a4da1b7 Shutting down the outbound side of the channel should not accept future writes
Motivation:
Implementations of DuplexChannel delegate the shutdownOutput to the underlying transport, but do not take any action on the ChannelOutboundBuffer. In the event of a write failure due to the underlying transport failing and application may attempt to shutdown the output and allow the read side the transport to finish and detect the close. However this may result in an issue where writes are failed, this generates a writability change, we continue to write more data, and this may lead to another writability change, and this loop may continue. Shutting down the output should fail all pending writes and not allow any future writes to avoid this scenario.

Modifications:
- Implementations of DuplexChannel should null out the ChannelOutboundBuffer and fail all pending writes

Result:
More controlled sequencing for shutting down the output side of a channel.
2017-08-04 10:59:57 -07:00
Norman Maurer
52f384b37f [maven-release-plugin] prepare for next development iteration 2017-08-02 12:55:10 +00:00
Norman Maurer
8cc1071881 [maven-release-plugin] prepare release netty-4.1.14.Final 2017-08-02 12:54:51 +00:00
Norman Maurer
8adb30bbe2 Correctly run all pending tasks for EmbeddedChannel when the Channel was closed.
Motivation:

When a user called ctx.close() and used the EmbeddedChannel we did not correctly run all pending tasks which means channelInactive was never called.

Modifications:

Ensure we run all pending tasks after all operations that may change the Channel state and are part of the Channel.Unsafe impl.

Result:

Fixes [#6894].
2017-07-30 06:57:18 +02:00
Carl Mastrangelo
60250f3795 Make DelegatingChannelPromiseNotifier use Vararg overload
Motivation:
ErrorProne complains that the array override doesn't match the
vararg super call.  See http://errorprone.info/bugpattern/Overrides

Additionally, almost every other Future uses the vararg form, so
it would be stylistically consistent to keep it that way.

Modifications:
Use vararg override.

Result:
Cleaner, less naggy code.
2017-07-28 07:29:43 +02:00
Norman Maurer
339131c660 DefaultChannelPipeline.estimatorHandle needs to be volatile
Motivation:

DefaultChannelPipeline.estimatorHandle needs to be volatile as its accessed from different threads.

Modifications:

Make DefaultChannelPipeline.estimatorHandle volatile and correctly init it via CAS

Result:

No more race.
2017-07-27 06:57:22 +02:00
Norman Maurer
529025d9d5 Allow to use oldest Channel out of the Simple / FixedChannelPool on acquire
Motivation:

We previously used pollLast() to retrieve a Channel from the queue that backs SimpleChannelPool. This could lead to the problem that some Channels are very unfrequently used and so when these are used the connection was already be closed and so could not be reused.

Modifications:

Allow to configure if the last recent used Channel should be used or the "oldest".

Result:

More flexible usage of ChannelPools
2017-07-26 20:37:19 +02:00
Norman Maurer
ef22e65b57 Allow to delay registration when creating a EmbeddedChannel
Motivation:

Some ChannelOptions must be set before the Channel is really registered to have the desired effect.

Modifications:

Add another constructor argument which allows to not register the EmbeddedChannel to its EventLoop until the user calls register().

Result:

More flexible usage of EmbeddedChannel. Also Fixes [#6968].
2017-07-19 19:35:03 +02:00
Norman Maurer
4c14d1198b Add testcase to ensure NioEventLoop.rebuildSelector() works correctly.
Motivation:

We had recently a report that the issue [#6607] is still not fixed.

Modifications:

Add a testcase to prove the issue is fixed.

Result:

More tests.
2017-07-18 07:20:16 +02:00
Scott Mitchell
7cfe416182 Use unbounded queues from JCTools 2.0.2
Motivation:
JCTools 2.0.2 provides an unbounded MPSC linked queue. Before we shaded JCTools we had our own unbounded MPSC linked queue and used it in various places but gave this up because there was no public equivalent available in JCTools at the time.

Modifications:
- Use JCTool's MPSC linked queue when no upper bound is specified

Result:
Fixes https://github.com/netty/netty/issues/5951
2017-07-10 12:32:15 -07:00
Scott Mitchell
86e653e04f SslHandler aggregation of plaintext data on write
Motivation:
Each call to SSL_write may introduce about ~100 bytes of overhead. The OpenSslEngine (based upon OpenSSL) is not able to do gathering writes so this means each wrap operation will incur the ~100 byte overhead. This commit attempts to increase goodput by aggregating the plaintext in chunks of <a href="https://tools.ietf.org/html/rfc5246#section-6.2">2^14</a>. If many small chunks are written this can increase goodput, decrease the amount of calls to SSL_write, and decrease overall encryption operations.

Modifications:
- Introduce SslHandlerCoalescingBufferQueue in SslHandler which will aggregate up to 2^14 chunks of plaintext by default
- Introduce SslHandler#setWrapDataSize to control how much data should be aggregated for each write. Aggregation can be disabled by setting this value to <= 0.

Result:
Better goodput when using SslHandler and the OpenSslEngine.
2017-07-10 12:22:08 -07:00
Norman Maurer
2a376eeb1b [maven-release-plugin] prepare for next development iteration 2017-07-06 13:24:06 +02:00
Norman Maurer
c7f8168324 [maven-release-plugin] prepare release netty-4.1.13.Final 2017-07-06 13:23:51 +02:00
Norman Maurer
91b62da8c1 Remove @deprecation keyword on AbstractUnsafe.ensureOpen(...)
Motivation:

e845670043 marked AbstractUnsafe.ensureOpen(...) as deprecated for no reason.

Modifications:

Remove `@deprecation`

Result:

Remove incorrect annotation
2017-06-29 14:28:55 +02:00
Norman Maurer
b7a5743e8b Return the correct Future from FixedChannelPool.release()
Motivation:

The behaviour of the FixedChannelPool.release was inconsistent with the
SimpleChannelPool implementation, in that given promise is returned.

In the FixedChannelPool implementation a new promise was return and
this meant that the completion of that promise can be different.
Specifically on releasing a channel to a closed pool, the parameter
promise is failed with an IllegalStateException but the returned one
will have been successful (as it was completed by call to super
.release)

Modification:

Return the given promise as the result of FixedChannelPool.release

Result:

Returned promise will reflect the result of the release operation.
2017-06-28 18:57:10 +02:00
Norman Maurer
32b3f58f63 Close channels that are released to a closed FixedChannelPool.
Motivation:

Channels returned to a FixedChannelPool after closing it remain active.

Since channels that where acquired from the pool are not closed during the close operation, they remain open even after releasing the channel back to the pool where they are then in accessible and become in-effect a connection leak.

Modification:

Close the released channel on releasing back to a closed pool.

Result:

Much harder to create a connection leak by closing an active
FixedChannelPool instance.
2017-06-28 18:50:51 +02:00
Norman Maurer
94c0ef3c96 Not fail the promise when a closed Channel is offered back to the ChannelPool
Motivation:

We should not fail the promise when a closed Channel is offereed back to the ChannelPool as we explicit mention that the Channel must always be returned.

Modifications:

- Not fail the promise
- Add test-case

Result:

Fixes [#6831]
2017-06-13 18:50:20 +02:00
Norman Maurer
fd67a2354d [maven-release-plugin] prepare for next development iteration 2017-06-08 21:06:24 +02:00
Norman Maurer
3acd5c68ea [maven-release-plugin] prepare release netty-4.1.12.Final 2017-06-08 21:06:01 +02:00
Norman Maurer
80aa5dcdcc Revert "Not add ChannelHandler to ChannelPipeline once the pipeline was destroyed."
This reverts commit 4aa8002596.
2017-06-08 19:50:17 +02:00
Norman Maurer
4aa8002596 Not add ChannelHandler to ChannelPipeline once the pipeline was destroyed.
Motivation:

ChannelPipeline will happily add a handler to a closed Channel's pipeline and will call handlerAdded(...) but will not call handlerRemoved(...).

Modifications:

Check if pipeline was destroyed and if so not add the handler at all but propergate an exception.

Result:

Fixes [#6768]
2017-05-31 07:37:39 +02:00
Norman Maurer
0db2901f4d [maven-release-plugin] prepare for next development iteration 2017-05-11 16:00:55 +02:00