Commit Graph

7311 Commits

Author SHA1 Message Date
luchen
e08a361bfc Motivation:
We had unused import in  RedisEncoderTest, we should remove it to keep code clean.

Modifications:

Remove unused import.

Result:

Cleanup done.
2016-04-20 11:07:48 -07:00
Trustin Lee
38d05abd84 Reorganize imports 2016-04-14 18:58:47 +09:00
Norman Maurer
03638869b9 Update dependencies
Motivation:

Before release 4.1.0.Final we should update all our dependencies.

Modifications:

Update dependencies.

Result:

Up-to-date dependencies used.
2016-04-14 11:05:53 +02:00
Norman Maurer
4fb55beed1 Re-add codec-smtp which was removed by 96455a9558 by mistake. 2016-04-14 11:01:12 +02:00
Norman Maurer
d698746609 Add ByteBuf.asReadOnly()
Motivation:

We lately added ByteBuf.isReadOnly() which allows to detect if a buffer is read-only or not. We should add ByteBuf.asReadOnly() to allow easily access a read-only version of a buffer.

Modifications:

- Add ByteBuf.asReadOnly()
- Deprecate Unpooled.unmodifiableBuffer(Bytebuf)

Result:

More consistent api.
2016-04-14 10:51:20 +02:00
Jongyeol Choi
96455a9558 Implement codec-redis
Motivation:

- To encode/decode RESP (REdis Serialization Protocol) using Netty
- http://redis.io/topics/protocol

Modifications:

- Add RedisEncoder, RedisDecoder
- Add RedisBulkStringAggregator and RedisArrayAggregator
- Add tests

Result:

- Added codec-redis
- codec-redis can encode/decode RESP (REdis Serialization Protocol)
2016-04-14 10:23:16 +02:00
Norman Maurer
6108b7297b Correctly handle ChannelInputShutdownEvent in ReplayingDecoder
Motivation:

b112673554 added ChannelInputShutdownEvent support to ByteToMessageDecoder but missed updating the code for ReplayingDecoder. This has the effect:

- If a ChannelInputShutdownEvent is fired ByteToMessageDecoder (the super-class of ReplayingDecoder) will call the channelInputClosed(...) method which will pass the incorrect buffer to the decode method of ReplayingDecoder.

Modifications:

Share more code between ByteToMessageDEcoder and ReplayingDecoder and so also support ChannelInputShutdownEvent correctly in ReplayingDecoder

Result:

ChannelInputShutdownEvent is corrrectly handle in ReplayingDecoder as well.
2016-04-14 10:20:58 +02:00
Norman Maurer
7c734fcf73 Fix resource leak in tests introduced by 69070c37ba. 2016-04-14 10:13:55 +02:00
Norman Maurer
0035630bd0 We need to ensure we correct reset decoder in decodeLast() to not produce multiple LastHttpContent instances.
Motivation:

We missed to reset the decoder when asked for it in HttpObjectDecoder and so sometimes could produce more then one LastHttpContent in a sequence during channelInactive.

This did show up as AssertionError:

22:22:35.499 [nioEventLoopGroup-3-1] WARN  i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.AssertionError: null
	at io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:205) ~[classes/:na]
	at io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:57) ~[classes/:na]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) ~[classes/:na]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) [classes/:na]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278) [classes/:na]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:428) [classes/:na]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:277) [classes/:na]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:343) [classes/:na]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:309) [classes/:na]
	at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:228) [classes/:na]
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:213) [classes/:na]
        ...

Modifications:

Correctly reset decoder.

Result:

Correctly only produce one LastHttpContent per sequence.
2016-04-14 09:33:23 +02:00
Norman Maurer
89268659f3 [#4836] Enforce 64bit JDK when build netty.
Motivation:

As we only provide tcnative jars for 64bit we should enforce 64bit when try to build netty, to make it easier for the user to understand why the build fails.

Modifications:

Add enforce rule.

Result:

Ensure 64bit is used when build netty.
2016-04-14 09:31:19 +02:00
Norman Maurer
1861461db2 [#4635] Stop decoding if decoder was removed in ReplayingDecoder
We need to check if this handler was removed before continuing with decoding.
If it was removed, it is not safe to continue to operate on the buffer. This was already fixed for ByteToMessageDecoder in 4cdbe39284  but missed for ReplayingDecoder.

Modifications:

Check if decoder was removed after fire messages through the pipeline.

Result:

No illegal buffer access when decoder was removed.
2016-04-14 08:25:59 +02:00
Norman Maurer
7d3ca7fb92 Fix IndexOutOfBoundsException when FixedCompositeByteBuf is constructed with an empty array.
Motivation:

When FixedCompositeByteBuf was constructed with new ByteBuf[0] and IndexOutOfboundsException was thrown.

Modifications:

Fix constructor

Result:

No more exception
2016-04-14 08:07:55 +02:00
Norman Maurer
4ddb81f36f Not cache SwappedByteBuf in AbstractByteBuf to reduce memory footprint.
Motivation:

We should not cache the SwappedByteBuf in AbstractByteBuf to reduce the memory footprint.

Modifications:

Not cache the SwappedByteBuf.

Result:

Less memory footprint.
2016-04-14 08:06:21 +02:00
Norman Maurer
2537880e5d Fix typo in exception message
Motivation:

Typo in exception message.

Modifications:

Fix the typo.

Result:

No more typo.
2016-04-14 08:03:35 +02:00
Trustin Lee
0b078314b2 Add ByteBuf.isReadOnly()
Motivation:

It is sometimes useful to determins if a buffer is read-only.

Modifications:

Add ByteBuf.isReadOnly()

Result:

One more feature
2016-04-13 21:41:27 +09:00
Trustin Lee
57063b6db0 Implement the ByteBuf methods that should have been overridden
Motivation:

Some ByteBuf implementations do not override all necessary methods,
which can lead to potentially sub-optimal behavior.

Also, SlicedByteBuf does not perform the range check correctly due to
missing overrides.

Modifications:

- Add missing overrides
- Use unwrap() instead of direct member access in derived buffers for
  consistency
- Merge unwrap0() into unwrap() using covariant return type
- Deprecate AbstractDerivedByteBuf and its subtypes, because they were
  not meant to be public

Result:

Correctness
2016-04-13 21:05:57 +09:00
nmittler
de2515ddd8 Support preloading of native epoll lib
Motivation:

Some applications may use alternative methods of loading the epoll JNI symbols. We should support this use case.

Modifications:

Attempt to use a side effect free JNI method. If that fails, load the library.

Result:

Fixes #5122
2016-04-12 11:09:06 -07:00
Norman Maurer
718bf2fa45 Fix resource-leak which was reported as a result of commit 69070c37ba 2016-04-12 16:27:02 +02:00
nmittler
379ad2c02e Support preloading of tcnative share lib
Motivation:

Some applications may use alternative methods of loading the tcnative JNI symbols. We should support this use case.

Modifications:

Separate the loading and initialzation of the tcnative library so that each can fail independently.

Result:

Fixes #5043
2016-04-11 11:28:02 -07:00
Norman Maurer
572bdfb494 [maven-release-plugin] prepare for next development iteration 2016-04-10 08:37:18 +02:00
Norman Maurer
c6121a6f49 [maven-release-plugin] prepare release netty-4.1.0.CR7 2016-04-10 08:36:56 +02:00
Norman Maurer
4652223dec Fix resource leak in test introduced by 69070c37ba 2016-04-10 08:04:57 +02:00
Norman Maurer
6e919f70f8 [maven-release-plugin] rollback the release of netty-4.1.0.CR7 2016-04-09 22:13:44 +02:00
Norman Maurer
4cdd51509a [maven-release-plugin] prepare release netty-4.1.0.CR7 2016-04-09 22:05:34 +02:00
Scott Mitchell
abce89d1bc Revert "[#5028] Fix re-entrance issue with channelWritabilityChanged(...) and write(...)"
Motivation:
Revert d0943dcd30. Delaying the notification of writability change may lead to notification being missed. This is a ABA type of concurrency problem.

Modifications:
- Revert d0943dcd30.

Result:
channelWritabilityChange will be called on every change, and will not be suppressed due to ABA scenario.
2016-04-09 20:32:47 +02:00
Norman Maurer
69070c37ba ByteBuf.readBytes(...) should use the allocator of the buffer to create the new buffer. Related to [#5093]
Motivation:

ByteBuf.readBytes(...) uses Unpooled.buffer(...) internally which will use a heap ByteBuf and also not able to make use of the allocator which may be pooled. We should better make use of the allocator.

Modifications:

Use the allocator for thenew buffer.

Result:

Take allocator into account when copy bytes.
2016-04-09 20:28:17 +02:00
Norman Maurer
200ca39b5c Add PooledByteBufAllocator.dumpStats() which allows to obtain a human-readable status of the allocator.
Motiviation:

Sometimes it is useful to dump the status of the PooledByteBufAllocator and log it. Doing this is currently a bit cumbersome as the user needs to basically iterate through all the metrics and compose the String. we would better provide an easy way to do this.

Modification:

Add dumpStats() method.

Result:

Easier to get a view into the status of the allocator.
2016-04-09 19:16:53 +02:00
Norman Maurer
9498d1a9b3 Allow to create a JdkSslContext from an existing JDK SSLContext. Related to [#5095] and [#4929]
Motivation:

Sometimes a user only has access to a preconfigured SSLContext but still would like to use our ssl sub-system. For this situations it would be very useful if the user could create a JdkSslContext instance from an existing SSLContext.

Modifications:

- Create new public constructors in JdkSslContext which allow to wrap an existing SSLContext and make the class non-abstract
- Mark JdkSslServerContext and JdkSslClientContext as deprecated as the user should not directly use these.

Result:

It's now possible to create an JdkSslContext from an existing SSLContext.
2016-04-09 19:06:17 +02:00
Norman Maurer
eb14e5b2f5 Ensure EpollSocketChannel.localAddress() returns correct address after connect(...) call.
Motivation:

We missed to correctly retrieve the localAddress() after we called Socket.connect(..) and so the user would always see an incorrect address when calling EpollSocketChannel.localAddress().

Modifications:

- Ensure we always retrieve the localAddress() after we called Socket.connect(...) as only after this we will be able to receive the correct address.
- Add unit test

Result:

Correct and consistent behaviour across different transports (NIO/OIO/EPOLL).
2016-04-09 19:04:45 +02:00
Norman Maurer
cfa258e096 PoolChunkList.allocate(...) should return false without the need to walk all the contained PoolChunks when the requested capacity is too big.
Motivation:

PoolChunkList.allocate(...) should return false without the need to walk all the contained PoolChunks when the requested capacity is larger then the capacity that can be allocated out of the PoolChunks in respect to the minUsage() and maxUsage() of the PoolChunkList.

Modifications:

Precompute the maximal capacity that can be allocated out of the PoolChunks that are contained in the PoolChunkList and use this to fast return from the allocate(...) method if an allocation capacity larger then that is requested.

Result:

Faster detection of allocations that can not be handled by the PoolChunkList and so faster allocations in general via the PoolArena.
2016-04-09 19:00:20 +02:00
Norman Maurer
2ae97946dd Allow to retrieve the number of active bytes per PoolArena.
Motivation:

To better understand how much memory is used by Netty for ByteBufs it is useful to understand how many bytes are currently active (allocated) per PoolArena.

Modifications:

- Add PoolArenaMetric.numActiveBytes()

Result:

The user is able to get better insight into the PooledByteBufAllocator.
2016-04-09 18:54:05 +02:00
Norman Maurer
787a55874b Cleanup PoolChunk and PoolArena
Motivation:

To make it easier to understand PoolChunk and PoolArena we should cleanup duplicated code.

Modifications:

- Move reused code into methods
- Use Math.max(...)

Result:

Cleaner code and easier to understand.
2016-04-09 18:47:40 +02:00
Norman Maurer
d081851156 Remove ByteBuf.readBytes(int) calls when possible
Motivation:

We use ByteBuf.readBytes(int) in various places where we could either remove it completely or use readSlice(int).retain().

Modifications:

- Remove ByteBuf.readBytes(int) when possible or replace by readSlice(int).retain().

Result:

Faster code.
2016-04-09 18:40:57 +02:00
Norman Maurer
1b4a5609d3 Not try to allocate out of the PoolChunkList that contains only full PoolChunks
Motivation:

When doing a normal allocation in PoolArena we also tried to allocate out of the PoolChunkList that only contains completely full PoolChunks. This makes no sense as these are full anyway so an allocation will never work here and just gives a perf hit as we need to walk the whole list of PoolChunks in the list.

Modifications:

Not try to allocate from PoolChunkList that only contains full PoolChunks

Result:

Faster allocation times when a new PoolChunk must be created.
2016-04-09 18:39:56 +02:00
Scott Mitchell
a0b28d6c82 Fix potential assertion error introduced by 0bc93dd
Motivation:
Commit 0bc93dd introduced a potential assertion failure, if the deprecated method would be used.

Modifications:
Fix the potential assertion error.

Result:
Regression removed
2016-04-08 09:44:44 -07:00
Scott Mitchell
9a63aafe22 EPOLL ET Set ReadFlag and Limit epollInReadyRunnable
Motivation:
441aa4c575 conditionally set the readFlag based upon if maybeMoreDataToRead is set. It is possible that the read flag will not be set, and nothing will be read by executeEpollInReadyRunnable and no actual data will be read even though the user requested it.

Modifications:
- Always set the readFlag in doBeginRead
- Make it so only a single epollInReadyRunnable can execute for a channel at a time

Result:
Less chance of missing read events in EPOLL transport.
2016-04-08 11:03:52 +02:00
Trustin Lee
32cfe25132 Fix checkstyle 2016-04-08 17:44:52 +09:00
Trustin Lee
c1a3cc32e7 Fix an assertion error introduced by 0bc93dda08
Motivation:
    
Commit 0bc93dda08 introduced an assertion
failure.

Modifications:

Fix the assertion error.

Result:

Regression removed
2016-04-08 17:23:32 +09:00
stroller
84e281ced9 fix one java doc issue: extra }
Motivation:

There is one extra } for WriteBufferWaterMark's javadoc:
{@linkplain #high}  high water mark}

The generated javadoc will show the content: "the high high water mark}"

Modifications:

remove the }

Result:
The generated javadoc will show the content: "the high water mark" instead of "the high high water mark}"
2016-04-08 09:47:26 +02:00
Xiaoyan Lin
bc6adab48c Make all InternalLoggerFactory implementations be singletons
Motivation:

It's better to make all InternalLoggerFactory implementations be singletons according to the discussions in #5047

Modifications:

Make all InternalLoggerFactory implementations be singletons and hide the construtors.

Result:

All InternalLoggerFactory implementations be singletons.
2016-04-08 09:39:32 +02:00
Scott Mitchell
0bc93dda08 NIO autoReadClear should also clear the SelectionKey
Motivation:
Prior to 5b48fc284e setting readPending to false when autoReadClear was called was enough because when/if the EventLoop woke up with a read event we would first check if readPending was true and if it is not, we would return early. Now that readPending will only be set in the EventLoop it is not necessary to check readPending at the start of the read methods, and we should instead remove the OP_READ from the SelectionKey when we also set readPending to false.

Modifications:
- autoReadCleared should call AbstractNioUnsafe.removeReadOp

Result:
NIO is now consistent with EPOLL and removes the READ operation on the selector when autoRead is cleared.
2016-04-08 00:03:37 -07:00
Norman Maurer
562d8d2200 Add smtp codec (client side only).
Motivation:

When writing a SMTP client a provided SMTP codec that follows RFC2821 is useful.

Modification:

Add client side codec and test.

Results:

People who want to write a SMTP client can reuse the codec.
2016-04-07 18:42:17 +02:00
Hyangtack Lee
24254b159f Propagate h2c upgrade success event to the next handler before removing source codec
Motivation:

When upgrading h2c, I found that sometimes both of http2 settings frame and http response message was arrived before receiving upgrade success event. It was because ByteToMessageDecoder propagated its internally buffered message to the next handler when removing itself from pipeline.(refer to ByteToMessageDecoder#handlerRemoved)
I think it's better to propagate upgrade success event when handling 101 switching protocol response.

Modifications:

Upgrade success event will be propagated before removing source codec.

Result:

It guarantees that upgrade success event will be arrived first at the next handler.
2016-04-07 17:41:46 +02:00
Xiaoyan Lin
e053b96b5c Set the length fields of memcache messages automatically
Motivation:

People need to set all length fields manually when creating a memcache message and it's error prone. See #2736 for more dicussion.

Modifications:

This patch adds the logic to update the keyLength, extrasLength and totalBodyLength when key, extras or content is set.

Result:

The length fields of memcache messages will be updated automatically.
2016-04-07 06:30:42 +02:00
Scott Mitchell
5b48fc284e Make OIO/NIO/EPOLL autoReadClear consistent
Motivation:
OIO/NIO use a volatile variable to track if a read is pending. EPOLL does not use a volatile an executes a Runnable on the event loop thread to set readPending to false. These mechansims should be consistent, and not using a volatile variable is preferable because the variable is written to frequently in the event loop thread.
OIO also does not set readPending to false before each fireChannelRead operation and may result in reading more data than the user desires.

Modifications:
- OIO/NIO should not use a volatile variable for readPending
- OIO should set readPending to false before each fireChannelRead

Result:
OIO/NIO/EPOLL are more consistent w.r.t. readPending and volatile variable operations are reduced
Fixes https://github.com/netty/netty/issues/5069
2016-04-06 12:32:14 -07:00
Norman Maurer
7b121c26ae Use Math.min(...) and Math.max(...) as these are intrinsics.
Motivation:

We should better use Math utilities as these are intrinsics. This is a cleanup for ea3ffb8536.

Modifications:

Use Math utilities.

Result:

Cleaner code and use of intrinsics.
2016-04-06 21:30:32 +02:00
Kun Zhang
c024dc92d0 Explicit thread group on DefaultThreadFactory.
Motivation:

Fixes #5084. We (gRPC) encountered a bug that was triggered by
grpc/grpc-java@d927180. After that commit, event loop threads are
created per task by NioEventLoopGroup, and inherits the thread group of
the caller, which in our case is an application-provided request-scope
thread. Things go south when the application tries to manipulate (e.g.,
interrupt and join) all threads of the request-scope thread group, which
unexpectedly include the event loop threads.

Modifications:

DefaultThreadFactory will save the current thread group in constructor,
and apply it to all new threads.

Result:

Threads created by DefaultThreadFactory will be in the same thread group
as the thread where the factory is created.
2016-04-06 21:19:43 +02:00
Norman Maurer
88b093c481 PoolChunkList not correctly move PoolChunks when these are moved.
Motivation:

When a PoolChunk needs to get moved to the previous PoolChunkList because of the minUsage / maxUsage constraints we always just moved it one level which is incorrect and so could lead to have PoolChunks in the wrong PoolChunkList (in respect to their minUsage / maxUsage settings). This then could have the effect that PoolChunks are not released / freed in a timely fashion and so.

Modifications:

- Correctly move PoolChunks between PoolChunkLists, which includes moving it multiple "levels".
- Add unit test

Result:

Correctlty move the PoolChunk to PoolChunkList when it is freed, even if its multiple layers.
2016-04-06 11:52:23 +02:00
Trustin Lee
4fa5d2cf52 Do now swallow an exception triggered by late listener notification
Related: #3449

Motivation:

When a user shut down an EventExecutor/Loop prematurely, a Promise will
fail to execute its listeners. When it happens, DefaultPromise will log
a message at ERROR level, but there's no way to get notified about it
programmatically.

Modifications:

Do not catch and log the RejectedExecutionException unconditionally,
but only catch and log for non-late listener notifications, so that a
user gets notified on submission failure at least when the listener is
late.

Result:

Remedies #3449 to some extent, although we will need fundamental fix for
that, such as #3566
2016-04-06 11:16:44 +02:00
Norman Maurer
d0943dcd30 [#5028] Fix re-entrance issue with channelWritabilityChanged(...) and write(...)
Motivation:

When always triggered fireChannelWritabilityChanged() directly when the update the pending bytes in the ChannelOutboundBuffer was made from within the EventLoop. This is problematic as this can cause some re-entrance issue if the user has a custom ChannelOutboundHandler that does multiple writes from within the write(...) method and also has a handler that will intercept the channelWritabilityChanged event and trigger another write when the Channel is writable. This can also easily happen if the user just use a MessageToMessageEncoder subclass and triggers a write from channelWritabilityChanged().

Beside this we also triggered fireChannelWritabilityChanged() too often when a user did a write from outside the EventLoop. In this case we increased the pending bytes of the outboundbuffer before scheduled the actual write and decreased again before the write then takes place. Both of this may trigger a fireChannelWritabilityChanged() event which then may be re-triggered once the actual write ends again in the ChannelOutboundBuffer.

The third gotcha was that a user may get multiple events even if the writability of the channel not changed.

Modification:

- Always invoke the fireChannelWritabilityChanged() later on the EventLoop.
- Only trigger the fireChannelWritabilityChanged() if the channel is still active and if the writability of the channel changed. No need to cause events that were already triggered without a real writability change.
- when write(...) is called from outside the EventLoop we only increase the pending bytes in the outbound buffer (so that Channel.isWritable() is updated directly) but not cause a fireChannelWritabilityChanged(). The fireChannelWritabilityChanged() is then triggered once the task is picked up by the EventLoop as usual.

Result:

No more re-entrance possible because of writes from within channelWritabilityChanged(...) method and no events without a real writability change.
2016-04-06 10:07:13 +02:00