Commit Graph

6619 Commits

Author SHA1 Message Date
Trustin Lee
239371abaf Add io.netty.noJdkZlibEncoder system property
Related issue: #2821

Motivation:

There's no way for a user to change the default ZlibEncoder
implementation.

It is already possible to change the default ZlibDecoder implementation.

Modification:

Add a new system property 'io.netty.noJdkZlibEncoder'.

Result:

A user can disable JDK ZlibEncoder, just like he or she can disable JDK
ZlibDecoder.
2014-08-26 15:49:29 +09:00
Scott Mitchell
892944eba8 HTTP/2 draft 14 HTTP message flow
Motivation:

HTTP/2 draft 14 came out a couple of weeks ago and we need to keep up
with the spec.

Modifications:
-Revert back to dispatching FullHttpMessage objects instead of individual HttpObjects
-Corrections to HttpObject comparitors to support test cases
-New test cases to support sending headers immediatley
-Bug fixes cleaned up to ensure the message flow is terminated properly

Result:
Netty HTTP/2 to HTTP/1.x translation layer will support the HTTP/2 draft message flow.
2014-08-25 11:34:15 -07:00
nmittler
5b1d50fa7c Changing HTTP/2 inbound flow control to use Http2FrameWriter
Motivation:

This is just some general cleanup to get rid of the FrameWriter inner
interface withing Http2InboundFlowController.  It's not necessary since
the flow controller can just use the Http2FrameWriter to send
WINDOW_UPDATE frames.

Modifications:

Updated DefaultHttp2InboundFlowController to use Http2FrameWriter.

Result:

The inbound flow control code is somewhat less smelly :).
2014-08-25 07:28:07 -07:00
Norman Maurer
8785ff5256 Move duplicated code into CodecUtil
Motivation:

We have some duplicated code that can be reused.

Modifications:

Create package private class called CodecUtil that now contains the shared code / helper method.

Result:

Less code-duplication
2014-08-25 07:37:35 +02:00
Norman Maurer
488f36d009 [#2812] Ensure we call checkForSharableAnnotation in all constructors of ByteToMessageCodec
Motivation:

ByteToMessageCodec miss to check for @Sharable annotation in one of its constructors.

Modifications:

Ensure we call checkForSharableAnnotation in all constructors.

Result:

After your change, what will change.
2014-08-23 21:03:57 +02:00
nmittler
3ffd205f57 Adding padding to HTTP/2 flow control.
Motivation:

We're currently out-of-spec with HTTP/2 in that we don't include padding
in the flow control logic.

Modifications:

Modified both DefaultHttp2InboundFlowController and
DefaultHttp2OutboundFlowController to properly take padding into
account. Outbound is more complicated since padding has to be properly
accounted for when splitting frames.

Result:

HTTP/2 codec properly flow controls padding.
2014-08-22 13:22:21 -07:00
Scott Mitchell
d3538dee2e HTTP/2 Priority Tree Restructure
Motivation:

The current interface and implementation for HTTP/2 priority tree events does not notify listeners of all parent change events. As a result operations which depend upon knowing about parent change events may be missing events and result in stale data. This interface also allows new listeners to easily consume priority tree change events.

Modifications:

-Http2Connection.Listener interface will change to support notifications on every node after the priority has changed and tree events have settled
-This will affect the outbound flow controller, DefaultHttp2Connection, and other listeners using the old interface

Result:
A modified (hopefully simplified and correct) Listener interface to get priority tree change event notification
2014-08-22 13:07:22 -07:00
nmittler
21bc279700 Refactoring outbound flow controller to use frame writer.
Motivation:

This is addressing a TODO in the outbound flow controller. We currently
have a separate writer interface passed into the outbound flow
controller. This is confusing and limiting as to how the flow controller
can perform its writes (e.g. no control over flushing). Instead it would
be better to just let the flow controller use the Http2FrameWriter
directly.

Modifications:

- Added a new Http2DataWriter interface, which is extended by
Http2FrameWriter and Http2OutboundFlowController.

- Removed automatic flushing from Http2DataWriter in order to facilitate
optimizing the case where there are multiple writes.

- Updated DefaultHttp2OutboundFlowController to properly optimize
flushing of the ChannelHandlerContext when multiple writes occur.

Result:

Code is greatly simplified WRT outbound flow control and flushes are
optimized for flow-controlled DATA frames.
2014-08-21 11:49:12 -07:00
Norman Maurer
d332c00e2f Reduce memory copies in spdy compression implementation.
Motivation:

Currently we do more memory copies then needed.

Modification:

- Directly use heap buffers to reduce memory copy
- Correctly release buffers to fix buffer leak

Result:

Less memory copies and no leaks
2014-08-21 11:32:36 +02:00
Norman Maurer
ee198d7cfc Allow efficient writing of CompositeByteBuf when using native epoll transport.
Motivation:

There were no way to efficient write a CompositeByteBuf as we always did a memory copy to a direct buffer in this case. This is not needed as we can just write a CompositeByteBuf as long as all the components are buffers with a memory address.

Modifications:

- Write CompositeByteBuf which contains only direct buffers without memory copy
- Also handle CompositeByteBuf that have more components then 1024.

Result:

More efficient writing of CompositeByteBuf.
2014-08-21 10:58:42 +02:00
Norman Maurer
a32652197b Avoid redundant reads of head in peakNode
Motivation:

There is not need todo redunant reads of head in peakNode as we can just spin on next() until it becomes visible.

Modifications:

Remove redundant reads of head in peakNode. This is based on @nitsanw's patch for akka.
See https://github.com/akka/akka/pull/15596

Result:

Less volatile access.
2014-08-21 09:01:34 +02:00
Norman Maurer
16a8f8b5aa Code-inspection fixes
Motivation:

Saw some code-inspection warnings

Modifications:

Fix warnings

Result:

Less code-inspection warnings
2014-08-21 07:20:00 +02:00
Norman Maurer
ceaab70153 [#2800] Fix NPE in SingleThreadEventExecutor on Android
Motivation:

The recent changes f8bee2e94c to use a FJP introduced a regression on Android that cause a NPE.

Modifications:

- Use AtomicReferenceFieldUpdater so it works also on Android
- Fix inspection warnings

Result:

Netty works again on Android too.
2014-08-21 06:31:56 +02:00
Norman Maurer
62e26a2818 [#2586] Use correct EventExecutor to notify for bind failures on late registration
Motivation:

We used the wrong EventExecutor to notify for bind failures if a late registration was done.

Modifications:

Use the correct EventExecutor to notify and only use the GlobelEventExecutor if the registration fails itself.

Result:

The correct Thread will do the notification.
2014-08-20 16:35:06 +02:00
Norman Maurer
5386c851c8 codec-socks test cleanup
Motivation:

The unit tests in codec-socks contained redundant casts and empty test classes.

Modifications:

- Remove redundant casts
- Delete empty test classes

Result:

Cleanup
2014-08-20 06:54:55 +02:00
Scott Mitchell
075190f711 Add HTTP/2 codec to netty-all maven dependency Motivation:
The HTTP/2 codec is missing from the netty-all maven dependency.
This means third party libraries have to pull in additional dependencies
above netty-all to use this codec.

Modifications:

Add HTTP/2 codec to netty-all

Result:

Third party libraries don't need to pull in additional dependencies
to use HTTP/2 codec.
2014-08-18 19:07:49 +02:00
nmittler
0c817d61b3 Upgrading to HTTP/2 draft 14 framing
Motivation:

HTTP/2 draft 14 came out a couple of weeks ago and we need to keep up
with the spec.

Modifications:

- Removed use of segment throughout.
- Added new setting for MAX_FRAME_SIZE. Used by the frame reader/writer
rather than a constant.
- Added new setting for MAX_HEADER_LIST_SIZE. This is currently unused.
- Expanded the header size to 9 bytes. The frame length field is now 3
bytes and added logic for checking that it falls within the valid range.

Result:

Netty will support HTTP/2 draft 14 framing. There will still be some
work to do to be compliant with the HTTP adaptation layer.
2014-08-18 07:39:36 -07:00
fredericBregier
26116541ed Fix example for Http Upload
Motivation:
The example mis handle two elements:
1) Last message is a LastHttpContent and is not taken into account by
the server handler
2) The client makes a sync on last write (chunked) but there is no flush
before, therefore the sync is waiting forever.

Modifications:
1) Take into account the message LastHttpContent in simple Get.
2) Removes sync but add flush for each post and multipost parts

Results:
Example is no more blocked after get test.

Should be done also in 4.0 and Master (similar changes)
2014-08-18 10:48:17 +02:00
Norman Maurer
d150ee7886 Document the correct default value of SOMAXCONN
Motivation:

Recently we changed the default value of SOMAXCONN that is used when we can not determine it by reading /proc/sys/net/core/somaxconn. While doing this we missed to update the javadocs to reflect the new default value that is used.

Modifications:

List correct default value in the javadocs of SOMAXCONN.

Result:

Correct javadocs.
2014-08-18 05:57:35 +02:00
Jakob Buchgraber
d7734861c8 Support IPv4 default route in ipfilter.
Motivation:

In GitHub issue #2767 a bug was reported that the IPv4
default route leads to the ipfilter package denying
instead of accepting all addresses.

While the issue was reported for Netty 3.9, this bug
also applies to Netty 4 and higher.

Modifications:

When computing the subnet address from the CIDR prefix,
correctly handle the case where the prefix is set to zero.

Result:

Ipfilter accepts all addresses when passed the
IPv4 default route.
2014-08-17 14:32:57 +02:00
Frédéric Brégier
5aa6bcde9e Better fix for TrafficShapingHandlerTest
Motivation:
The test procedure is unstable when testing quick time (factor less or equal to 1). Changing to default 10ms in this case will force time to be correct and time to be checked only when factor is >= 2.

Modifications:
When factor is <= 1, minimalWaitBetween is 10ms

Result:
Hoping this version is finally stable.
2014-08-16 18:08:43 +02:00
Frédéric Brégier
4c30067b58 Better fix for TrafficShapingHandlerTest
Motivation:
It seems that in certain conditions, the write back from the server is so quick that the handler has no time to compute traffic shaping. So 10ms of wait before acknowledging is added in server side.

Modifications:
Add 10ms waiting before server ackonwledge the client.

Result:
The timing is now suppsed to be stable.
2014-08-16 10:54:00 +02:00
Frédéric Brégier
4d20f03b6d Update TrafficShapingHandlerTest for Master
Motivation:
Test seems not yet stable (20ms instead of 40ms), so this small fix provides a "lightest" check.
If this is not passing again in master, one could deactivate this test by commenting the assertTrue in line 431.

Modification:
Divide by 2 the to be compared elapse time.

Result:
Locally this test is now passing.
2014-08-16 10:44:20 +02:00
Trustin Lee
138b366b00 Fix checkstyle 2014-08-15 11:09:42 -07:00
Trustin Lee
98dfc91471 Fix for issue #2765 relative to unstable trafficshaping test procedure
Motivation:

The test procedure is unstable due to not enough precise timestamping
during the check.

Modifications:

Reducing the test cases and cibling "stable" test ("timestamp-able")
bring more stability to the tests.

Result:

Tests for TrafficShapingHandler seem more stable (whatever using JVM 6,
7 or 8).
2014-08-15 10:21:11 -07:00
Trustin Lee
44d9163169 Fix a bug where ChannelOutboundBuffer.removeBytes() throws ClassCastException
When a ChannelOutboundBuffer contains ByteBufs followed by a FileRegion,
removeBytes() will fail with a ClassCastException.  It should break the
loop instead.
2014-08-15 09:57:12 -07:00
Trustin Lee
23d15c71ee Fix the regression caused by f31c630c8c
f31c630c8c was causing
SocketGatheringWriteTest to fail because it does not take the case where
an empty buffer exists in a gathering write.

When there is an empty buffer in a gathering write, the number of
buffers returned by ChannelOutboundBuffer.nioBuffer() and the actual
number of write attemps can differ.

To remove the write requests correctly, a byte transport must use
ChannelOutboundBuffer.removeBytes()
2014-08-15 09:44:05 -07:00
Norman Maurer
cb666c1169 [#2771] Correctly handle constructing of EmbeddedChannel
Motivation:

Because of an incorrect logic in teh EmbeddedChannel constructor it is not possible to use EmbeddedChannel with a ChannelInitializer as constructor argument. This is because it adds the internal LastInboundHandler to its ChannelPipeline before it register itself to the EventLoop.

Modifications:

First register self to EventLoop before add LastInboundHandler to the ChannelPipeline.

Result:

It's now possible to use EmbeddedChannel with ChannelInitializer.
2014-08-15 12:59:25 +02:00
Norman Maurer
050e7ed933 [#2769] Fix regression when writing different message types
Motivation:

Due a regression NioSocketChannel.doWrite(...) will throw a ClassCastException if you do something like:

channel.write(bytebuf);
channel.write(fileregion);
channel.flush();

Modifications:

Correctly handle writing of different message types by using the correct message count while loop over them.

Result:

No more ClassCastException
2014-08-15 11:56:27 +02:00
plucury
a2416481e3 Allow ChunkedInput to provide the progress of its transfer
Related issue: #2741 and #2151

Motivation:

There is no way for ChunkedWriteHandler to know the progress of the
transfer of a ChannelInput. Therefore, ChannelProgressiveFutureListener
cannot get exact information about the progress of the transfer.

If you add a few methods that optionally provides the transfer progress
to ChannelInput, it becomes possible for ChunkedWriteHandler to notify
ChannelProgressiveFutureListeners.

If the input has no definite length, we can still use the progress so
far, and consider the length of the input as 'undefined'.

Modifications:

- Add ChunkedInput.progress() and ChunkedInput.length()
- Modify ChunkedWriteHandler to use progress() and length() to notify
  the transfer progress

Result:

ChunkedWriteHandler now notifies ChannelProgressiveFutureListener.
2014-08-14 16:55:35 -07:00
Trustin Lee
d92875402d Overall clean-up on socksx package
- SocksV[45] -> Socks[45]
- Make encodeAsByteBuf package private with some hassle
- Split SocksMessageEncoder into Socks4MessageEncoder and
  Socks5MessageEncoder, and remove the original
- Remove lazy singleton instantiation; we don't need it.
- Remove the deprecated methods
- Fix Javadoc errors
2014-08-14 16:53:45 -07:00
bk1te
a242224646 Add socksx package which supports SOCKS 4/4a/5
Motivation:

SOCKS 4 and 5 are very different protocols although they share the same
name.  It is not possible to incorporate the two protocol versions into
a single package.

Modifications:

- Add a new package called 'socksx' to supercede 'socks' package.
- Add SOCKS 4/4a support to the 'socksx' package

Result:

codec-socks now supports all SOCKS versions
2014-08-14 16:52:43 -07:00
Trustin Lee
951ff902bd Reduce the fallback SOMAXCONN value
Related issue: #2407

Motivation:

The current fallback SOMAXCONN value is 3072.  It is way too large
comparing to the default SOMAXCONN value of popular OSes.

Modifications:

Decrease the fallback SOMAXCONN value to 128 or 200 depending on the
current OS

Result:

Saner fallback value
2014-08-14 15:40:18 -07:00
Trustin Lee
2af7a482d9 Rename SnappyFramedEncoder/Decoder to SnappyFrameEncoder/Decoder
Related issue: #2766

Motivation:

Forgot to rename them before the final release by mistake.

Modifications:

Rename the classes

Result:

Better naming
2014-08-14 15:19:15 -07:00
Idel Pivnitskiy
6a7fd374a2 Implemented LZ4 compression codec
Motivation:

LZ4 compression codec provides sending and receiving data encoded by very fast LZ4 algorithm.

Modifications:

- Added `lz4` library which implements LZ4 algorithm.
- Implemented Lz4FramedEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Added tests to verify the Lz4FramedEncoder and how it can compress data for the next uncompression using the original library.
- Implemented Lz4FramedDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages.
- Added tests to verify the Lz4FramedDecoder and how it can uncompress data after compression using the original library.
- Added integration tests for Lz4FramedEncoder/Decoder.

Result:

Full LZ4 compression codec which can compress/uncompress data using LZ4 algorithm.
2014-08-14 15:05:57 -07:00
Norman Maurer
3f10ed3082 [#2768] Correctly duplicate buffer for CloseWebSocketFrames
Motivation:

The _0XFF_0X00 buffer is not duplicated and empty after the first usage preventing the connection close to happen on subsequent close frames.

Modifications:

Correctly duplicate the buffer.

Result:

Multiple CloseWebSocketFrames are handled correctly.
2014-08-14 09:54:52 +02:00
Trustin Lee
83f978e734 Fix class description of FastLzFrameDecoder 2014-08-13 22:58:21 -07:00
Trustin Lee
2a9ace86af Rename FastLzFramed* to FastLzFrame* 2014-08-13 22:55:51 -07:00
Norman Maurer
1d1dafdc5e Fix and clearify javadocs
Motivation:

ByteToMessageDecoder and ReplayingDecoder have incorrect javadocs in some places.

Modifications:

Fix incorrect javadocs for both classes.

Result:

Correct javadocs for both classes
2014-08-14 06:46:41 +02:00
Trustin Lee
2ded0dd6fe Fix data corruption in FileRegion transfer with epoll transport
Related issue: #2764

Motivation:

EpollSocketChannel.writeFileRegion() does not handle the case where the
position of a FileRegion is non-zero properly.

Modifications:

- Improve SocketFileRegionTest so that it tests the cases where the file
  transfer begins from the middle of the file
- Add another jlong parameter named 'base_off' so that we can take the
  position of a FileRegion into account

Result:

Improved test passes. Corruption is gone.
2014-08-13 16:58:35 -07:00
Trustin Lee
5c24ca2216 Reduce the execution time of maven-antrun-plugin
Related issue: #2508

Motivation:

The '<exec/>' task takes unnecessarily long time due to a known issue:

- https://issues.apache.org/bugzilla/show_bug.cgi?id=54128

Modifications:

- Reduce the number of '<exec/>' tasks for faster build
- Use '<propertyregex/>' to extract the output

Result:

Slightly faster build
2014-08-13 15:45:00 -07:00
Trustin Lee
f1c757c089 Use heap buffers for Unpooled.copiedBuffer()
Related issue: #2028

Motivation:

Some copiedBuffer() methods in Unpooled allocated a direct buffer.  An
allocation of a direct buffer is an expensive operation, and thus should
be avoided for unpooled buffers.

Modifications:

- Use heap buffers in all copiedBuffer() methods

Result:

Unpooled.copiedBuffers() are less expensive now.
2014-08-13 15:09:41 -07:00
Norman Maurer
88e9c7366b Revert "[#2761] ChannelOutboundBuffer can cause data-corruption because of caching ByteBuffers"
This reverts commit c7be4e4937.
2014-08-13 21:46:36 +02:00
Norman Maurer
bb9ab03462 [#2761] Proper work-around for data-corruption caused by cached ByteBuffers
Motivation:

The previous fix did disable the caching of ByteBuffers completely which can cause performance regressions. This fix makes sure we use nioBuffers() for all writes in NioSocketChannel and so prevent data-corruptions. This is still kind of a workaround which will be replaced by a more fundamental fix later.

Modifications:

- Revert 4059c9f354
- Use nioBuffers() for all writes to prevent data-corruption

Result:

No more data-corruption but still retain the original speed.
2014-08-13 21:25:00 +02:00
Frédéric Brégier
8b23ff1171 Update TrafficShapingTest for Master
Motivation:
Master test seems not ok as in 4.1 in some tests so relax the internal time check but keeping the outside check.

Modifications:
Comment the assertTrue regarding the internal time test and extending to 15s the tests using Exec.

Result:
No more false negative tests (until Master is stable enough to recheck the condition of validation).
2014-08-13 19:20:42 +02:00
Norman Maurer
ed7364f384 Revert "[#2762] Not expand ByteBuffer[] in ChannelOutboundBuffer"
This reverts commit 8cfc64fa1d as it may cause problem if a CompositeByteBuf has more then 1024 components.
2014-08-13 16:36:40 +02:00
Norman Maurer
4a379f02a1 Change back default allocator to pooled.
Motivation:

While porting some changes from 4.0 to 4.1 and master branch I changed the default allocator from pooled to unpooled by mistake. This should be reverted. The guilty commit is 4a3ef90381.

Thanks to @blucas for spotting this.

Modifications:

Revert changes related to allocator.

Result:

Use the correct default allocator again.
2014-08-13 12:09:58 +02:00
Norman Maurer
8cfc64fa1d [#2762] Not expand ByteBuffer[] in ChannelOutboundBuffer
Motivation:

At the moment we expand the ByteBuffer[] when we have more then 1024 ByteBuffer to write and replace the stored instance in its FastThreadLocal. This is not needed and may even harm performance on linux as IOV_MAX is 1024 and so this may cause the JVM to do an array copy.

Modifications:

Just exit the nioBuffers() method if we can not fit more ByteBuffer in the array. This way we will pick them up on the next call.

Result:

Remove uncessary array copy and simplify the code.
2014-08-13 11:55:06 +02:00
Norman Maurer
c7be4e4937 [#2761] ChannelOutboundBuffer can cause data-corruption because of caching ByteBuffers
Motivation:

We cache the ByteBuffers in ChannelOutboundBuffer.nioBuffers() for the Entries in the ChannelOutboundBuffer to reduce some overhead. The problem is this can lead to data-corruption if an incomplete write happens and next time we try to do a non-gathering write.

To fix this we should remove the caching which does not help a lot anyway and just make the code buggy.

Modifications:

Remove the caching of ByteBuffers.

Result:

No more data-corruption.
2014-08-13 11:16:14 +02:00
fbregier
ba58f949e8 [#2721] Improve Traffic Shaping handler
Motivation:
Currently Traffic Shaping is using 1 timer only and could lead to
"partial" wrong bandwidth computation when "short" time occurs between
adding used bytes and when the TrafficCounter updates itself and finally
when the traffic is computed.
Indeed, the TrafficCounter is updated every x delay and it is at the
same time saved into "lastXxxxBytes" and set to 0. Therefore, when one
request the counter, it first updates the TrafficCounter with the added
used bytes. If this value is set just before the TrafficCounter is
updated, then the bandwidth computation will use the TrafficCounter with
a "0" value (this value being reset once the delay occurs). Therefore,
the traffic shaping computation is wrong in rare cases.

Secondly the traffic shapping should avoid if possible the "Timeout"
effect by not stopping reading or writing more than a maxTime, this
maxTime being less than the TimeOut limit.

Thirdly the traffic shapping in read had an issue since the readOp
was not set but should, turning in no read blocking from socket
point of view.

Modifications:
The TrafficCounter has 2 new methods that compute the time to wait
according to read or write) using in priority the currentXxxxBytes (as
before), but could used (if current is at 0) the lastXxxxxBytes, and
therefore having more chance to take into account the real traffic.

Moreover the Handler could change the default "max time to wait", which
is by default set to half of "standard" Time Out (30s:2 = 15s).

Finally we add the setAutoRead(boolean) accordingly to the situation,
as proposed in #2696 (this pull request is in error for unknown reason).

Result:
The Traffic Shaping is better take into account (no 0 value when it
shouldn't) and it tries to not block traffic more than Time Out event.

Moreover the read is really stopped from socket point of view.

This version is similar to #2388 and #2450.
This version is for Master, and includes the #2696 pull request
to ease the merge process.

Including also #2748

The test minimizes time check by reducing to 66ms steps (55s).

Conflicts:
	handler/src/main/java/io/netty/handler/traffic/AbstractTrafficShapingHandler.java
2014-08-13 02:21:07 +02:00