Commit Graph

6248 Commits

Author SHA1 Message Date
Frédéric Brégier
4c30067b58 Better fix for TrafficShapingHandlerTest
Motivation:
It seems that in certain conditions, the write back from the server is so quick that the handler has no time to compute traffic shaping. So 10ms of wait before acknowledging is added in server side.

Modifications:
Add 10ms waiting before server ackonwledge the client.

Result:
The timing is now suppsed to be stable.
2014-08-16 10:54:00 +02:00
Frédéric Brégier
4d20f03b6d Update TrafficShapingHandlerTest for Master
Motivation:
Test seems not yet stable (20ms instead of 40ms), so this small fix provides a "lightest" check.
If this is not passing again in master, one could deactivate this test by commenting the assertTrue in line 431.

Modification:
Divide by 2 the to be compared elapse time.

Result:
Locally this test is now passing.
2014-08-16 10:44:20 +02:00
Trustin Lee
138b366b00 Fix checkstyle 2014-08-15 11:09:42 -07:00
Trustin Lee
98dfc91471 Fix for issue #2765 relative to unstable trafficshaping test procedure
Motivation:

The test procedure is unstable due to not enough precise timestamping
during the check.

Modifications:

Reducing the test cases and cibling "stable" test ("timestamp-able")
bring more stability to the tests.

Result:

Tests for TrafficShapingHandler seem more stable (whatever using JVM 6,
7 or 8).
2014-08-15 10:21:11 -07:00
Trustin Lee
44d9163169 Fix a bug where ChannelOutboundBuffer.removeBytes() throws ClassCastException
When a ChannelOutboundBuffer contains ByteBufs followed by a FileRegion,
removeBytes() will fail with a ClassCastException.  It should break the
loop instead.
2014-08-15 09:57:12 -07:00
Trustin Lee
23d15c71ee Fix the regression caused by f31c630c8c
f31c630c8c was causing
SocketGatheringWriteTest to fail because it does not take the case where
an empty buffer exists in a gathering write.

When there is an empty buffer in a gathering write, the number of
buffers returned by ChannelOutboundBuffer.nioBuffer() and the actual
number of write attemps can differ.

To remove the write requests correctly, a byte transport must use
ChannelOutboundBuffer.removeBytes()
2014-08-15 09:44:05 -07:00
Norman Maurer
cb666c1169 [#2771] Correctly handle constructing of EmbeddedChannel
Motivation:

Because of an incorrect logic in teh EmbeddedChannel constructor it is not possible to use EmbeddedChannel with a ChannelInitializer as constructor argument. This is because it adds the internal LastInboundHandler to its ChannelPipeline before it register itself to the EventLoop.

Modifications:

First register self to EventLoop before add LastInboundHandler to the ChannelPipeline.

Result:

It's now possible to use EmbeddedChannel with ChannelInitializer.
2014-08-15 12:59:25 +02:00
Norman Maurer
050e7ed933 [#2769] Fix regression when writing different message types
Motivation:

Due a regression NioSocketChannel.doWrite(...) will throw a ClassCastException if you do something like:

channel.write(bytebuf);
channel.write(fileregion);
channel.flush();

Modifications:

Correctly handle writing of different message types by using the correct message count while loop over them.

Result:

No more ClassCastException
2014-08-15 11:56:27 +02:00
plucury
a2416481e3 Allow ChunkedInput to provide the progress of its transfer
Related issue: #2741 and #2151

Motivation:

There is no way for ChunkedWriteHandler to know the progress of the
transfer of a ChannelInput. Therefore, ChannelProgressiveFutureListener
cannot get exact information about the progress of the transfer.

If you add a few methods that optionally provides the transfer progress
to ChannelInput, it becomes possible for ChunkedWriteHandler to notify
ChannelProgressiveFutureListeners.

If the input has no definite length, we can still use the progress so
far, and consider the length of the input as 'undefined'.

Modifications:

- Add ChunkedInput.progress() and ChunkedInput.length()
- Modify ChunkedWriteHandler to use progress() and length() to notify
  the transfer progress

Result:

ChunkedWriteHandler now notifies ChannelProgressiveFutureListener.
2014-08-14 16:55:35 -07:00
Trustin Lee
d92875402d Overall clean-up on socksx package
- SocksV[45] -> Socks[45]
- Make encodeAsByteBuf package private with some hassle
- Split SocksMessageEncoder into Socks4MessageEncoder and
  Socks5MessageEncoder, and remove the original
- Remove lazy singleton instantiation; we don't need it.
- Remove the deprecated methods
- Fix Javadoc errors
2014-08-14 16:53:45 -07:00
bk1te
a242224646 Add socksx package which supports SOCKS 4/4a/5
Motivation:

SOCKS 4 and 5 are very different protocols although they share the same
name.  It is not possible to incorporate the two protocol versions into
a single package.

Modifications:

- Add a new package called 'socksx' to supercede 'socks' package.
- Add SOCKS 4/4a support to the 'socksx' package

Result:

codec-socks now supports all SOCKS versions
2014-08-14 16:52:43 -07:00
Trustin Lee
951ff902bd Reduce the fallback SOMAXCONN value
Related issue: #2407

Motivation:

The current fallback SOMAXCONN value is 3072.  It is way too large
comparing to the default SOMAXCONN value of popular OSes.

Modifications:

Decrease the fallback SOMAXCONN value to 128 or 200 depending on the
current OS

Result:

Saner fallback value
2014-08-14 15:40:18 -07:00
Trustin Lee
2af7a482d9 Rename SnappyFramedEncoder/Decoder to SnappyFrameEncoder/Decoder
Related issue: #2766

Motivation:

Forgot to rename them before the final release by mistake.

Modifications:

Rename the classes

Result:

Better naming
2014-08-14 15:19:15 -07:00
Idel Pivnitskiy
6a7fd374a2 Implemented LZ4 compression codec
Motivation:

LZ4 compression codec provides sending and receiving data encoded by very fast LZ4 algorithm.

Modifications:

- Added `lz4` library which implements LZ4 algorithm.
- Implemented Lz4FramedEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Added tests to verify the Lz4FramedEncoder and how it can compress data for the next uncompression using the original library.
- Implemented Lz4FramedDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages.
- Added tests to verify the Lz4FramedDecoder and how it can uncompress data after compression using the original library.
- Added integration tests for Lz4FramedEncoder/Decoder.

Result:

Full LZ4 compression codec which can compress/uncompress data using LZ4 algorithm.
2014-08-14 15:05:57 -07:00
Norman Maurer
3f10ed3082 [#2768] Correctly duplicate buffer for CloseWebSocketFrames
Motivation:

The _0XFF_0X00 buffer is not duplicated and empty after the first usage preventing the connection close to happen on subsequent close frames.

Modifications:

Correctly duplicate the buffer.

Result:

Multiple CloseWebSocketFrames are handled correctly.
2014-08-14 09:54:52 +02:00
Trustin Lee
83f978e734 Fix class description of FastLzFrameDecoder 2014-08-13 22:58:21 -07:00
Trustin Lee
2a9ace86af Rename FastLzFramed* to FastLzFrame* 2014-08-13 22:55:51 -07:00
Norman Maurer
1d1dafdc5e Fix and clearify javadocs
Motivation:

ByteToMessageDecoder and ReplayingDecoder have incorrect javadocs in some places.

Modifications:

Fix incorrect javadocs for both classes.

Result:

Correct javadocs for both classes
2014-08-14 06:46:41 +02:00
Trustin Lee
2ded0dd6fe Fix data corruption in FileRegion transfer with epoll transport
Related issue: #2764

Motivation:

EpollSocketChannel.writeFileRegion() does not handle the case where the
position of a FileRegion is non-zero properly.

Modifications:

- Improve SocketFileRegionTest so that it tests the cases where the file
  transfer begins from the middle of the file
- Add another jlong parameter named 'base_off' so that we can take the
  position of a FileRegion into account

Result:

Improved test passes. Corruption is gone.
2014-08-13 16:58:35 -07:00
Trustin Lee
5c24ca2216 Reduce the execution time of maven-antrun-plugin
Related issue: #2508

Motivation:

The '<exec/>' task takes unnecessarily long time due to a known issue:

- https://issues.apache.org/bugzilla/show_bug.cgi?id=54128

Modifications:

- Reduce the number of '<exec/>' tasks for faster build
- Use '<propertyregex/>' to extract the output

Result:

Slightly faster build
2014-08-13 15:45:00 -07:00
Trustin Lee
f1c757c089 Use heap buffers for Unpooled.copiedBuffer()
Related issue: #2028

Motivation:

Some copiedBuffer() methods in Unpooled allocated a direct buffer.  An
allocation of a direct buffer is an expensive operation, and thus should
be avoided for unpooled buffers.

Modifications:

- Use heap buffers in all copiedBuffer() methods

Result:

Unpooled.copiedBuffers() are less expensive now.
2014-08-13 15:09:41 -07:00
Norman Maurer
88e9c7366b Revert "[#2761] ChannelOutboundBuffer can cause data-corruption because of caching ByteBuffers"
This reverts commit c7be4e4937.
2014-08-13 21:46:36 +02:00
Norman Maurer
bb9ab03462 [#2761] Proper work-around for data-corruption caused by cached ByteBuffers
Motivation:

The previous fix did disable the caching of ByteBuffers completely which can cause performance regressions. This fix makes sure we use nioBuffers() for all writes in NioSocketChannel and so prevent data-corruptions. This is still kind of a workaround which will be replaced by a more fundamental fix later.

Modifications:

- Revert 4059c9f354
- Use nioBuffers() for all writes to prevent data-corruption

Result:

No more data-corruption but still retain the original speed.
2014-08-13 21:25:00 +02:00
Frédéric Brégier
8b23ff1171 Update TrafficShapingTest for Master
Motivation:
Master test seems not ok as in 4.1 in some tests so relax the internal time check but keeping the outside check.

Modifications:
Comment the assertTrue regarding the internal time test and extending to 15s the tests using Exec.

Result:
No more false negative tests (until Master is stable enough to recheck the condition of validation).
2014-08-13 19:20:42 +02:00
Norman Maurer
ed7364f384 Revert "[#2762] Not expand ByteBuffer[] in ChannelOutboundBuffer"
This reverts commit 8cfc64fa1d as it may cause problem if a CompositeByteBuf has more then 1024 components.
2014-08-13 16:36:40 +02:00
Norman Maurer
4a379f02a1 Change back default allocator to pooled.
Motivation:

While porting some changes from 4.0 to 4.1 and master branch I changed the default allocator from pooled to unpooled by mistake. This should be reverted. The guilty commit is 4a3ef90381.

Thanks to @blucas for spotting this.

Modifications:

Revert changes related to allocator.

Result:

Use the correct default allocator again.
2014-08-13 12:09:58 +02:00
Norman Maurer
8cfc64fa1d [#2762] Not expand ByteBuffer[] in ChannelOutboundBuffer
Motivation:

At the moment we expand the ByteBuffer[] when we have more then 1024 ByteBuffer to write and replace the stored instance in its FastThreadLocal. This is not needed and may even harm performance on linux as IOV_MAX is 1024 and so this may cause the JVM to do an array copy.

Modifications:

Just exit the nioBuffers() method if we can not fit more ByteBuffer in the array. This way we will pick them up on the next call.

Result:

Remove uncessary array copy and simplify the code.
2014-08-13 11:55:06 +02:00
Norman Maurer
c7be4e4937 [#2761] ChannelOutboundBuffer can cause data-corruption because of caching ByteBuffers
Motivation:

We cache the ByteBuffers in ChannelOutboundBuffer.nioBuffers() for the Entries in the ChannelOutboundBuffer to reduce some overhead. The problem is this can lead to data-corruption if an incomplete write happens and next time we try to do a non-gathering write.

To fix this we should remove the caching which does not help a lot anyway and just make the code buggy.

Modifications:

Remove the caching of ByteBuffers.

Result:

No more data-corruption.
2014-08-13 11:16:14 +02:00
fbregier
ba58f949e8 [#2721] Improve Traffic Shaping handler
Motivation:
Currently Traffic Shaping is using 1 timer only and could lead to
"partial" wrong bandwidth computation when "short" time occurs between
adding used bytes and when the TrafficCounter updates itself and finally
when the traffic is computed.
Indeed, the TrafficCounter is updated every x delay and it is at the
same time saved into "lastXxxxBytes" and set to 0. Therefore, when one
request the counter, it first updates the TrafficCounter with the added
used bytes. If this value is set just before the TrafficCounter is
updated, then the bandwidth computation will use the TrafficCounter with
a "0" value (this value being reset once the delay occurs). Therefore,
the traffic shaping computation is wrong in rare cases.

Secondly the traffic shapping should avoid if possible the "Timeout"
effect by not stopping reading or writing more than a maxTime, this
maxTime being less than the TimeOut limit.

Thirdly the traffic shapping in read had an issue since the readOp
was not set but should, turning in no read blocking from socket
point of view.

Modifications:
The TrafficCounter has 2 new methods that compute the time to wait
according to read or write) using in priority the currentXxxxBytes (as
before), but could used (if current is at 0) the lastXxxxxBytes, and
therefore having more chance to take into account the real traffic.

Moreover the Handler could change the default "max time to wait", which
is by default set to half of "standard" Time Out (30s:2 = 15s).

Finally we add the setAutoRead(boolean) accordingly to the situation,
as proposed in #2696 (this pull request is in error for unknown reason).

Result:
The Traffic Shaping is better take into account (no 0 value when it
shouldn't) and it tries to not block traffic more than Time Out event.

Moreover the read is really stopped from socket point of view.

This version is similar to #2388 and #2450.
This version is for Master, and includes the #2696 pull request
to ease the merge process.

Including also #2748

The test minimizes time check by reducing to 66ms steps (55s).

Conflicts:
	handler/src/main/java/io/netty/handler/traffic/AbstractTrafficShapingHandler.java
2014-08-13 02:21:07 +02:00
Idel Pivnitskiy
0b307fe083 Implemented FastLZ compression codec
Motivation:

FastLZ compression codec provides sending and receiving data encoded by fast FastLZ algorithm using block mode.

Modifications:

- Added part of `jfastlz` library which implements FastLZ algorithm. See FastLz class.
- Implemented FastLzFramedEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Implemented FastLzFramedDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages.
- Added integration tests for `FastLzFramedEncoder/Decoder`.

Result:

Full FastLZ compression codec which can compress/uncompress data using FastLZ algorithm.
2014-08-12 15:15:54 -07:00
Scott Mitchell
b41b11c53d Introduced HTTP/2 frame to HTTP/1.x translation layer.
Motivation:

The HTTP/2 codec currently provides direct callbacks to access stream events/data. The HTTP/2 codec provides the protocol support for HTTP/2 but it does not pass messages up the context pipeline. It would be nice to have a decoder which could collect the data framed by HTTP/2 and translate this into traditional HTTP type objects. This would allow the traditional Netty context pipeline to be used to separate processing concerns (i.e. HttpContentDecompressor).  It would also be good to have a layer which can translate FullHttp[Request|Response] objects into HTTP/2 frame outbound events.

Modifications:

Introduce a new InboundHttp2ToHttpAdapter and supporting classes which will translate HTTP/2 stream events/data into HttpObject objects. Introduce a new DelegatingHttp2HttpConnectionHandler which will translate FullHttp[Request|Response] objects to HTTP/2 frame events.

Result:

Introduced HTTP/2 frame events to HttpObject layer.
Introduced FullHttp[Request|Response] to HTTP/2 frame events.
Introduced new unit tests to support new code.
Updated HTTP/2 client example to use new code.
Miscelaneous updates and bug fixes made to support new code.
2014-08-12 12:31:26 -07:00
Trustin Lee
9b755d79e7 Do not throw an exception when failed to get a header
Motivation:

It is often very expensive to instantiate an exception. TextHeader
should not raise an exception when it failed to find a header or when
its header value is not valid.

Modification:

- Change the return type of the getter methods to Integer and Long so
  that null is returned when no header is found or its value is invalid
- Update Javadoc

Result:

- Fixes #2758
- No unnecessary instantiation of exceptions
2014-08-12 11:15:11 -07:00
jxu
7607eb8761 Add TextHeaders.getAndRemove(...) and its variants
Related issue: #2649 and #2745

Motivation:

At the moment there is no way to get and remove a header with one call.
This means you need to search the headers two times. We should add
getAndRemove(...) to allow doing so with one call.

Modifications:

Add getAndRemove(...) and getUnconvertedAndRemove(...) and their
variants

Result:

More efficient API
2014-08-12 10:33:17 -07:00
Norman Maurer
c5b6808645 Allow to obtain RecvByteBufAllocator.Handle to allow more flexible implementations
Motivation:

At the moment it's only possible for a user to set the RecvByteBufAllocator for a Channel but not access the Handle once it is assigned. This makes it hard to write more flexible implementations.

Modifications:

Add a new method to the Channel.Unsafe to allow access the the used Handle for the Channel. The RecvByteBufAllocator.Handle is created lazily.

Result:

It's possible to write more flexible implementatons that allow to adjust stuff on the fly for a Handle that is used by a Channel
2014-08-12 06:54:29 +02:00
Norman Maurer
7764b0e3de [#2752] Add PendingWriteQueue for queue up writes
Motivation:

Sometimes ChannelHandler need to queue writes to some point and then process these. We currently have no datastructure for this so the user will use an Queue or something like this. The problem is with this Channel.isWritable() will not work as expected and so the user risk to write to fast. That's exactly what happened in our SslHandler. For this purpose we need to add a special datastructure which will also take care of update the Channel and so be sure that Channel.isWritable() works as expected.

Modifications:

- Add PendingWriteQueue which can be used for this purpose
- Make use of PendingWriteQueue in SslHandler

Result:

It is now possible to queue writes in a ChannelHandler and still have Channel.isWritable() working as expected. This also fixes #2752.
2014-08-12 06:41:08 +02:00
Jakob Buchgraber
220660e351 Deregistration of a Channel from an EventLoop
Motivation:

After a channel is created it's usually assigned to an
EventLoop. During the lifetime of a Channel the
EventLoop is then responsible for processing all I/O
and compute tasks of the Channel.

For various reasons (e.g. load balancing) a user might
require the ability for a Channel to be assigned to
another EventLoop during its lifetime.

Modifications:

Introduce under the hood changes that ensure that Netty's
thread model is obeyed during and after the deregistration
of a channel.

Ensure that tasks (one time and periodic) are executed by
the right EventLoop at all times.

Result:

A Channel can be deregistered from one and re-registered with
another EventLoop.
2014-08-11 15:28:46 -07:00
Jakob Buchgraber
f8bee2e94c Make Nio/EpollEventLoop run on a ForkJoinPool
Related issue: #2250

Motivation:

Prior to this commit, Netty's non blocking EventLoops
were each assigned a fixed thread by which all of the
EventLoop's I/O and handler logic would be performed.

While this is a fine approach for most users of Netty,
some advanced users require more flexibility in
scheduling the EventLoops.

Modifications:

Remove all direct usages of threads in MultithreadEventExecutorGroup,
SingleThreadEventExecutor et al., and introduce an Executor
abstraction instead.

The way to think about this change is, that each
iteration of an eventloop is now a task that gets scheduled
in a ForkJoinPool.

While the ForkJoinPool is the default, one also has the
ability to plug in his/her own Executor (aka thread pool)
into a EventLoop(Group).

Result:

Netty hands off thread management to a ForkJoinPool by default.
Users can also provide their own thread pool implementation and
get some control over scheduling Netty's EventLoops
2014-08-11 15:28:46 -07:00
Trustin Lee
59cf8ffcb4 Fix a resource leak in StompSubframeAggregatorTest 2014-08-11 10:46:17 -07:00
Jeff Pinner
f7e183f8e4 SPDY: fix SpdySessionHandler::updateSendWindowSize
In Netty 3, downstream writes of SPDY data frames and upstream reads of
SPDY window udpate frames occur on different threads.

When receiving a window update frame, we synchronize on a java object
(SpdySessionHandler::flowControlLock) while sending any pending writes
that are now able to complete.

When writing a data frame, we check the send window size to see if we
are allowed to write it to the socket, or if we have to enqueue it as a
pending write. To prevent races with the window update frame, this is
also synchronized on the same SpdySessionHandler::flowControlLock.

In Netty 4, upstream and downstream operations on any given channel now
occur on the same thread. Since java locks are re-entrant, this now
allows downstream writes to occur while processing window update frames.

In particular, when we receive a window update frame that unblocks a
pending write, this write completes which triggers an event notification
on the response, which in turn triggers a write of a data frame. Since
this is on the same thread it re-enters the lock and modifies the send
window. When the write completes, we continue processing pending writes
without knowledge that the window size has been decremented.
2014-08-11 11:13:22 +02:00
Trustin Lee
3fc7518367 Fix resource leaks in StompSubframeDecoderTest 2014-08-08 11:26:10 -07:00
Norman Maurer
7022dcd4a1 [#2744] Fix flakey HashedWheelTimerTest.testExecutionOnTime()
Motivation:

The calculation of the max wait time for HashedWheelTimerTest.testExecutionOnTime() was wrong and so the test sometimes failed.

Modifications:

Fix the max wait time.

Result:

No more test-failures
2014-08-06 07:01:24 +02:00
Trustin Lee
653262d2af Fix resource leaks in StompSubframeAggregatorTest 2014-08-05 18:11:45 -07:00
Trustin Lee
168e81e9cd Fix a bug where SpdySession.getActiveStreams() returns incorrect set
Related issue: #2743

Motivation:

When there are more than one stream with the same priority, the set
returned by SpdySession.getActiveStream() will not include all of them,
because it uses TreeSet and only compares the priority of streams. If
two different streams have the same priority, one of them will be
discarded by TreeSet.

Modification:

- Rename getActiveStreams() to activeStreams()
- Replace PriorityComparator with StreamComparator

Result:

Two different streams with the same priority are compared correctly.
2014-08-05 17:46:13 -07:00
Trustin Lee
6ecca2722e Add test cases for HttpContentCompressor
- Ported from 386a06dbfa
2014-08-05 15:48:14 -07:00
Idel Pivnitskiy
12c614a7eb Consider writerIndex when LzfDecoder writes into a new heap buffer
Motivation:

Now LzfDecoder do not consider writerIndex when it writes into array of a new heap buffer (when it decodes a compressed chuck of data)
2014-08-05 22:51:21 +02:00
Trustin Lee
e59ec5384d Fix a bug where ChannelFuture.setFailure(null) doesn't fail
Motivation:

We forgot to do a null check on the cause parameter of
ChannelFuture.setFailure(cause)

Modifications:

Add a null check

Result:

Fixed issue: #2728
2014-08-05 11:23:52 -07:00
Norman Maurer
4a3ef90381 Port ChannelOutboundBuffer and related changes from 4.0
Motivation:

We did various changes related to the ChannelOutboundBuffer in 4.0 branch. This commit port all of them over and so make sure our branches are synced in terms of these changes.

Related to [#2734], [#2709], [#2729], [#2710] and [#2693] .

Modification:
Port all changes that was done on the ChannelOutboundBuffer.

This includes the port of the following commits:
 - 73dfd7c01b
 - 997d8c32d2
 - e282e504f1
 - 5e5d1a58fd
 - 8ee3575e72
 - d6f0d12a86
 - 16e50765d1
 - 3f3e66c31a

Result:
 - Less memory usage by ChannelOutboundBuffer
 - Same code as in 4.0 branch
 - Make it possible to use ChannelOutboundBuffer with Channel implementation that not extends AbstractChannel
2014-08-05 15:00:56 +02:00
Norman Maurer
2258b32549 [#2732] HttpRequestEncoder may produce invalid uri if uri parameters are included.
Motivation:

If the requests contains uri parameters but not path the HttpRequestEncoder does produce an invalid uri while try to add the missing path.

Modifications:

Correctly handle the case of uri with paramaters but no path.

Result:

HttpRequestEncoder produce correct uri in all cases.
2014-08-05 10:14:06 +02:00
Trustin Lee
730bfd4fdb Add more utility methods to check the availability of the epoll transport
Related issue: #2733

Motivation:

Unlike OpenSsl, Epoll lacks a couple useful availability checker
methods:

- ensureAvailability()
- unavailabilityCause()

Modifications:

Add missing methods

Result:

More ways to check the availability and to get the cause of
unavailability programatically.
2014-08-04 15:05:13 -07:00
Trustin Lee
2506dc778c Clean-up d9cccccbb3
- Revert irrelevant formatting changes
- Rename resource files
  - Add .pem
  - Remove 'netty' from names
2014-08-04 10:54:13 -07:00