Commit Graph

6045 Commits

Author SHA1 Message Date
Idel Pivnitskiy
401a6db84e Implemented a Bzip2Encoder
Motivation:

Bzip2Encoder provides sending data compressed in bzip2 format.

Modifications:

Added classes:
- Bzip2Encoder
- Bzip2BitWriter
- Bzip2BlockCompressor
- Bzip2DivSufSort
- Bzip2HuffmanAllocator
- Bzip2HuffmanStageEncoder
- Bzip2MTFAndRLE2StageEncoder
- Bzip2EncoderTest

Modified classes:
- Bzip2Constants (splited BLOCK_HEADER_MAGIC and END_OF_STREAM_MAGIC)
- Bzip2Decoder (use splited magic numbers)

Added integration tests for Bzip2Encoder/Decoder

Result:

Implemented new encoder which can compress outgoing data in bzip2 format.
2014-07-17 16:19:59 +02:00
Idel Pivnitskiy
2b37b692d8 Implemented LZF compression codec
Motivation:

LZF compression codec provides sending and receiving data encoded by very fast LZF algorithm.

Modifications:

- Added Compress-LZF library which implements LZF algorithm
- Implemented LzfEncoder which extends MessageToByteEncoder and provides compression of outgoing messages
- Added tests to verify the LzfEncoder and how it can compress data for the next uncompression using the original library
- Implemented LzfDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages
- Added tests to verify the LzfDecoder and how it can uncompress data after compression using the original library
- Added integration tests for LzfEncoder/Decoder

Result:

Full LZF compression codec which can compress/uncompress data using LZF algorithm.
2014-07-17 07:18:53 +02:00
Gernot Pansy
d8a78424b3 Fixed NPE in WebSocket00FrameDecoder if end couldn't be found in text frame
Motivation:
When we receive an incomplete WebSocketFrame we need to make sure to wait for more data. Because we not did this we could produce a NPE.

Modification:
Make sure we not try to add null into the RecyclableArrayList

Result:
no more NPE on incomplete frames.
2014-07-16 20:04:21 +02:00
Norman Maurer
7e76cbd689 [#2653] Remove unnecessary ensureAccessible() calls
Motivation:

I introduced ensureAccessible() class as part of 6c47cc9711 in some places. Unfortunally I also added some where these are not needed and so caused a performance regression.

Modification:

Remove calls where not needed.

Result:

Fixed performance regression.
2014-07-14 21:04:21 +02:00
Norman Maurer
115b246ca1 [#2653] Remove uncessary range checks for performance reasons
Motivation:

I introduced range checks as part of 6c47cc9711 in some places. Unfortunally I also added some where these are not needed and so caused a performance regression.

Modification:

Remove range checks where not needed

Result:

Fixed performance regression.
2014-07-14 11:43:00 +02:00
Norman Maurer
0eb64b5a5d Fix over-sensible testcase 2014-07-13 17:19:47 +02:00
Norman Maurer
1cdbbe0f9b [#2651] Fix possible infinite-loop when cancel tasks
Motivations:
In our new version of HWT we used some kind of lazy cancelation of timeouts by put them back in the queue and let them pick up on the next tick. This  multiple problems:
 - we may corrupt the MpscLinkedQueue if the task is used as tombstone
 - this sometimes lead to an uncessary delay especially when someone did executed some "heavy" logic in the TimeTask

Modifications:
Use a Lock per HashedWheelBucket for save and fast removal.

Modifications:
Cancellation of tasks can be done fast and so stuff can be GC'ed and no more infinite-loop possible
2014-07-11 15:41:44 +02:00
Norman Maurer
edc7abb461 [#2650] Allow to disable http header validation in SpdyHttpDecoder and SpdyHttpCodec
Motivation:

HTTP header validation can be expensive so we should allow to disable it like we do in HttpObjectDecoder.

Modification:

Add constructor argument to disable validation.

Result:
Performance improvement
2014-07-11 08:31:39 +02:00
Norman Maurer
dfa5a88978 Reuse previous created HttpHeaders by HttpObjectAggregator
Motivation:

HttpObjectAggregator currently creates a new FullHttpResponse / FullHttpRequest for each message it needs to aggregate. While doing so it also creates 2 DefaultHttpHeader instances (one for the headers and one for the trailing headers). This is bad for two reasons:
  - More objects are created then needed and also populate the headers is not for free
  - Headers may get validated even if the validation was disabled in the decoder

Modification:

- Wrap the previous created HttpResponse / HttpRequest and so reuse the original HttpHeaders
- Reuse the previous created trailing HttpHeader.
- Fix a bug where the trailing HttpHeader was incorrectly mixed in the headers.

Result:

- Less GC
- Faster HttpObjectAggregator implementation
2014-07-11 07:23:58 +02:00
Norman Maurer
34febdc987 Fix checkstyle error introduced by 52cb55d388 2014-07-10 07:12:21 +02:00
Norman Maurer
52cb55d388 [#2643] Throw TooLongFrameException instead of using fireExceptionCaught
Motivation:

It's not always the case that there is another handler in the pipeline that will intercept the exceptionCaught event because sometimes users just sub-class. In this case the exception will just hit the end of the pipeline.

Modification:
Throw the TooLongFrameException so that sub-classes can handle it in the exceptionCaught(...) method directly.

Result:
Sub-classes can correctly handle the exception,
2014-07-10 06:53:51 +02:00
Norman Maurer
639db72839 Fix buffer leak in test introduced by 282d6e73b8 2014-07-10 06:43:09 +02:00
fboucher38
282d6e73b8 Add WebSocket Extension handlers for client and server.
Add permessage-deflate and deflate-frame WebSocket extension
implementations.

Motivation:

Need to compress HTTP WebSocket frames payload.

Modifications:

- Move UTF8 checking of WebSocketFrames from frame decoder to and
external handler.
- Change ZlibCodecFactory in order to use ZLib implentation instead of
JDK if windowSize or memLevel is different from the default one.
- Add WebSocketServerExtensionHandler and
WebSocketClientExtensionHandler to handle WebSocket Extension headers.
- Add DeflateFrame and PermessageDeflate extension implementations.
2014-07-09 16:28:40 +02:00
Daniel Bevenius
cbe20faf6c Adding payload to the Http2Client example.
Modifications:
When trying out the Http2Client example I noticed that adding a request
payload would not cause a data frame to written. This seems to be
because writeHeaders completes the promise, and then the writeData
call ends up in FlowControlWriter.writeFrame, isDone is true and
the data released and the call aborted and returned.
Adding a new promise for the writeData method allows a data frame to be
written.

Result:
A body/payload can now be sent to the server. The example was updated to
simply echo the payload received back to the calling client.
2014-07-09 13:53:01 +02:00
Idel Pivnitskiy
9d08a34c34 Moved bit-level read operations from Bzip2Decoder to the new Bzip2BitReader
Motivation:

Collect all bit-level read operations in one class is better. And now it's easy to use not only in Bzip2Decoder. For example, in Bzip2HuffmanStageDecoder.

Modifications:

Created a new class - Bzip2BitReader which provides bit-level reads.
Removed bit-level read operations from Bzip2Decoder.
Improved javadoc.

Result:

Bzip2BitReader allows the reading of single bit booleans, bit strings of arbitrary length (up to 24 bits), and bit aligned 32-bit integers.
2014-07-09 13:48:29 +02:00
Norman Maurer
6b71089373 Fix JVM segfault during JNI call. Part of [#2647]
Motivation:
Currently when Native.writev(...) is used it is possible to see a JVM segfault because the offset is updated to early.

Modification:
Only update the offset once it is safe to do so.

Result:
No more segfault
2014-07-09 13:36:15 +02:00
Norman Maurer
578068c7e6 [#2647] Respect IOV_MAX when call writev in native transport
Motivation:

epoll transport fails on gathering write of more then 1024 buffers. As linux supports max. 1024 iov entries when calling writev(...) the epoll transport throws an exception.

Thanks again to @blucas to provide me with a reproducer and so helped me to understand what the issue is.

Modifications:

Make sure we break down the writes if to many buffers are uses for gathering writes.

Result:

Gathering writes work with any number of buffers
2014-07-09 12:18:48 +02:00
nmittler
67159e7119 Upgrade to HTTP/2 draft 13
Motivation:

Need to upgrade HTTP/2 implementation to latest draft.

Modifications:

Various changes to support draft 13.

Result:

Support for HTTP/2 draft 13.
2014-07-08 21:10:46 +02:00
Brendt Lucas
58cc16d106 [#2642] CompositeByteBuf.deallocate memory/GC improvement
Motivation:

CompositeByteBuf.deallocate generates unnecessary GC pressure when using the 'foreach' loop, as a 'foreach' loop creates an iterator when looping.

Modification:

Convert 'foreach' loop into regular 'for' loop.

Result:

Less GC pressure (and possibly more throughput) as the 'for' loop does not create an iterator
2014-07-08 21:04:55 +02:00
Norman Maurer
696e355a70 Move generic code to HttpOrSpdyChooser to simplify implementations
Motivation:

HttpOrSpdyChooser can be simplified so the user not need to implement getProtocol(...) method.

Modification:

Add implementation for the method. The user can override it if necessary.

Result:

Easier usage of HttpOrSpdyChooser.
2014-07-07 09:44:50 +02:00
Trustin Lee
93a265ed0a Add missing m2eclipse life cycle mapping
Also, use ignore instead of execution because those plugins are not
really useful when building from Eclipse anyway.
2014-07-06 16:53:42 +09:00
Trustin Lee
65199873c2 Fix another buffer leaks in JsonObjectDecoderTest 2014-07-04 16:11:38 +09:00
Trustin Lee
3921f7c88a Reduce the perceived time taken to retrieve initialSeedUniquifier
Motivation:

When system is in short of entrophy, the initialization of
ThreadLocalRandom can take at most 3 seconds.  The initialization occurs
when ThreadLocalRandom.current() is invoked first time, which might be
much later than the moment when the application has started.  If we
start the initialization of ThreadLocalRandom as early as possible, we
can reduce the perceived time taken for the retrieval.

Modification:

Begin the initialization of ThreadLocalRandom in InternalLoggerFactory,
potentially one of the firstly initialized class in a Netty application.

Make DefaultChannelId retrieve the current process ID before retrieving
the current machine ID, because retrieval of a machine ID is more likely
to use ThreadLocalRandom.current().

Use a dummy channel ID for EmbeddedChannel, which prevents many unit
tests from creating a ThreadLocalRandom instance.

Result:

We gain extra 100ms at minimum for initialSeedUniquifier generation.  If
an application has its own initialization that takes long enough time
and generates good amount of entrophy, it is very likely that we will
gain a lot more.
2014-07-04 16:04:37 +09:00
Trustin Lee
b4c2ae5e59 Log the time taken for generating the initialSeedUniquifier
- Sometimes useful to know it how long it takes from the log, to make
  sure it's not something else that is blocking.
2014-07-04 13:25:26 +09:00
Trustin Lee
13a0a36db3 Make JsonObjectDecoder discard everything after stream corruption
Motivation:

There's no way to recover from a corrupted JSON stream. The current
implementation will raise an infinite exception storm when a peer sends
a large corrupted stream.

Modification:

Discard everything once stream corruption is detected.

Result:

Fixes a buffer leak
Fixes exception storm
2014-07-04 11:15:43 +09:00
Norman Maurer
9c4a733471 [#2586] Use correct EventLoop to notify delayed bind failures
Motivation:

When a bind fails AbstractBootstrap will use the GlobalEventExecutor to notify the ChannelPromise. We should use the EventLoop of the Channel if possible.

Modification:

Use EventLoop of the Channel if possible to use the correct Thread to notify and so guaranteer the right order of events.

Result:

Use the correct EventLoop for notification
2014-07-03 21:31:27 +02:00
Norman Maurer
2438ec5e24 [#2618] Introduce ChannelPromise.unvoid() and ChannelFuture.isVoid()
Motivation:

There is no way for a ChannelHandler to check if the passed in ChannelPromise for a write(...) call is a VoidChannelPromise. This is a problem as some handlers need to add listeners to the ChannelPromise which is not possible in the case of a VoidChannelPromise.

Modification:

- Introduce ChannelFuture.isVoid() which will return true if it is not possible to add listeners or wait on the result.
- Add ChannelPromise.unvoid() which allows to create a ChannelFuture out of a void ChannelFuture which supports all the operations.

Result:

It's now easy to write ChannelHandler implementations which also works when a void ChannelPromise is used.
2014-07-03 14:16:46 +02:00
Trustin Lee
9c4970363e Fix an inspector warning in JsonObjectDecoder 2014-07-03 20:01:23 +09:00
Trustin Lee
74d0c693fb Fix a buffer leak in JsonObjectDecoderTest 2014-07-03 19:58:31 +09:00
Jakob Buchgraber
c63378cd30 Split a JSON byte stream into JSON objects/arrays. Fixes #2536
Motivation:

See GitHub Issue #2536.

Modifications:

Introduce the class JsonObjectDecoder to split a JSON byte stream
into individual JSON objets/arrays.

Result:

A Netty application can now handle a byte stream where multiple JSON
documents follow eachother as opposed to only a single JSON document
per request.
2014-07-03 18:33:48 +09:00
Trustin Lee
8784fd627b Let OkResponseHandler extend SimpleChannelInboundHandler
Motivation:

OkResponseHandler is the last handler in the pipeline of the HTTP CORS
example.  It is responsible for releasing all messages it handled.

Modification:

Extend SimpleChannelInboundHandler instead of ChannelHandlerAdapter

Result:

Fixed a leak
2014-07-03 18:15:23 +09:00
Trustin Lee
71737a2a65 Fix the build timeout when 'leak' profile is active
Motivation:

AbstractByteBufTest.testInternalBuffer() uses writeByte() operations to
populate the sample data.  Usually, this isn't a problem, but it starts
to take a lot of time when the resource leak detection level gets
higher.

In our CI machine, testInternalBuffer() takes more than 30 minutes,
causing the build timeout when the 'leak' profile is active (paranoid
level resource detection.)

Modification:

Populate the sample data using ThreadLocalRandom.nextBytes() instead of
using millions of writeByte() operations.

Result:

Test runs much faster when leak detection level is high.
2014-07-03 17:51:15 +09:00
Daniel Bevenius
43bfc244cf Changed Http2ClientConnectionHandler to print out the aggregated buffers content
Motivation:
When running the Http2Client the data returned from the server, the
"Hello World" string, is supposed to be printed but instead the following
is displayed:
Received message:

Modifications:
Use the aggregated buffer to print out.

Result:
The example now logs the correct data sent from the server:
Received message: Hello World
2014-07-03 07:06:19 +02:00
nmittler
812e1719d7 Removing unused Http2PrefaceHandler.
Motivation:

All of the functionality has been merged into
AbstractHttp2ConnectionHandler.

Modifications:

Removed the Http2PrefaceHandler/Test and all uses.

Result:

Code cleaned up a bit.
2014-07-02 07:10:55 -07:00
Trustin Lee
ba81831e0a Fix the cruft produced from the refactoring
.. in 330404da07
2014-07-02 20:30:45 +09:00
Trustin Lee
330404da07 Fix most inspector warnings
Motivation:

It's good to minimize potentially broken windows.

Modifications:

Fix most inspector warnings from our profile

Result:

Cleaner code
2014-07-02 19:04:11 +09:00
Trustin Lee
cea3b6b2ab Clean up HTTP/2 integration tests
Related pull request: #2481 written by @nmittler

Motivation:

Some tests were still sending requests from the test thread rather than
from the event loop.

Modifications:

- Modified the roundtrip tests to use a new utility Http2TestUtil to
  perform the write operations in the event loop thread.
- Modified the Http2Client under examples to write all requests in the
  event loop thread.
- Added HttpPrefaceHandler and its test which were missing.
- Fixed various inspector warnings

Result:

Tests and examples now send requests in the correct thread context.
2014-07-02 16:58:24 +09:00
Trustin Lee
66d8d8219a Fix inspector warnings
- Move the methods used by an inner class to the inner class
- Removal of various redundant things (throws, parens, ..)
2014-07-02 16:47:48 +09:00
Norman Maurer
750f023ced [#2558] Define SO_REUSEPORT if not defined
Motivation:

Currently it is impossible to build netty on linux system that not define SO_REUSEPORT even if it is supported.

Modification:

Define SO_REUSEPORT if not defined.

Result:

Possible to build on more linux dists.
2014-07-02 09:40:36 +02:00
Norman Maurer
887838a06b Correctly return from selector loop one a scheduled task is ready for processing
Motivation:

We use the nanoTime of the scheduledTasks to calculate the milli-seconds to wait for a select operation to select something. Once these elapsed we check if there was something selected or some task is ready for processing. Unfortunally we not take into account scheduled tasks here so the selection loop will continue if only scheduled tasks are ready for processing. This will delay the execution of these tasks.

Modification:

- Check if a scheduled task is ready after selecting
- also make a tiny change in NioEventLoop to not trigger a rebuild if nothing was selected because the timeout was reached a few times in a row.

Result:

Execute scheduled tasks on time.
2014-07-02 09:10:49 +02:00
Norman Maurer
5a6719406c [#2623] Release local references to guard against StackOverflow in JNI
Motivation:

When we do a (env*)->GetObjectArrayElement(...) call we may created many local references which will only be cleaned up once we exist the native method. Thus a lot of memory can be used and so a StackOverFlow may be triggered. Beside this the JNI specification only say that an implementation must cope with 16 local references.

Modification:

Call (env*)->ReleaseLocalRef(...) to release the resource once not needed anymore.

Result:

Less memory usage and guard against StackOverflow
2014-06-30 09:53:13 +02:00
Norman Maurer
85b39d894f [#2622] Correctly check reference count before try to work on the underlying memory
Motivation:

Because of how we use reference counting we need to check for the reference count before each operation that touches the underlying memory. This is especially true as we use sun.misc.Cleaner.clean() to release the memory ASAP when possible. Because of this the user may cause a SEGFAULT if an operation is called that tries to access the backing memory after it was released.

Modification:

Correctly check the reference count on all methods that access the underlying memory or expose it via a ByteBuffer.

Result:

Safer usage of ByteBuf
2014-06-30 07:14:37 +02:00
Norman Maurer
bebcf81687 Fix compile error 2014-06-28 22:27:15 +02:00
Jay
db3c592e6a Maintain decoder result in HttpObjectAggregator
Motivation:
DecodeResult is dropped when aggregate HTTP messages.

Modification:

Make sure we not drop the DecodeResult while aggregate HTTP messages.

Result:

Correctly include the DecodeResult for later processing.
2014-06-28 22:09:46 +02:00
Michael Klishin
c8b170e9eb Correct a JavaDoc typo 2014-06-28 21:44:48 +02:00
xmxsuperstar
85993e10c1 fix example missing break statement in ReplayingDecoder 2014-06-28 21:41:49 +02:00
Norman Maurer
0ee7f9764e Cleanup comment / code 2014-06-27 21:37:21 +02:00
Norman Maurer
773be9787d Let EpollReuseAddrTest also work with kernel versions that not have bugfix release part 2014-06-27 17:55:55 +02:00
Norman Maurer
dbd041efc1 [#2615] Correctly update SelectionKey after selector rebuild
Motivation:

When a select rebuild was triggered the reference to the SelectionKey is not updated in AbstractNioChannel. This will cause a CancelledKeyException later.

Modification:

Correctly update SelectionKey reference after rebuild

Result:

Fix exception
2014-06-27 17:21:48 +02:00
Trustin Lee
bb9ebb4bd0 Upgrade JACOCO to the latest version
.. to fix the build errors triggered by the old JACOCO release
2014-06-27 17:24:05 +09:00