Commit Graph

6030 Commits

Author SHA1 Message Date
Idel Pivnitskiy
dc9d933d74 Fixes for compression codecs
Motivation:

Fixed founded mistakes in compression codecs.

Modifications:

- Changed return type of ZlibUtil.inflaterException() from CompressionException to DecompressionException
- Updated @throws in javadoc of JZlibDecoder to throw DecompressionException instead of CompressionException
- Fixed JdkZlibDecoder to throw DecompressionException instead of CompressionException
- Removed unnecessary empty lines in JdkZlibEncoder and JZlibEncoder
- Removed public modifier from Snappy class
- Added MAX_UNCOMPRESSED_DATA_SIZE constant in SnappyFramedDecoder
- Used in.readableBytes() instead of (in.writerIndex() - in.readerIndex()) in SnappyFramedDecoder
- Added private modifier for enum ChunkType in SnappyFramedDecoder
- Fixed potential bug (sum overflow) in Bzip2HuffmanAllocator.first(). For more info, see http://googleresearch.blogspot.ru/2006/06/extra-extra-read-all-about-it-nearly.html

Result:

Fixed sum overflow in Bzip2HuffmanAllocator, improved exceptions in ZlibDecoder implementations, hid Snappy class
2014-07-20 09:32:53 +02:00
Idel Pivnitskiy
ad1389be9d Small performance improvements
Modifications:

- Added a static modifier for CompositeByteBuf.Component.
This class is an inner class, but does not use its embedded reference to the object which created it. This reference makes the instances of the class larger, and may keep the reference to the creator object alive longer than necessary.
- Removed unnecessary boxing/unboxing operations in HttpResponseDecoder, RtspResponseDecoder, PerMessageDeflateClientExtensionHandshaker and PerMessageDeflateServerExtensionHandshaker
A boxed primitive is created from a String, just to extract the unboxed primitive value.
- Removed unnecessary 3 times calculations in DiskAttribute.addContent(...).
- Removed unnecessary checks if file exists before call mkdirs() in NativeLibraryLoader and PlatformDependent.
Because the method mkdirs() has this check inside.
- Removed unnecessary `instanceof AsciiString` check in StompSubframeAggregator.contentLength(StompHeadersSubframe) and StompSubframeDecoder.getContentLength(StompHeaders, long).
Because StompHeaders.get(CharSequence) always returns java.lang.String.
2014-07-20 09:26:04 +02:00
Idel Pivnitskiy
0cc3eccc2b Close ObjectInputStream in ObjectDecoder.decode(...)
Motivation:

We create a new CompactObjectInputStream with ByteBufInputStream in ObjectDecoder.decode(...) method and don't close this InputStreams before return statement.

Modifications:

Save link to the ObjectInputStream and close it before return statement.

Result:

Close InputStreams and clean up unused resources. It will be better for GC.
2014-07-20 09:23:35 +02:00
Norman Maurer
53141b04a8 Fix buffer leak in Bzip2EncoderTest 2014-07-19 14:41:18 +02:00
Norman Maurer
aa66f556e5 [#2667] Write until EAGAIN in native transport and only call setEpollOut() in this case
Motivation:

In the previous fix for #2667 I did introduce a bit overhead by calling setEpollOut() too often.

Modification:

Only call setEpollOut() if really needed and remove unused code.

Result:

Less overhead when saturate network.
2014-07-18 20:44:37 +02:00
Norman Maurer
7e61538790 [#2665] Continue writing on IOException when using DatagramChannel
Motivation:

As a DatagramChannel supports to write to multiple remote peers we must not close the Channel once a IOException accours as this error may be only valid for one remote peer.

Modification:

Continue writing on IOException.

Result:

DatagramChannel can be used even after an IOException accours during writing.
2014-07-18 12:49:34 +02:00
Norman Maurer
14096d85cd [#2667] Write until EAGAIN in native transport
Motivation:

We need to continue write until we hit EAGAIN to make sure we not see an starvation

Modification:

Write until EAGAIN is returned

Result:

No starvation when using native transport with ET.
2014-07-18 11:25:37 +02:00
Norman Maurer
89593a719b [#2666] Fix possible NPE when try to fullfill connect ChannelPromise
Motivation:

Because of a missing return statement we may produce a NPE when try to fullfill the connect ChannelPromise when it was fullfilled before.

Modification:

Add missing return statement.

Result:

No more NPE.
2014-07-18 07:10:11 +02:00
Norman Maurer
4db6c65534 [#2647] Handle IOV_MAX in java code
Motivation:

The handling of IOV_MAX was done in JNI code base which makes stuff really complicated to maintain etc.

Modifications:

Move handling of IOV_MAX to java code to simplify stuff

Result:

Cleaner code.
2014-07-18 06:59:12 +02:00
Norman Maurer
89261fd8df [#2664] Support write-spinning also in native transport
Motivation:

In our nio implementation we use write-spinning for maximize throughput, but in the native implementation this is not used.

Modification:

Respect writeSpinCount in native transport.

Result:

Better throughput
2014-07-18 06:33:16 +02:00
Idel Pivnitskiy
ed7240b597 Implemented a Bzip2Encoder
Motivation:

Bzip2Encoder provides sending data compressed in bzip2 format.

Modifications:

Added classes:
- Bzip2Encoder
- Bzip2BitWriter
- Bzip2BlockCompressor
- Bzip2DivSufSort
- Bzip2HuffmanAllocator
- Bzip2HuffmanStageEncoder
- Bzip2MTFAndRLE2StageEncoder
- Bzip2EncoderTest

Modified classes:
- Bzip2Constants (splited BLOCK_HEADER_MAGIC and END_OF_STREAM_MAGIC)
- Bzip2Decoder (use splited magic numbers)

Added integration tests for Bzip2Encoder/Decoder

Result:

Implemented new encoder which can compress outgoing data in bzip2 format.
2014-07-17 16:19:39 +02:00
Idel Pivnitskiy
3c6017a9b1 Implemented LZF compression codec
Motivation:

LZF compression codec provides sending and receiving data encoded by very fast LZF algorithm.

Modifications:

- Added Compress-LZF library which implements LZF algorithm
- Implemented LzfEncoder which extends MessageToByteEncoder and provides compression of outgoing messages
- Added tests to verify the LzfEncoder and how it can compress data for the next uncompression using the original library
- Implemented LzfDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages
- Added tests to verify the LzfDecoder and how it can uncompress data after compression using the original library
- Added integration tests for LzfEncoder/Decoder

Result:

Full LZF compression codec which can compress/uncompress data using LZF algorithm.
2014-07-17 07:18:07 +02:00
Gernot Pansy
7d8d9b2c6e Fixed NPE in WebSocket00FrameDecoder if end couldn't be found in text frame
Motivation:
When we receive an incomplete WebSocketFrame we need to make sure to wait for more data. Because we not did this we could produce a NPE.

Modification:
Make sure we not try to add null into the RecyclableArrayList

Result:
no more NPE on incomplete frames.
2014-07-16 20:03:57 +02:00
Norman Maurer
f88dfd0430 [#2653] Remove unnecessary ensureAccessible() calls
Motivation:

I introduced ensureAccessible() class as part of 6c47cc9711 in some places. Unfortunally I also added some where these are not needed and so caused a performance regression.

Modification:

Remove calls where not needed.

Result:

Fixed performance regression.
2014-07-14 21:04:12 +02:00
Norman Maurer
93c306602a [#2653] Remove uncessary range checks for performance reasons
Motivation:

I introduced range checks as part of 6c47cc9711 in some places. Unfortunally I also added some where these are not needed and so caused a performance regression.

Modification:

Remove range checks where not needed

Result:

Fixed performance regression.
2014-07-14 11:43:19 +02:00
Norman Maurer
c75783abb5 Fix over-sensible testcase 2014-07-13 17:19:34 +02:00
Norman Maurer
8b0bc7f33d [#2651] Fix possible infinite-loop when cancel tasks
Motivations:
In our new version of HWT we used some kind of lazy cancelation of timeouts by put them back in the queue and let them pick up on the next tick. This  multiple problems:
 - we may corrupt the MpscLinkedQueue if the task is used as tombstone
 - this sometimes lead to an uncessary delay especially when someone did executed some "heavy" logic in the TimeTask

Modifications:
Use a Lock per HashedWheelBucket for save and fast removal.

Modifications:
Cancellation of tasks can be done fast and so stuff can be GC'ed and no more infinite-loop possible
2014-07-11 15:41:33 +02:00
Norman Maurer
8edfde3f26 [#2650] Allow to disable http header validation in SpdyHttpDecoder and SpdyHttpCodec
Motivation:

HTTP header validation can be expensive so we should allow to disable it like we do in HttpObjectDecoder.

Modification:

Add constructor argument to disable validation.

Result:
Performance improvement
2014-07-11 08:33:37 +02:00
Norman Maurer
909d1f2c0f Reuse previous created HttpHeaders by HttpObjectAggregator
Motivation:

HttpObjectAggregator currently creates a new FullHttpResponse / FullHttpRequest for each message it needs to aggregate. While doing so it also creates 2 DefaultHttpHeader instances (one for the headers and one for the trailing headers). This is bad for two reasons:
  - More objects are created then needed and also populate the headers is not for free
  - Headers may get validated even if the validation was disabled in the decoder

Modification:

- Wrap the previous created HttpResponse / HttpRequest and so reuse the original HttpHeaders
- Reuse the previous created trailing HttpHeader.
- Fix a bug where the trailing HttpHeader was incorrectly mixed in the headers.

Result:

- Less GC
- Faster HttpObjectAggregator implementation
2014-07-11 07:08:12 +02:00
Norman Maurer
81126ab429 Fix checkstyle error introduced by 52cb55d388 2014-07-10 07:12:40 +02:00
Norman Maurer
17280116c4 [#2643] Throw TooLongFrameException instead of using fireExceptionCaught
Motivation:

It's not always the case that there is another handler in the pipeline that will intercept the exceptionCaught event because sometimes users just sub-class. In this case the exception will just hit the end of the pipeline.

Modification:
Throw the TooLongFrameException so that sub-classes can handle it in the exceptionCaught(...) method directly.

Result:
Sub-classes can correctly handle the exception,
2014-07-10 06:56:28 +02:00
Idel Pivnitskiy
deda8f15a2 Moved bit-level read operations from Bzip2Decoder to the new Bzip2BitReader
Motivation:

Collect all bit-level read operations in one class is better. And now it's easy to use not only in Bzip2Decoder. For example, in Bzip2HuffmanStageDecoder.

Modifications:

Created a new class - Bzip2BitReader which provides bit-level reads.
Removed bit-level read operations from Bzip2Decoder.
Improved javadoc.

Result:

Bzip2BitReader allows the reading of single bit booleans, bit strings of arbitrary length (up to 24 bits), and bit aligned 32-bit integers.
2014-07-09 13:50:30 +02:00
Norman Maurer
62bbd4220a Fix JVM segfault during JNI call. Part of [#2647]
Motivation:
Currently when Native.writev(...) is used it is possible to see a JVM segfault because the offset is updated to early.

Modification:
Only update the offset once it is safe to do so.

Result:
No more segfault
2014-07-09 13:34:15 +02:00
Norman Maurer
fb22d34925 [#2647] Respect IOV_MAX when call writev in native transport
Motivation:

epoll transport fails on gathering write of more then 1024 buffers. As linux supports max. 1024 iov entries when calling writev(...) the epoll transport throws an exception.

Thanks again to @blucas to provide me with a reproducer and so helped me to understand what the issue is.

Modifications:

Make sure we break down the writes if to many buffers are uses for gathering writes.

Result:

Gathering writes work with any number of buffers
2014-07-09 12:21:58 +02:00
Brendt Lucas
ac8ac59148 [#2642] CompositeByteBuf.deallocate memory/GC improvement
Motivation:

CompositeByteBuf.deallocate generates unnecessary GC pressure when using the 'foreach' loop, as a 'foreach' loop creates an iterator when looping.

Modification:

Convert 'foreach' loop into regular 'for' loop.

Result:

Less GC pressure (and possibly more throughput) as the 'for' loop does not create an iterator
2014-07-08 21:08:14 +02:00
Norman Maurer
33a810a513 Move generic code to HttpOrSpdyChooser to simplify implementations
Motivation:

HttpOrSpdyChooser can be simplified so the user not need to implement getProtocol(...) method.

Modification:

Add implementation for the method. The user can override it if necessary.

Result:

Easier usage of HttpOrSpdyChooser.
2014-07-07 09:37:10 +02:00
Trustin Lee
0fe2824729 Add missing m2eclipse life cycle mapping
Also, use ignore instead of execution because those plugins are not
really useful when building from Eclipse anyway.
2014-07-06 16:53:29 +09:00
Trustin Lee
e167b02d52 [maven-release-plugin] prepare for next development iteration 2014-07-04 17:26:02 +09:00
Trustin Lee
ba50cb829b [maven-release-plugin] prepare release netty-4.1.0.Beta1 2014-07-04 17:25:54 +09:00
Trustin Lee
787663a644 [maven-release-plugin] rollback the release of netty-4.1.0.Beta1 2014-07-04 17:11:14 +09:00
Trustin Lee
83eae705e1 [maven-release-plugin] prepare release netty-4.1.0.Beta1 2014-07-04 17:02:17 +09:00
Trustin Lee
97825598d2 Fix another buffer leaks in JsonObjectDecoderTest 2014-07-04 16:12:06 +09:00
Trustin Lee
5b87cdc8bd Reduce the perceived time taken to retrieve initialSeedUniquifier
Motivation:

When system is in short of entrophy, the initialization of
ThreadLocalRandom can take at most 3 seconds.  The initialization occurs
when ThreadLocalRandom.current() is invoked first time, which might be
much later than the moment when the application has started.  If we
start the initialization of ThreadLocalRandom as early as possible, we
can reduce the perceived time taken for the retrieval.

Modification:

Begin the initialization of ThreadLocalRandom in InternalLoggerFactory,
potentially one of the firstly initialized class in a Netty application.

Make DefaultChannelId retrieve the current process ID before retrieving
the current machine ID, because retrieval of a machine ID is more likely
to use ThreadLocalRandom.current().

Use a dummy channel ID for EmbeddedChannel, which prevents many unit
tests from creating a ThreadLocalRandom instance.

Result:

We gain extra 100ms at minimum for initialSeedUniquifier generation.  If
an application has its own initialization that takes long enough time
and generates good amount of entrophy, it is very likely that we will
gain a lot more.
2014-07-04 16:04:48 +09:00
Trustin Lee
11fdec3c4a Log the time taken for generating the initialSeedUniquifier
- Sometimes useful to know it how long it takes from the log, to make
  sure it's not something else that is blocking.
2014-07-04 13:26:58 +09:00
Trustin Lee
cde7157c39 Make JsonObjectDecoder discard everything after stream corruption
Motivation:

There's no way to recover from a corrupted JSON stream. The current
implementation will raise an infinite exception storm when a peer sends
a large corrupted stream.

Modification:

Discard everything once stream corruption is detected.

Result:

Fixes a buffer leak
Fixes exception storm
2014-07-04 11:16:00 +09:00
Norman Maurer
f8c95c3827 [#2586] Use correct EventLoop to notify delayed bind failures
Motivation:

When a bind fails AbstractBootstrap will use the GlobalEventExecutor to notify the ChannelPromise. We should use the EventLoop of the Channel if possible.

Modification:

Use EventLoop of the Channel if possible to use the correct Thread to notify and so guaranteer the right order of events.

Result:

Use the correct EventLoop for notification
2014-07-03 21:31:05 +02:00
Norman Maurer
217fb0de05 [#2618] Introduce ChannelPromise.unvoid() and ChannelFuture.isVoid()
Motivation:

There is no way for a ChannelHandler to check if the passed in ChannelPromise for a write(...) call is a VoidChannelPromise. This is a problem as some handlers need to add listeners to the ChannelPromise which is not possible in the case of a VoidChannelPromise.

Modification:

- Introduce ChannelFuture.isVoid() which will return true if it is not possible to add listeners or wait on the result.
- Add ChannelPromise.unvoid() which allows to create a ChannelFuture out of a void ChannelFuture which supports all the operations.

Result:

It's now easy to write ChannelHandler implementations which also works when a void ChannelPromise is used.
2014-07-03 14:17:12 +02:00
Trustin Lee
a1974ef35b Fix an inspector warning in JsonObjectDecoder 2014-07-03 20:01:48 +09:00
Trustin Lee
d0b355b26e Fix a buffer leak in JsonObjectDecoderTest 2014-07-03 19:58:06 +09:00
Jakob Buchgraber
aed13ba5ef Split a JSON byte stream into JSON objects/arrays. Fixes #2536
Motivation:

See GitHub Issue #2536.

Modifications:

Introduce the class JsonObjectDecoder to split a JSON byte stream
into individual JSON objets/arrays.

Result:

A Netty application can now handle a byte stream where multiple JSON
documents follow eachother as opposed to only a single JSON document
per request.
2014-07-03 18:34:29 +09:00
Trustin Lee
10623ae133 Let OkResponseHandler extend SimpleChannelInboundHandler
Motivation:

OkResponseHandler is the last handler in the pipeline of the HTTP CORS
example.  It is responsible for releasing all messages it handled.

Modification:

Extend SimpleChannelInboundHandler instead of
ChannelInboundHandlerAdapter

Result:

Fixed a leak
2014-07-03 18:14:40 +09:00
Trustin Lee
fbf1bdbef1 Fix the build timeout when 'leak' profile is active
Motivation:

AbstractByteBufTest.testInternalBuffer() uses writeByte() operations to
populate the sample data.  Usually, this isn't a problem, but it starts
to take a lot of time when the resource leak detection level gets
higher.

In our CI machine, testInternalBuffer() takes more than 30 minutes,
causing the build timeout when the 'leak' profile is active (paranoid
level resource detection.)

Modification:

Populate the sample data using ThreadLocalRandom.nextBytes() instead of
using millions of writeByte() operations.

Result:

Test runs much faster when leak detection level is high.
2014-07-03 17:55:10 +09:00
Trustin Lee
3c6bc0b4cb Fix unclean backport in InternalLoggerFactory
.. which leaked in from d0912f2709
2014-07-02 20:27:06 +09:00
Trustin Lee
d0912f2709 Fix most inspector warnings
Motivation:

It's good to minimize potentially broken windows.

Modifications:

Fix most inspector warnings from our profile
Update IntObjectHashMap

Result:

Cleaner code
2014-07-02 19:55:07 +09:00
Norman Maurer
7a05a617f6 [#2558] Define SO_REUSEPORT if not defined
Motivation:

Currently it is impossible to build netty on linux system that not define SO_REUSEPORT even if it is supported.

Modification:

Define SO_REUSEPORT if not defined.

Result:

Possible to build on more linux dists.
2014-07-02 09:40:10 +02:00
Norman Maurer
36b80c25f7 Correctly return from selector loop one a scheduled task is ready for processing
Motivation:

We use the nanoTime of the scheduledTasks to calculate the milli-seconds to wait for a select operation to select something. Once these elapsed we check if there was something selected or some task is ready for processing. Unfortunally we not take into account scheduled tasks here so the selection loop will continue if only scheduled tasks are ready for processing. This will delay the execution of these tasks.

Modification:

- Check if a scheduled task is ready after selecting
- also make a tiny change in NioEventLoop to not trigger a rebuild if nothing was selected because the timeout was reached a few times in a row.

Result:

Execute scheduled tasks on time.
2014-07-02 09:10:37 +02:00
Norman Maurer
23840b76d7 [#2623] Release local references to guard against StackOverflow in JNI
Motivation:

When we do a (env*)->GetObjectArrayElement(...) call we may created many local references which will only be cleaned up once we exist the native method. Thus a lot of memory can be used and so a StackOverFlow may be triggered. Beside this the JNI specification only say that an implementation must cope with 16 local references.

Modification:

Call (env*)->ReleaseLocalRef(...) to release the resource once not needed anymore.

Result:

Less memory usage and guard against StackOverflow
2014-06-30 09:57:13 +02:00
Norman Maurer
9594a81b95 [#2622] Correctly check reference count before try to work on the underlying memory
Motivation:

Because of how we use reference counting we need to check for the reference count before each operation that touches the underlying memory. This is especially true as we use sun.misc.Cleaner.clean() to release the memory ASAP when possible. Because of this the user may cause a SEGFAULT if an operation is called that tries to access the backing memory after it was released.

Modification:

Correctly check the reference count on all methods that access the underlying memory or expose it via a ByteBuffer.

Result:

Safer usage of ByteBuf
2014-06-30 07:14:25 +02:00
Jay
461d3c876f Maintain decoder result in HttpObjectAggregator
Motivation:
DecodeResult is dropped when aggregate HTTP messages.

Modification:

Make sure we not drop the DecodeResult while aggregate HTTP messages.

Result:

Correctly include the DecodeResult for later processing.
2014-06-28 22:08:53 +02:00
xmxsuperstar
79195da0d7 fix example missing break statement in ReplayingDecoder 2014-06-28 21:42:23 +02:00