Commit Graph

414 Commits

Author SHA1 Message Date
Idel Pivnitskiy
c8841bc9de Implemented LZ4 compression codec
Motivation:

LZ4 compression codec provides sending and receiving data encoded by very fast LZ4 algorithm.

Modifications:

- Added `lz4` library which implements LZ4 algorithm.
- Implemented Lz4FramedEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Added tests to verify the Lz4FramedEncoder and how it can compress data for the next uncompression using the original library.
- Implemented Lz4FramedDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages.
- Added tests to verify the Lz4FramedDecoder and how it can uncompress data after compression using the original library.
- Added integration tests for Lz4FramedEncoder/Decoder.

Result:

Full LZ4 compression codec which can compress/uncompress data using LZ4 algorithm.
2014-08-14 15:05:24 -07:00
Trustin Lee
1aa20e8609 Fix class description of FastLzFrameDecoder 2014-08-13 22:58:45 -07:00
Trustin Lee
f311012455 Rename FastLzFramed* to FastLzFrame* 2014-08-13 22:55:34 -07:00
Norman Maurer
d315bbaa14 Fix and clearify javadocs
Motivation:

ByteToMessageDecoder and ReplayingDecoder have incorrect javadocs in some places.

Modifications:

Fix incorrect javadocs for both classes.

Result:

Correct javadocs for both classes
2014-08-14 06:46:29 +02:00
Idel Pivnitskiy
a0c466a276 Implemented FastLZ compression codec
Motivation:

FastLZ compression codec provides sending and receiving data encoded by fast FastLZ algorithm using block mode.

Modifications:

- Added part of `jfastlz` library which implements FastLZ algorithm. See FastLz class.
- Implemented FastLzFramedEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Implemented FastLzFramedDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages.
- Added integration tests for `FastLzFramedEncoder/Decoder`.

Result:

Full FastLZ compression codec which can compress/uncompress data using FastLZ algorithm.
2014-08-12 15:14:59 -07:00
Trustin Lee
60764200d7 Do not throw an exception when failed to get a header
Motivation:

It is often very expensive to instantiate an exception. TextHeader
should not raise an exception when it failed to find a header or when
its header value is not valid.

Modification:

- Change the return type of the getter methods to Integer and Long so
  that null is returned when no header is found or its value is invalid
- Update Javadoc

Result:

- Fixes #2758
- No unnecessary instantiation of exceptions
2014-08-12 11:14:06 -07:00
Trustin Lee
de724063f3 Reduce the initial capacity of the value list from 4 to 2
Motivation:

DefaultTextHeaders.getAll*() methods create an ArrayList whose initial
capacity is 4.  However, it is more likely that the actual number of
values is smaller than that.

Modifications:

Reduce the initial capacity of the value list from 4 to 2

Result:

Slightly reduced memory footprint
2014-08-12 10:36:13 -07:00
jxu
2d36caa9f6 Add TextHeaders.getAndRemove(...) and its variants
Related issue: #2649 and #2745

Motivation:

At the moment there is no way to get and remove a header with one call.
This means you need to search the headers two times. We should add
getAndRemove(...) to allow doing so with one call.

Modifications:

Add getAndRemove(...) and getUnconvertedAndRemove(...) and their
variants

Result:

More efficient API
2014-08-12 10:33:04 -07:00
Idel Pivnitskiy
073ec8d10a Consider writerIndex when LzfDecoder writes into a new heap buffer
Motivation:

Now LzfDecoder do not consider writerIndex when it writes into array of a new heap buffer (when it decodes a compressed chuck of data)
2014-08-05 22:51:02 +02:00
Idel Pivnitskiy
c13419750d Improve Bzip2BitReader/Writer
Motivation:

Before this changes Bzip2BitReader and Bzip2BitWriter accessed to ByteBuf byte by byte. So tests for Bzip2 compression codec takes a lot of time if we ran them with paranoid level of resource leak detection. For more information see comments to #2681 and #2689.

Modifications:

- Increased size of bit buffers from 8 to 64 bits.
- Improved reading and writing operations.
- Save link to incoming ByteBuf inside Bzip2BitReader.
- Added methods to check possible readable bits and bytes in Bzip2BitReader.
- Updated Bzip2 classes to use new API of Bzip2BitReader.
- Added new constants to Bzip2Constants.

Result:

Increased size of bit buffers and improved performance of Bzip2 compression codec (for general work by 13% and for tests with paranoid level of resource leak detection by 55%).
2014-08-04 07:52:40 +02:00
Norman Maurer
e1cc1fbabc [#2705] Call fireChannelReadComplete() if channelActive(...) decodes messages in ReplayingDecoder / ByteToMessageDecoder
Motivation:

In ReplayingDecoder / ByteToMessageDecoder channelInactive(...) method we try to decode a last time and fire all decoded messages throw the pipeline before call ctx.fireChannelInactive(...). To keep the correct order of events we also need to call ctx.fireChannelReadComplete() if we read anything.

Modifications:

- Channel channelInactive(...) to call ctx.fireChannelReadComplete() if something was decoded
- Move out.recycle() to finally block

Result:

Correct order of events.
2014-07-24 14:38:46 +02:00
Idel Pivnitskiy
4816533638 Refactor Bzip2 tests
Motivation:

Complicated code of Bzip2 tests with some unnecessary actions.

Modifications:

- Reduce size of BYTES_LARGE array of random test data for Bzip2  tests.
- Removed unnecessary creations of EmbeddedChannel instances in Bzip2 tests.
- Simplified tests in Bzip2DecoderTest which expect exception.
- Removed unnecessary testStreamInitialization() from Bzip2EncoderTest.

Result:

Reduced time to test the 'codec' package by 7 percent, simplified code of Bzip2 tests.
2014-07-23 19:46:00 +02:00
Idel Pivnitskiy
99cf6f0732 Refactor integration tests of compression codecs
Motivation:

Duplicated code of integration tests for different compression codecs.

Modifications:

- Added abstract class IntegrationTest which contains common tests for any compression codec.
- Removed common tests from Bzip2IntegrationTest and LzfIntegrationTest.
- Implemented abstract methods of IntegrationTest in Bzip2IntegrationTest, LzfIntegrationTest and SnappyIntegrationTest.

Result:

Removed duplicated code of integration tests for compression codecs and simplified an addition of integration tests for new compression codecs.
2014-07-23 19:44:10 +02:00
Idel Pivnitskiy
ca87cc887e Simplify Bzip2 tests
Motivation:

Sometimes we have a 'build time out' error because tests for bzip2 codec take a long time.

Modifications:

Removed cycles from Bzip2EncoderTest.testCompression(byte[]) and Bzip2DecoderTest.testDecompression(byte[]).

Result:

Reduced time to test the 'codec' package by 30 percent.
2014-07-22 18:00:26 +02:00
Idel Pivnitskiy
dc9d933d74 Fixes for compression codecs
Motivation:

Fixed founded mistakes in compression codecs.

Modifications:

- Changed return type of ZlibUtil.inflaterException() from CompressionException to DecompressionException
- Updated @throws in javadoc of JZlibDecoder to throw DecompressionException instead of CompressionException
- Fixed JdkZlibDecoder to throw DecompressionException instead of CompressionException
- Removed unnecessary empty lines in JdkZlibEncoder and JZlibEncoder
- Removed public modifier from Snappy class
- Added MAX_UNCOMPRESSED_DATA_SIZE constant in SnappyFramedDecoder
- Used in.readableBytes() instead of (in.writerIndex() - in.readerIndex()) in SnappyFramedDecoder
- Added private modifier for enum ChunkType in SnappyFramedDecoder
- Fixed potential bug (sum overflow) in Bzip2HuffmanAllocator.first(). For more info, see http://googleresearch.blogspot.ru/2006/06/extra-extra-read-all-about-it-nearly.html

Result:

Fixed sum overflow in Bzip2HuffmanAllocator, improved exceptions in ZlibDecoder implementations, hid Snappy class
2014-07-20 09:32:53 +02:00
Idel Pivnitskiy
0cc3eccc2b Close ObjectInputStream in ObjectDecoder.decode(...)
Motivation:

We create a new CompactObjectInputStream with ByteBufInputStream in ObjectDecoder.decode(...) method and don't close this InputStreams before return statement.

Modifications:

Save link to the ObjectInputStream and close it before return statement.

Result:

Close InputStreams and clean up unused resources. It will be better for GC.
2014-07-20 09:23:35 +02:00
Norman Maurer
53141b04a8 Fix buffer leak in Bzip2EncoderTest 2014-07-19 14:41:18 +02:00
Idel Pivnitskiy
ed7240b597 Implemented a Bzip2Encoder
Motivation:

Bzip2Encoder provides sending data compressed in bzip2 format.

Modifications:

Added classes:
- Bzip2Encoder
- Bzip2BitWriter
- Bzip2BlockCompressor
- Bzip2DivSufSort
- Bzip2HuffmanAllocator
- Bzip2HuffmanStageEncoder
- Bzip2MTFAndRLE2StageEncoder
- Bzip2EncoderTest

Modified classes:
- Bzip2Constants (splited BLOCK_HEADER_MAGIC and END_OF_STREAM_MAGIC)
- Bzip2Decoder (use splited magic numbers)

Added integration tests for Bzip2Encoder/Decoder

Result:

Implemented new encoder which can compress outgoing data in bzip2 format.
2014-07-17 16:19:39 +02:00
Idel Pivnitskiy
3c6017a9b1 Implemented LZF compression codec
Motivation:

LZF compression codec provides sending and receiving data encoded by very fast LZF algorithm.

Modifications:

- Added Compress-LZF library which implements LZF algorithm
- Implemented LzfEncoder which extends MessageToByteEncoder and provides compression of outgoing messages
- Added tests to verify the LzfEncoder and how it can compress data for the next uncompression using the original library
- Implemented LzfDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages
- Added tests to verify the LzfDecoder and how it can uncompress data after compression using the original library
- Added integration tests for LzfEncoder/Decoder

Result:

Full LZF compression codec which can compress/uncompress data using LZF algorithm.
2014-07-17 07:18:07 +02:00
Norman Maurer
81126ab429 Fix checkstyle error introduced by 52cb55d388 2014-07-10 07:12:40 +02:00
Norman Maurer
17280116c4 [#2643] Throw TooLongFrameException instead of using fireExceptionCaught
Motivation:

It's not always the case that there is another handler in the pipeline that will intercept the exceptionCaught event because sometimes users just sub-class. In this case the exception will just hit the end of the pipeline.

Modification:
Throw the TooLongFrameException so that sub-classes can handle it in the exceptionCaught(...) method directly.

Result:
Sub-classes can correctly handle the exception,
2014-07-10 06:56:28 +02:00
Idel Pivnitskiy
deda8f15a2 Moved bit-level read operations from Bzip2Decoder to the new Bzip2BitReader
Motivation:

Collect all bit-level read operations in one class is better. And now it's easy to use not only in Bzip2Decoder. For example, in Bzip2HuffmanStageDecoder.

Modifications:

Created a new class - Bzip2BitReader which provides bit-level reads.
Removed bit-level read operations from Bzip2Decoder.
Improved javadoc.

Result:

Bzip2BitReader allows the reading of single bit booleans, bit strings of arbitrary length (up to 24 bits), and bit aligned 32-bit integers.
2014-07-09 13:50:30 +02:00
Trustin Lee
97825598d2 Fix another buffer leaks in JsonObjectDecoderTest 2014-07-04 16:12:06 +09:00
Trustin Lee
cde7157c39 Make JsonObjectDecoder discard everything after stream corruption
Motivation:

There's no way to recover from a corrupted JSON stream. The current
implementation will raise an infinite exception storm when a peer sends
a large corrupted stream.

Modification:

Discard everything once stream corruption is detected.

Result:

Fixes a buffer leak
Fixes exception storm
2014-07-04 11:16:00 +09:00
Trustin Lee
a1974ef35b Fix an inspector warning in JsonObjectDecoder 2014-07-03 20:01:48 +09:00
Trustin Lee
d0b355b26e Fix a buffer leak in JsonObjectDecoderTest 2014-07-03 19:58:06 +09:00
Jakob Buchgraber
aed13ba5ef Split a JSON byte stream into JSON objects/arrays. Fixes #2536
Motivation:

See GitHub Issue #2536.

Modifications:

Introduce the class JsonObjectDecoder to split a JSON byte stream
into individual JSON objets/arrays.

Result:

A Netty application can now handle a byte stream where multiple JSON
documents follow eachother as opposed to only a single JSON document
per request.
2014-07-03 18:34:29 +09:00
Trustin Lee
d0912f2709 Fix most inspector warnings
Motivation:

It's good to minimize potentially broken windows.

Modifications:

Fix most inspector warnings from our profile
Update IntObjectHashMap

Result:

Cleaner code
2014-07-02 19:55:07 +09:00
xmxsuperstar
79195da0d7 fix example missing break statement in ReplayingDecoder 2014-06-28 21:42:23 +02:00
Zhihui Jiao
c4441642bb Fix inconsistent code in the doc 2014-06-27 06:48:40 +02:00
Trustin Lee
d2e385acc2 Fix incorrect bytesBefore/indexOf() in ReplayingDecoderBuffer
Motivation:

bytesBefore(length, ...), bytesBefore(index, length, ...), and
indexOf(fromIndex, toIndex,...) in ReplayingDecoderBuffer are buggy.
They trigger 'REPLAY even when they don't need to.

Modification:

Implement the buggy methods properly so that REPLAYs are not triggered
unnecessarily.

Result:

Correct behvaior
2014-06-26 18:56:33 +09:00
Norman Maurer
4d2b78ca3c Reduce the memory copies in JdkZlibEncoder
Motivation:

At the moment we use a lot of unnecessary memory copies in JdkZlibEncoder. This is caused by either allocate a to small ByteBuf and expand it later or using a temporary byte array.
Beside this the memory footprint of JdkZlibEncoder is pretty high because of the byte[] used for compressing.

Modification:

- Override allocateBuffer(...) and calculate the estimatedsize in there, this reduce expanding of the ByteBuf later
- Not use byte[] in the instance itself but allocate a heap ByteBuf and write directly into the byte array

Result:

Less memory copies and smaller memory footprint
2014-06-26 11:12:19 +02:00
Trustin Lee
937f790f70 Checkstyle 2014-06-26 17:48:32 +09:00
Trustin Lee
5f889d92a1 Fix buffer leaks in Bzip2Decoder(Test)
If decompression fails, the buffer that contains the decompressed data
is not released.  Bzip2DecoderTest.testStreamCrcError() also does not
release the partial output Bzip2Decoder produces.
2014-06-26 17:48:32 +09:00
Norman Maurer
56d732d439 Fix buffer leaks in Bzip2DecoderTest 2014-06-26 09:21:44 +02:00
Trustin Lee
bf85af5743 Fix buffer leaks in Bzip2DecoderTest 2014-06-24 16:47:14 +09:00
Idel Pivnitskiy
f9021a6061 Implement a Bzip2Decoder
Motivation:

Bzip2Decoder provides receiving data compressed in bzip2 format.

Modifications:

Added classes:
- Bzip2Decoder
- Bzip2Constants
- Bzip2BlockDecompressor
- Bzip2HuffmanStageDecoder
- Bzip2MoveToFrontTable
- Bzip2Rand
- Crc32
- Bzip2DecoderTest

Result:

Implemented and tested new decoder which can uncompress incoming data in bzip2 format.
2014-06-24 14:50:09 +09:00
Norman Maurer
12a3e23e47 MessageToByteEncoder always starts with ByteBuf that use initalCapacity == 0
Motivation:

MessageToByteEncoder always starts with ByteBuf that use initalCapacity == 0 when preferDirect is used. This is really wasteful in terms of performance as every first write into the buffer will cause an expand of the buffer itself.

Modifications:

 - Change ByteBufAllocator.ioBuffer() use the same default initialCapacity as heapBuffer() and directBuffer()
 - Add new allocateBuffer method to MessageToByteEncoder that allow the user to do some smarter allocation based on the message that will be encoded.

Result:

Less expanding of buffer and more flexibilty when allocate the buffer for encoding.
2014-06-24 13:55:21 +09:00
Trustin Lee
8c25830b0b Move haproxy codec to a separate module 2014-06-21 15:59:21 +09:00
Jon Keys
d7b2affe32 Add HAProxy protocol decoder
Motivation:

The proxy protocol provides client connection information for proxied
network services. Several implementations exist (e.g. Haproxy, Stunnel,
Stud, Postfix), but the primary motivation for this implementation is to
support the proxy protocol feature of Amazon Web Services Elastic Load
Balancing.

Modifications:

This commit adds a proxy protocol decoder for proxy protocol version 1
as specified at:

  http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt

The foundation for version 2 support is also in this commit but it is
explicitly NOT supported due to a lack of external implementations to
test against.

Result:

The proxy protocol decoder can be used to send client connection
information to inbound handlers in a channel pipeline from services
which support the proxy protocol.
2014-06-21 15:59:21 +09:00
Trustin Lee
085a61a310 Refactor FastThreadLocal to simplify TLV management
Motivation:

When Netty runs in a managed environment such as web application server,
Netty needs to provide an explicit way to remove the thread-local
variables it created to prevent class loader leaks.

FastThreadLocal uses different execution paths for storing a
thread-local variable depending on the type of the current thread.
It increases the complexity of thread-local removal.

Modifications:

- Moved FastThreadLocal and FastThreadLocalThread out of the internal
  package so that a user can use it.
- FastThreadLocal now keeps track of all thread local variables it has
  initialized, and calling FastThreadLocal.removeAll() will remove all
  thread-local variables of the caller thread.
- Added FastThreadLocal.size() for diagnostics and tests
- Introduce InternalThreadLocalMap which is a mixture of hard-wired
  thread local variable fields and extensible indexed variables
- FastThreadLocal now uses InternalThreadLocalMap to implement a
  thread-local variable.
- Added ThreadDeathWatcher.unwatch() so that PooledByteBufAllocator
  tells it to stop watching when its thread-local cache has been freed
  by FastThreadLocal.removeAll().
- Added FastThreadLocalTest to ensure that removeAll() works
- Added microbenchmark for FastThreadLocal and JDK ThreadLocal
- Upgraded to JMH 0.9

Result:

- A user can remove all thread-local variables Netty created, as long as
  he or she did not exit from the current thread. (Note that there's no
  way to remove a thread-local variable from outside of the thread.)
- FastThreadLocal exposes more useful operations such as isSet() because
  we always implement a thread local variable via InternalThreadLocalMap
  instead of falling back to JDK ThreadLocal.
- FastThreadLocalBenchmark shows that this change improves the
  performance of FastThreadLocal even more.
2014-06-19 21:13:55 +09:00
Norman Maurer
984b0aa961 [#2572] Correctly calculate length of output buffer before inflate to fix IndexOutOfBoundException
Motivation:

JdkZlibDecoder fails to decode because the length of the output buffer is not calculated correctly.
This can cause an IndexOutOfBoundsException or data-corruption when the PooledByteBuffAllocator is used.

Modifications:

Correctly calculate the length

Result:

No more IndexOutOfBoundsException or data-corruption.
2014-06-16 10:17:02 +02:00
Trustin Lee
776ac4ba19 Use FastThreadLocal in more places 2014-06-14 17:46:10 +09:00
Trustin Lee
c076c33901 Backport the additional AsciiString/TextHeader changes from master
- Add useful static methods to AsciiString
- Add more getters in TextHeaders
- Remove unnecessary utility methods in SpdyHttpHeaders
2014-06-14 17:33:34 +09:00
Trustin Lee
681d460938 Introduce TextHeaders and AsciiString
Motivation:

We have quite a bit of code duplication between HTTP/1, HTTP/2, SPDY,
and STOMP codec, because they all have a notion of 'headers', which is a
multimap of string names and values.

Modifications:

- Add TextHeaders and its default implementation
- Add AsciiString to replace HttpHeaderEntity
  - Borrowed some portion from Apache Harmony's java.lang.String.
- Reimplement HttpHeaders, SpdyHeaders, and StompHeaders using
  TextHeaders
- Add AsciiHeadersEncoder to reuse the encoding a TextHeaders
  - Used a dedicated encoder for HTTP headers for better performance
    though
- Remove shortcut methods in SpdyHeaders
- Replace SpdyHeaders.getStatus() with HttpResponseStatus.parseLine()

Result:

- Removed quite a bit of code duplication in the header implementations.
- Slightly better performance thanks to improved header validation and
  hash code calculation
2014-06-14 15:36:19 +09:00
belliottsmith
2a2a21ec59 Introduce FastThreadLocal which uses an EnumMap and a predefined fixed set of possible thread locals
Motivation:
Provide a faster ThreadLocal implementation

Modification:
Add a "FastThreadLocal" which uses an EnumMap and a predefined fixed set of possible thread locals (all of the static instances created by netty) that is around 10-20% faster than standard ThreadLocal in my benchmarks (and can be seen having an effect in the direct PooledByteBufAllocator benchmark that uses the DEFAULT ByteBufAllocator which uses this FastThreadLocal, as opposed to normal instantiations that do not, and in the new RecyclableArrayList benchmark);

Result:
Improved performance
2014-06-13 10:56:18 +02:00
Trustin Lee
8b0a0f9a8f Introduce MessageAggregator and DecoderResultProvider
Motivation:

We have different message aggregator implementations for different
protocols, but they are very similar with each other.  They all stems
from HttpObjectAggregator.  If we provide an abstract class that provide
generic message aggregation functionality, we will remove their code
duplication.

Modifications:

- Add MessageAggregator which provides generic message aggregation
- Reimplement all existing aggregators using MessageAggregator
- Add DecoderResultProvider interface and extend it wherever possible so
  that MessageAggregator respects the state of the decoded message

Result:

Less code duplication
2014-06-05 16:51:14 +09:00
Norman Maurer
b0ddfb9b65 [#2525] Use VoidChannelPromise in MessageToMessageEncoder when possible
Motivation:
At the moment MessageToMessageEncoder uses ctx.write(msg) when have more then one message was produced. This may produce more GC pressure then necessary as when the original ChannelPromise is a VoidChannelPromise we can safely also use one when write messages.

Modifications:
Use VoidChannelPromise when the original ChannelPromise was of this type

Result:
Less object creation and GC pressure
2014-06-01 19:26:20 +02:00
Norman Maurer
880acbca72 Use ByteBuf.readSlice(...).retain() to minimize memory copies.
Motivation:
At the moment we call ByteBuf.readBytes(...) in these handlers but with optimizations done as part of 25e0d9d we can just use readSlice(...).retain() and eliminate the memory copy.

Modifications:
Replace ByteBuf.readBytes(...) usage with readSlice(...).retain().

Result:
Less memory copies.
2014-05-10 17:20:53 +02:00
Trustin Lee
db3709e652 Synchronized between 4.1 and master
Motivation:

4 and 5 were diverged long time ago and we recently reverted some of the
early commits in master.  We must make sure 4.1 and master are not very
different now.

Modification:

Fix found differences

Result:

4.1 and master got closer.
2014-04-25 00:38:02 +09:00