Commit Graph

246 Commits

Author SHA1 Message Date
Trustin Lee
3f7b674db8 Fix bugs in ZLIB codec where they produce malformed stream or their streams are not flushed on time
- Fixes #2014
- Add the tests that mix JDK ZLIB codec and JZlib codecs
- Fix a bug where JdkZlibEncoder does not encode the GZIP header when nothing was written to te channel
- Fix a bug where the encoders do not consider the overhead of the wrapper format when calculating the estimated compressed output size.
- Fix a bug where the decoders do not discard the received data after the compressed stream is finished
2013-11-29 18:09:04 +09:00
Luke Wood
31c1513229 [#1958] Use CRC32-C in Snappy as per the spec 2013-11-01 20:54:55 +01:00
Norman Maurer
d946659520 [#1906] Use a ByteBuf allocator from the ByteBufAllocator when encode Strings 2013-10-09 21:18:08 +02:00
Norman Maurer
00f99dbff6 [#1893] Fix bug in JdkZlibDecoder which did not let it handle large data 2013-10-07 07:31:21 +02:00
Norman Maurer
aedd631dbb [#1798] Correctly handle skipping of too big frames 2013-08-29 13:51:50 +02:00
Norman Maurer
51a536fd30 Free up all buffers after the tests 2013-08-28 11:00:55 +02:00
Norman Maurer
7a38163743 [#1788] Correctly decode CRC32 and ISIZE when using JdkZlibDecoder
Because of incorrect decoding a CompressionException was thrown before
2013-08-28 11:00:47 +02:00
Norman Maurer
39cabcd36c [#1481] Add a JdkZlibDecoder which has no dependencies 2013-08-14 07:41:28 +02:00
Norman Maurer
4d064d85df Rename class 2013-08-02 07:24:49 +02:00
Norman Maurer
ebee56ef49 More tests related to self removal from pipeline in ByteToMessageDecoder and ReplayingDecoder 2013-08-02 07:23:46 +02:00
Norman Maurer
487c28d93e Add testcases to prove that remove ReplayingDecoder/ByteToMessageDecoder from within the decode(...) method works 2013-08-01 20:42:46 +02:00
Norman Maurer
6278d88d2f [#1656] Fix IndexOutOfBoundsException in MessageToMessageException 2013-07-26 11:46:26 +02:00
Trustin Lee
7215c011ca Fix resource leaks in the tests 2013-07-18 23:42:51 +09:00
Trustin Lee
fe59ab1b9c Fix resource leak in LengthFieldBasedFrameDecoderTest 2013-07-18 20:55:02 +09:00
Norman Maurer
2f7759a264 Remove unused package private classes 2013-07-14 23:50:27 +02:00
Trustin Lee
26e9d70457 Remove channelReadSuspended / Rename messageReceived(Last) to channelRead(Complete)
- Remove channelReadSuspended because it's actually same with messageReceivedLast
- Rename messageReceived to channelRead
- Rename messageReceivedLast to channelReadComplete

We renamed messageReceivedLast to channelReadComplete because it
reflects what it really is for.  Also, we renamed messageReceived to
channelRead for consistency in method names.
2013-07-09 23:58:51 +09:00
Trustin Lee
cbd8817905 Remove MessageList from public API and change ChannelInbound/OutboundHandler accordingly
I must admit MesageList was pain in the ass.  Instead of forcing a
handler always loop over the list of messages, this commit splits
messageReceived(ctx, list) into two event handlers:

- messageReceived(ctx, msg)
- mmessageReceivedLast(ctx)

When Netty reads one or more messages, messageReceived(ctx, msg) event
is triggered for each message.  Once the current read operation is
finished, messageReceivedLast() is triggered to tell the handler that
the last messageReceived() was the last message in the current batch.

Similarly, for outbound, write(ctx, list) has been split into two:

- write(ctx, msg)
- flush(ctx, promise)

Instead of writing a list of message with a promise, a user is now
supposed to call write(msg) multiple times and then call flush() to
actually flush the buffered messages.

Please note that write() doesn't have a promise with it.  You must call
flush() to get notified on completion. (or you can use writeAndFlush())

Other changes:

- Because MessageList is completely hidden, codec framework uses
  List<Object> instead of MessageList as an output parameter.
2013-07-09 23:51:48 +09:00
Norman Maurer
7dda4b9ce4 [#1532] Remove @deprecated ByteBufIndexFinder and all methods that take it as argument 2013-07-06 20:14:53 +02:00
Trustin Lee
fa277eda20 Fix reported leaks 2013-06-13 15:24:19 +09:00
Trustin Lee
427d9c4bf2 Fix test failures and reported leaks 2013-06-13 15:18:11 +09:00
Trustin Lee
6d1cd0d0cd ReferenceCountException -> IllegalReferenceCountException 2013-06-13 14:00:15 +09:00
Trustin Lee
14158070bf Revamp the core API to reduce memory footprint and consumption
The API changes made so far turned out to increase the memory footprint
and consumption while our intention was actually decreasing them.

Memory consumption issue:

When there are many connections which does not exchange data frequently,
the old Netty 4 API spent a lot more memory than 3 because it always
allocates per-handler buffer for each connection unless otherwise
explicitly stated by a user.  In a usual real world load, a client
doesn't always send requests without pausing, so the idea of having a
buffer whose life cycle if bound to the life cycle of a connection
didn't work as expected.

Memory footprint issue:

The old Netty 4 API decreased overall memory footprint by a great deal
in many cases.  It was mainly because the old Netty 4 API did not
allocate a new buffer and event object for each read.  Instead, it
created a new buffer for each handler in a pipeline.  This works pretty
well as long as the number of handlers in a pipeline is only a few.
However, for a highly modular application with many handlers which
handles connections which lasts for relatively short period, it actually
makes the memory footprint issue much worse.

Changes:

All in all, this is about retaining all the good changes we made in 4 so
far such as better thread model and going back to the way how we dealt
with message events in 3.

To fix the memory consumption/footprint issue mentioned above, we made a
hard decision to break the backward compatibility again with the
following changes:

- Remove MessageBuf
- Merge Buf into ByteBuf
- Merge ChannelInboundByte/MessageHandler and ChannelStateHandler into ChannelInboundHandler
  - Similar changes were made to the adapter classes
- Merge ChannelOutboundByte/MessageHandler and ChannelOperationHandler into ChannelOutboundHandler
  - Similar changes were made to the adapter classes
- Introduce MessageList which is similar to `MessageEvent` in Netty 3
- Replace inboundBufferUpdated(ctx) with messageReceived(ctx, MessageList)
- Replace flush(ctx, promise) with write(ctx, MessageList, promise)
- Remove ByteToByteEncoder/Decoder/Codec
  - Replaced by MessageToByteEncoder<ByteBuf>, ByteToMessageDecoder<ByteBuf>, and ByteMessageCodec<ByteBuf>
- Merge EmbeddedByteChannel and EmbeddedMessageChannel into EmbeddedChannel
- Add SimpleChannelInboundHandler which is sometimes more useful than
  ChannelInboundHandlerAdapter
- Bring back Channel.isWritable() from Netty 3
- Add ChannelInboundHandler.channelWritabilityChanges() event
- Add RecvByteBufAllocator configuration property
  - Similar to ReceiveBufferSizePredictor in Netty 3
  - Some existing configuration properties such as
    DatagramChannelConfig.receivePacketSize is gone now.
- Remove suspend/resumeIntermediaryDeallocation() in ByteBuf

This change would have been impossible without @normanmaurer's help. He
fixed, ported, and improved many parts of the changes.
2013-06-10 16:10:39 +09:00
Trustin Lee
e58af1bc94 Readability / Fix inception year 2013-05-17 17:59:13 +09:00
kxbmap
3a5aa12000 Add LengthFieldPrepender.lengthAdjustment parameter 2013-05-17 17:58:53 +09:00
Luke Wood
f841056752 Fix SnappyFramedDecoder checksum validation and add a pair of suitable unit tests 2013-05-16 14:10:54 +01:00
Trustin Lee
80f4c0b334 Merge SnappyChecksumUtil into Snappy and make calculateChecksum() public / Overall clean up 2013-05-09 15:23:56 +09:00
Norman Maurer
73c35aef4e [#1309] Make sure ReplayDecoder respect isSingleDecode()
* This could cause for example corrupt WebSocketFrame's if they was written from the server
  to the client directly after it send the handshake response.
2013-04-25 08:32:07 +02:00
Trustin Lee
1b3d7f5325 Make sure handlerAdded() is called before forwarding the buffer content of the removed handler
- Added a test case that reproduces the problem in ReplayingDecoderTest
- Call newHandler.handlerAdded() *before* oldHandler.handlerRemoved() to ensure newHandlerAdded() is called before forwarding the buffer content of the old handler in replace0().
2013-04-24 19:25:43 +09:00
Trustin Lee
5dd35448a2 Add a failing test that will pass once #1306 is fixed 2013-04-24 18:47:36 +09:00
Trustin Lee
b5989e2449 Reduce exception instantiation overhead in SslHandler / Reduce unnecessary empty array creation
- Added EmptyArrays as an internal utility class
2013-04-24 09:32:53 +09:00
Norman Maurer
9c4bfa44d9 [#1294] Make sure ByteBuf is released once written to channel 2013-04-22 11:04:56 +02:00
Trustin Lee
4a5dc32224 Fix a bug where LineBasedFrameDecoder does not handle too long lines correctly
- Related: #1287
2013-04-19 13:04:20 +09:00
Andrei Pozolotin
a3e760a003 fix #1234 - duplicate package-info.java errors in eclipse requires release of netty-build v 19 and netty-parent update. 2013-04-05 05:38:05 +09:00
Norman Maurer
af4b71a00e Remove special handling of Object[] in codec framework (a.k.a unfolding)
- Fixes #1229
- Primarily written by @normanmaurer and revised by @trustin

This commit removes the notion of unfolding from the codec framework
completely.  Unfolding was introduced in Netty 3.x to work around the
shortcoming of the codec framework where encode() and decode() did not
allow generating multiple messages.

Such a shortcoming can be fixed by changing the signature of encode()
and decode() instead of introducing an obscure workaround like
unfolding.  Therefore, we changed the signature of them in 4.0.

The change is simple, but backward-incompatible.  encode() and decode()
do not return anything.  Instead, the codec framework will pass a
MessageBuf<Object> so encode() and decode() can add the generated
messages into the MessageBuf.
2013-04-03 21:44:54 +09:00
Prajwal Tuladhar
05850da863 enable checkstyle for test source directory and fix checkstyle errors 2013-03-30 13:18:57 +01:00
Norman Maurer
806e9b1f8c Make sure we handle outbound messages of type ByteBuf special 2013-03-11 08:59:00 +09:00
Luke Wood
e45db60b5e [#1069] Snappy decoding fixes
* Correct reading offset of 1-byte-offset copies
* Keep track of how much we've written so far in order to validate offsets
* Uncomment and reduce number of tests
2013-02-25 16:36:50 +00:00
Trustin Lee
58c72acd42 Add more failing tests to SnappyIntegrationTest 2013-02-22 10:06:24 -08:00
Luke Wood
7a8e9d7993 Ensure that at least 4 bytes are remaining in input buffer when compiling our hash table in search of duplicates, and restrict encoded frame sizes to 32,767 bytes (2^15 - 1) 2013-02-22 14:24:50 +00:00
Trustin Lee
17641d52fb Integrate @timboudreau's SnappyIntegrationTest improvements 2013-02-21 18:38:58 -08:00
Norman Maurer
c93f5afa99 [#1012] Cleanup 2013-02-20 12:52:55 +01:00
Atsuhiko Yamanaka
8fdf788cbd [#1012] Replace forked jzlib with official jzlib and add a test. 2013-02-20 12:49:05 +01:00
Luke Wood
c1783cc8d3 More fixes to snappy: * Make Snappy.decode stateful instead of relying on the uncompressed length being equal to the compressed length * Correctly handle copies where offset < length * Take copies from the output buffer in decoding * Make the maximum encoded chunk size 32kB for compressed data 2013-02-11 15:05:27 +01:00
Trustin Lee
b4f4b95739 Move io.netty.logging to io.netty.internal / Move Signal out of internal because we use it in Channel*MessageAdapters 2013-02-11 20:08:18 +09:00
Trustin Lee
b9996908b1 Implement reference counting
- Related: #1029
- Replace Freeable with ReferenceCounted
- Add AbstractReferenceCounted
- Add AbstractReferenceCountedByteBuf
- Add AbstractDerivedByteBuf
- Add EmptyByteBuf
2013-02-10 13:10:09 +09:00
Trustin Lee
46a249a26b Add more failing tests for Snappy codec 2013-02-10 00:33:41 +09:00
Trustin Lee
bd87284829 Clean up SnappyIntegrationTest / Fix checksum encoding 2013-02-10 00:20:49 +09:00
Trustin Lee
8bd7ea2f93 Fix SnappyFramedEncoderTest & SnappyFramedEncoder
- chunkLength is 3 bytes
2013-02-10 00:03:47 +09:00
Trustin Lee
3268781607 Fix SnappyFramedDecoderTest 2013-02-09 23:57:14 +09:00
Trustin Lee
319b7fa69a Fix SnappyFramedEncoder/Decoder / Fix Snappy preamble encoding / Add test for #1002
- The new test still fails due to a bug in Snappy.encode/decode()
2013-02-09 23:39:33 +09:00
Trustin Lee
82c46180c9 Tighten access modifier of encode/decode() 2013-02-08 17:37:16 +09:00
Trustin Lee
fa1b49de98 More robust automatic messageType detection for ChannelInboundMessageHandlerAdapter and MessageToMessageDecoder 2013-02-08 15:45:17 +09:00
Trustin Lee
d8c0bf3be2 Add the 'nextBufferType' parameter to ByteArrayEncoder like did to StringEncoder / Consistent parameter order 2013-02-08 01:36:41 +09:00
Luke Wood
bfe44180f9 Fix SnappyFramedDecoder issues
- Checksum header was being incorrectly read due to incorrect order of
  shift and masking operations.
- Length field of 1-byte copy was being incorrectly interpreted due to a
  typo in the binary mask used to extract it.
- Use ByteBuf.readUnsignedByte() instead of readByte() & 0xff
- Use bitwise-OR wherever possible
- Use EmbeddedByteChannel to test
- Use ByteBuf comparison instead of array comparison
- Work done by @lw346 and then revised by @trustin
2013-02-01 13:10:34 +09:00
Trustin Lee
2ec932798f Replace .readable() and .writable() to .isReadable() and .isWritable() 2013-01-31 18:24:33 +01:00
Luke Wood
bd0339ce1a Fix buffer over-run in Snappy codec when emitting a copy that leaves < 4 bytes remaining 2013-01-30 12:15:57 +01:00
Norman Maurer
b742dcc209 [#902] Remove usage of generics for output of Encoder/Decoder to make them more flexible again 2013-01-09 07:13:31 +01:00
Trustin Lee
e353540d47 Move snappy codec to netty-codec 2012-12-19 18:12:18 +09:00
Norman Maurer
d7d8503f5c Add a line-based frame decoder with good performance.
Using DelimiterBasedFrameDecoder with Delimiters.lineDelimiter() has
quadratic performance in the size of the input buffer.  Needless to
say, the performance degrades pretty quickly as the size of the buffer
increases.  Larger MTUs or loopback connections can make it so bad that
it appears that the code is "busy waiting", when in fact it's spending
almost 100% of the CPU time in DelimiterBasedFrameDecoder.indexOf().

Add a new LineBasedFrameDecoder that decodes line-delimited frames
in O(n) instead of DelimiterBasedFrameDecoder's O(n^2) implementation.
In OpenTSDB's telnet-style protocol decoder this resulted in throughput
increases of an order of magnitude.

Change DelimiterBasedFrameDecoder to automatically detect when the
frames are delimited by line endings, and automatically switch to
using LineBasedFrameDecoder under the hood.  This means that all Netty
applications out there that using the combo DelimiterBasedFrameDecoder
with Delimiters.lineDelimiter() will automatically benefit from the
better performance of LineBasedFrameDecoder, without requiring a code
change.
2012-11-25 13:36:55 +01:00
Trustin Lee
9ac522382a Fix visibility warnings from the inspector. 2012-11-12 14:03:43 +09:00
Trustin Lee
a05064d3eb Fix more inspection warnings + compilation errors 2012-11-12 13:25:00 +09:00
Trustin Lee
6f2840193a Fix inspection warnings related with JUnit usage 2012-11-12 12:45:06 +09:00
Trustin Lee
b4f796c5e3 Use 'x' over "x" wherever possible / String.equals("") -> isEmpty() 2012-11-10 08:03:52 +09:00
Trustin Lee
0d0eb0abfb Remove unused imports 2012-11-10 07:05:18 +09:00
Trustin Lee
58ba0de659 Remove unnecessarily qualified static access 2012-11-10 01:32:21 +09:00
Norman Maurer
516351b82c Add a basic unit test that I wrote while porting niosmtp to netty 4 2012-09-15 19:14:59 +02:00
Trustin Lee
0b11fb2ead [#531] Move io.netty.util.Signal to io.netty.util.internal 2012-08-18 18:50:54 +09:00
Trustin Lee
e157ea1a66 Allow a user use any type as a ReplayingDecoder state / AIO cleanup
- Removed VoidEnum because a user can now specify Void instead
- AIO: Prefer discardReadBytes to clear
- AIO: Fixed a potential bug where notifyFlushFutures() is not called
       if flush() was requested with no outbound data
2012-07-09 20:40:43 +09:00
Norman Maurer
220e4e886f Fix ArrayIndexOutOfBoundsException in ReplayDecoderBuffer which could happen when calling getByte(..) or getUnsignedByte(..). See #445 2012-07-08 13:47:17 +02:00
Trustin Lee
876847fd20 Merge MessageBufs and ByteBufs into Unpooled
- e.g. Unpooled.messageBuffer()
- It will make much more sense once we introduce pooling:
  - i.e. Pooled.buffer()
2012-06-11 17:02:29 +09:00
Trustin Lee
a849d11877 ChannelBuffers -> ByteBufs / Add MessageBuf & ChannelBuf
- Add MessageBuf which replaces java.util.Queue
- Add ChannelBuf which is common type of ByteBuf and ChannelBuf
- ChannelBuffers was renamed to ByteBufs
- Add MessageBufs
- All these changes are going to replace ChannelBufferHolder.
2012-06-10 11:31:39 +09:00
Trustin Lee
5164d91255 Rename ChannelBuffer to ByteBuf as discussed before
- ChannelBuffer gives a perception that it's a buffer of a
  channel, but channel's buffer is now a byte buffer or a message
  buffer.  Therefore letting it be as is is going to be confusing.
2012-06-10 11:08:43 +09:00
Trustin Lee
e376888d48 Replace 'Stream' with 'Byte'
- In computing, 'stream' means both byte stream and message stream,
  which is confusing.
- Also, we were already mixing stream and byte in some places and
  it's better use the terms consistently.
  (e.g. inboundByteBuffer & inbound stream)
2012-06-09 21:05:59 +09:00
Trustin Lee
3442ff90e8 Ported all tests in netty-codec to the new embedder 2012-06-07 20:17:26 +09:00
Trustin Lee
7bc10f2eba Replace codec embedder with EmbeddedChannel which can test any handlers
- Added EventExecutor.inEventLoop(Thread) and replaced executor identity
  comparison in DefaultChannelPipeline with it - more elegant IMO
- Removed the test classes that needs rewrite or is of no use
2012-06-07 19:39:37 +09:00
Trustin Lee
5e93d206ff Overhaul - Split ChannelHandler & Merge ChannelHandlerContext
- Extracted some handler methods from ChannelInboundHandler into
  ChannelStateHandler
- Extracted some handler methods from ChannelOutboundHandler into
  ChannelOperationHandler
- Moved exceptionCaught and userEventTriggered are now in
  ChannelHandler
  
- Channel(Inbound|Outbound)HandlerContext is merged into
  ChannelHandlerContext
- ChannelHandlerContext adds direct access methods for inboud and
  outbound buffers
  - The use of ChannelBufferHolder is minimal now.
    - Before: inbound().byteBuffer()
    - After: inboundByteBuffer()
    - Simpler and better performance
    
- Bypass buffer types were removed because it just does not work at all
  with the thread model.
  - All handlers that uses a bypass buffer are broken.  Will fix soon.

- CombinedHandlerAdapter does not make sense anymore either because
  there are four handler interfaces to consider and often the two
  handlers will implement the same handler interface such as
  ChannelStateHandler.  Thinking of better ways to provide this feature
2012-06-07 14:52:33 +09:00
Trustin Lee
1eced1e9e3 Update license headers 2012-06-04 13:31:44 -07:00
Trustin Lee
cc4f705029 Replace ChannelBuffer.toByteBuffer() with hasNioBuffer() and nioBuffer()
... just like we do with byte arrays.  toByteBuffer() and
toByteBuffers() had an indeterministic behavior and thus it could not
tell when the returned NIO buffer is shared or not.  nioBuffer() always
returns a view buffer of the Netty buffer.  The only case where
hasNioBuffer() returns false and nioBuffer() fails is the
CompositeChannelBuffer, which is not very commonly used and *slow*.
2012-06-02 01:30:55 -07:00
Trustin Lee
14cf227c46 Fix a test failure 2012-06-01 23:11:01 -07:00
Trustin Lee
77274ae743 Automated code clean-up 2012-05-31 12:03:01 -07:00
Trustin Lee
34f697a06c Forward-port the JBoss Marshalling codec 2012-05-30 23:36:03 -07:00
Trustin Lee
d7a198a60f Zap marshalling and spdy codec to forwardport again 2012-05-30 23:14:18 -07:00
norman
14b2a0db99 Add JBoss Marshalling Encoder/Decoder. See #324 2012-05-30 19:09:54 -07:00
Trustin Lee
b10cf29393 Add isDecodable/isEncodable() to codecs to support stacked codecs 2012-05-29 13:34:01 -07:00
Trustin Lee
626c5ef9c9 Remove the classes that are not part of Netty 4.0.0.Alpha1
- Will add them back before Beta1
2012-05-27 19:39:10 -07:00
Trustin Lee
9d3eeb9719 Fix compilation errors 2012-05-18 17:50:38 +09:00
Trustin Lee
ca12e41406 Fix a compiler warning 2012-05-18 16:58:40 +09:00
Trustin Lee
5c3b432f60 Fix test failures due to recent changes with codec exceptions 2012-05-18 15:45:12 +09:00
Trustin Lee
251a18160c Move VoidEnum to util 2012-05-18 14:34:42 +09:00
Trustin Lee
5344dc242c Finish porting the codec package to the new API
- Removed deprecated classes
- Changed type parameter of StreamToMessageDecoder and
  MessageToMessageDecoder for more flexibility
- Made all tests in the codec package pass
2012-05-17 16:38:25 +09:00
Trustin Lee
1bf0dfe64a Retrofit ReplayingDecoder with the new API
- Moved up to 'codec' from 'codec.replay'
- Test passes with Redis client codec
2012-05-17 12:37:37 +09:00
Trustin Lee
92a688e5b2 Retrofit the codec framework with the new API (in progress)
- Replaced FrameDecoder and OneToOne(Encoder|Decoder) with:
  - (Stream|Message)To(String|Message)(Encoder|Decoder)
- Moved the classes in 'codec.frame' up to 'codec'
- Fixed some bugs found while running unit tests
2012-05-16 23:02:06 +09:00
Trustin Lee
894ececbb7 Convert DOS line ending to UNIX line ending 2012-05-15 17:14:02 +09:00
norman
0a6f7395f3 Merge redis codec to master branch. See #201 2012-03-12 12:28:07 +01:00
Trustin Lee
303c1b5f79 Overall cleanup / Add lost old jzlib headers 2012-01-13 17:41:18 +09:00
Trustin Lee
8663716d38 Issue #60: Make the project multi-module
Split the project into the following modules:
* common
* buffer
* codec
* codec-http
* transport
* transport-*
* handler
* example
* testsuite (integration tests that involve 2+ modules)
* all (does nothing yet, but will make it generate netty.jar)

This commit also fixes the compilation errors with transport-sctp on
non-Linux systems.  It will at least compile without complaints.
2011-12-28 19:44:04 +09:00