Commit Graph

552 Commits

Author SHA1 Message Date
Jakob Buchgraber
aed13ba5ef Split a JSON byte stream into JSON objects/arrays. Fixes #2536
Motivation:

See GitHub Issue #2536.

Modifications:

Introduce the class JsonObjectDecoder to split a JSON byte stream
into individual JSON objets/arrays.

Result:

A Netty application can now handle a byte stream where multiple JSON
documents follow eachother as opposed to only a single JSON document
per request.
2014-07-03 18:34:29 +09:00
Trustin Lee
d0912f2709 Fix most inspector warnings
Motivation:

It's good to minimize potentially broken windows.

Modifications:

Fix most inspector warnings from our profile
Update IntObjectHashMap

Result:

Cleaner code
2014-07-02 19:55:07 +09:00
xmxsuperstar
79195da0d7 fix example missing break statement in ReplayingDecoder 2014-06-28 21:42:23 +02:00
Zhihui Jiao
c4441642bb Fix inconsistent code in the doc 2014-06-27 06:48:40 +02:00
Trustin Lee
d2e385acc2 Fix incorrect bytesBefore/indexOf() in ReplayingDecoderBuffer
Motivation:

bytesBefore(length, ...), bytesBefore(index, length, ...), and
indexOf(fromIndex, toIndex,...) in ReplayingDecoderBuffer are buggy.
They trigger 'REPLAY even when they don't need to.

Modification:

Implement the buggy methods properly so that REPLAYs are not triggered
unnecessarily.

Result:

Correct behvaior
2014-06-26 18:56:33 +09:00
Norman Maurer
4d2b78ca3c Reduce the memory copies in JdkZlibEncoder
Motivation:

At the moment we use a lot of unnecessary memory copies in JdkZlibEncoder. This is caused by either allocate a to small ByteBuf and expand it later or using a temporary byte array.
Beside this the memory footprint of JdkZlibEncoder is pretty high because of the byte[] used for compressing.

Modification:

- Override allocateBuffer(...) and calculate the estimatedsize in there, this reduce expanding of the ByteBuf later
- Not use byte[] in the instance itself but allocate a heap ByteBuf and write directly into the byte array

Result:

Less memory copies and smaller memory footprint
2014-06-26 11:12:19 +02:00
Trustin Lee
937f790f70 Checkstyle 2014-06-26 17:48:32 +09:00
Trustin Lee
5f889d92a1 Fix buffer leaks in Bzip2Decoder(Test)
If decompression fails, the buffer that contains the decompressed data
is not released.  Bzip2DecoderTest.testStreamCrcError() also does not
release the partial output Bzip2Decoder produces.
2014-06-26 17:48:32 +09:00
Idel Pivnitskiy
f9021a6061 Implement a Bzip2Decoder
Motivation:

Bzip2Decoder provides receiving data compressed in bzip2 format.

Modifications:

Added classes:
- Bzip2Decoder
- Bzip2Constants
- Bzip2BlockDecompressor
- Bzip2HuffmanStageDecoder
- Bzip2MoveToFrontTable
- Bzip2Rand
- Crc32
- Bzip2DecoderTest

Result:

Implemented and tested new decoder which can uncompress incoming data in bzip2 format.
2014-06-24 14:50:09 +09:00
Norman Maurer
12a3e23e47 MessageToByteEncoder always starts with ByteBuf that use initalCapacity == 0
Motivation:

MessageToByteEncoder always starts with ByteBuf that use initalCapacity == 0 when preferDirect is used. This is really wasteful in terms of performance as every first write into the buffer will cause an expand of the buffer itself.

Modifications:

 - Change ByteBufAllocator.ioBuffer() use the same default initialCapacity as heapBuffer() and directBuffer()
 - Add new allocateBuffer method to MessageToByteEncoder that allow the user to do some smarter allocation based on the message that will be encoded.

Result:

Less expanding of buffer and more flexibilty when allocate the buffer for encoding.
2014-06-24 13:55:21 +09:00
Trustin Lee
8c25830b0b Move haproxy codec to a separate module 2014-06-21 15:59:21 +09:00
Jon Keys
d7b2affe32 Add HAProxy protocol decoder
Motivation:

The proxy protocol provides client connection information for proxied
network services. Several implementations exist (e.g. Haproxy, Stunnel,
Stud, Postfix), but the primary motivation for this implementation is to
support the proxy protocol feature of Amazon Web Services Elastic Load
Balancing.

Modifications:

This commit adds a proxy protocol decoder for proxy protocol version 1
as specified at:

  http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt

The foundation for version 2 support is also in this commit but it is
explicitly NOT supported due to a lack of external implementations to
test against.

Result:

The proxy protocol decoder can be used to send client connection
information to inbound handlers in a channel pipeline from services
which support the proxy protocol.
2014-06-21 15:59:21 +09:00
Trustin Lee
085a61a310 Refactor FastThreadLocal to simplify TLV management
Motivation:

When Netty runs in a managed environment such as web application server,
Netty needs to provide an explicit way to remove the thread-local
variables it created to prevent class loader leaks.

FastThreadLocal uses different execution paths for storing a
thread-local variable depending on the type of the current thread.
It increases the complexity of thread-local removal.

Modifications:

- Moved FastThreadLocal and FastThreadLocalThread out of the internal
  package so that a user can use it.
- FastThreadLocal now keeps track of all thread local variables it has
  initialized, and calling FastThreadLocal.removeAll() will remove all
  thread-local variables of the caller thread.
- Added FastThreadLocal.size() for diagnostics and tests
- Introduce InternalThreadLocalMap which is a mixture of hard-wired
  thread local variable fields and extensible indexed variables
- FastThreadLocal now uses InternalThreadLocalMap to implement a
  thread-local variable.
- Added ThreadDeathWatcher.unwatch() so that PooledByteBufAllocator
  tells it to stop watching when its thread-local cache has been freed
  by FastThreadLocal.removeAll().
- Added FastThreadLocalTest to ensure that removeAll() works
- Added microbenchmark for FastThreadLocal and JDK ThreadLocal
- Upgraded to JMH 0.9

Result:

- A user can remove all thread-local variables Netty created, as long as
  he or she did not exit from the current thread. (Note that there's no
  way to remove a thread-local variable from outside of the thread.)
- FastThreadLocal exposes more useful operations such as isSet() because
  we always implement a thread local variable via InternalThreadLocalMap
  instead of falling back to JDK ThreadLocal.
- FastThreadLocalBenchmark shows that this change improves the
  performance of FastThreadLocal even more.
2014-06-19 21:13:55 +09:00
Norman Maurer
984b0aa961 [#2572] Correctly calculate length of output buffer before inflate to fix IndexOutOfBoundException
Motivation:

JdkZlibDecoder fails to decode because the length of the output buffer is not calculated correctly.
This can cause an IndexOutOfBoundsException or data-corruption when the PooledByteBuffAllocator is used.

Modifications:

Correctly calculate the length

Result:

No more IndexOutOfBoundsException or data-corruption.
2014-06-16 10:17:02 +02:00
Trustin Lee
776ac4ba19 Use FastThreadLocal in more places 2014-06-14 17:46:10 +09:00
Trustin Lee
c076c33901 Backport the additional AsciiString/TextHeader changes from master
- Add useful static methods to AsciiString
- Add more getters in TextHeaders
- Remove unnecessary utility methods in SpdyHttpHeaders
2014-06-14 17:33:34 +09:00
Trustin Lee
681d460938 Introduce TextHeaders and AsciiString
Motivation:

We have quite a bit of code duplication between HTTP/1, HTTP/2, SPDY,
and STOMP codec, because they all have a notion of 'headers', which is a
multimap of string names and values.

Modifications:

- Add TextHeaders and its default implementation
- Add AsciiString to replace HttpHeaderEntity
  - Borrowed some portion from Apache Harmony's java.lang.String.
- Reimplement HttpHeaders, SpdyHeaders, and StompHeaders using
  TextHeaders
- Add AsciiHeadersEncoder to reuse the encoding a TextHeaders
  - Used a dedicated encoder for HTTP headers for better performance
    though
- Remove shortcut methods in SpdyHeaders
- Replace SpdyHeaders.getStatus() with HttpResponseStatus.parseLine()

Result:

- Removed quite a bit of code duplication in the header implementations.
- Slightly better performance thanks to improved header validation and
  hash code calculation
2014-06-14 15:36:19 +09:00
belliottsmith
2a2a21ec59 Introduce FastThreadLocal which uses an EnumMap and a predefined fixed set of possible thread locals
Motivation:
Provide a faster ThreadLocal implementation

Modification:
Add a "FastThreadLocal" which uses an EnumMap and a predefined fixed set of possible thread locals (all of the static instances created by netty) that is around 10-20% faster than standard ThreadLocal in my benchmarks (and can be seen having an effect in the direct PooledByteBufAllocator benchmark that uses the DEFAULT ByteBufAllocator which uses this FastThreadLocal, as opposed to normal instantiations that do not, and in the new RecyclableArrayList benchmark);

Result:
Improved performance
2014-06-13 10:56:18 +02:00
Trustin Lee
8b0a0f9a8f Introduce MessageAggregator and DecoderResultProvider
Motivation:

We have different message aggregator implementations for different
protocols, but they are very similar with each other.  They all stems
from HttpObjectAggregator.  If we provide an abstract class that provide
generic message aggregation functionality, we will remove their code
duplication.

Modifications:

- Add MessageAggregator which provides generic message aggregation
- Reimplement all existing aggregators using MessageAggregator
- Add DecoderResultProvider interface and extend it wherever possible so
  that MessageAggregator respects the state of the decoded message

Result:

Less code duplication
2014-06-05 16:51:14 +09:00
Norman Maurer
b0ddfb9b65 [#2525] Use VoidChannelPromise in MessageToMessageEncoder when possible
Motivation:
At the moment MessageToMessageEncoder uses ctx.write(msg) when have more then one message was produced. This may produce more GC pressure then necessary as when the original ChannelPromise is a VoidChannelPromise we can safely also use one when write messages.

Modifications:
Use VoidChannelPromise when the original ChannelPromise was of this type

Result:
Less object creation and GC pressure
2014-06-01 19:26:20 +02:00
Norman Maurer
880acbca72 Use ByteBuf.readSlice(...).retain() to minimize memory copies.
Motivation:
At the moment we call ByteBuf.readBytes(...) in these handlers but with optimizations done as part of 25e0d9d we can just use readSlice(...).retain() and eliminate the memory copy.

Modifications:
Replace ByteBuf.readBytes(...) usage with readSlice(...).retain().

Result:
Less memory copies.
2014-05-10 17:20:53 +02:00
Trustin Lee
db3709e652 Synchronized between 4.1 and master
Motivation:

4 and 5 were diverged long time ago and we recently reverted some of the
early commits in master.  We must make sure 4.1 and master are not very
different now.

Modification:

Fix found differences

Result:

4.1 and master got closer.
2014-04-25 00:38:02 +09:00
Martin Krüger
d854d3a617 Fix chunk type for stream identifier
Motivation:
The problem with the current snappy implementation is that it does
not comply with framing format definition found on
https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt

The document describes that chunk type of the stream identifier is defined
as 0xff. The current implentation uses 0x80.

Modifications:
This patch replaces the first byte of the chunk type of the stream identifier
with 0xff.

Result:
After this modification the snappy implementation is compliant to the
framing format described at
https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt.
This results in a better compatibility with other implementations.
2014-04-19 21:06:28 +02:00
Norman Maurer
88481131be [#2353] Use a privileged block to get ClassLoader and System property if needed
Motivation:
When using System.getProperty(...) and various methods to get a ClassLoader it will fail when a SecurityManager is in place.

Modifications:
Use a priveled block if needed. This work is based in the PR #2353 done by @anilsaldhana .

Result:
Code works also when SecurityManager is present
2014-04-08 14:12:25 +02:00
Norman Maurer
cb9660f83d Allow the user to call slice().retain() or duplicate.retain() in his/her ByteToMessageDecoder.decode(...) method.
Motivation:
At the moment a user can not safetly call slice().retain() or duplicate.retain()in the ByteToMessageDecoder.decode(...) implementation without the risk to see coruption because we may call discardSomeReadBytes() to make room on the buffer once the handling is done.

Modifications:
Check for the refCnt() before call discardSomeReadBytes() and also check before call decode(...) to create a copy if needed.

Result:
The user can safetly call slice().retain() or duplicate.retain() in his/her ByteToMessageDecoder.decode(...) method.
2014-04-07 11:53:28 +02:00
Alexey Diomin
2a4999b6b8 [#2339] Reduce memory usage in ProtobufVarint32LengthFieldPrepender
Motivation:

Reduce memory usage in ProtobufVarint32LengthFieldPrepender.

Modifications:

Explicit set the buffer size that is needed for the header (between 1 and 5 bytes).

Result:

Less memory usage in ProtobufVarint32LengthFieldPrepender.
2014-03-28 19:57:06 +01:00
Norman Maurer
adcc2d4b4b Remove condition in ChannelHandlerAdapter.isSharable() by caching the result of the annotation lookup.
Motivation:
Remove the synchronization bottleneck and so speed up things

Modifications:
Introduce a ThreadLocal cache that holds mappings between classes of ChannelHandlerAdapater implementations and the result of checking if the @Sharable annotation is present.
This way we only will need to do the real check one time and server the other calls via the cache. A ThreadLocal and WeakHashMap combo is used to implement the cache
as this way we can minimize the conditions while still be sure we not leak class instances in containers.

Result:
Less conditions during adding ChannelHandlerAdapter to the ChannelPipeline
2014-03-12 12:31:52 +01:00
Norman Maurer
d89bfc593e Fix buffer leak in test which was introduced while implement ZLIB_OR_NONE support. Related to [#2269] 2014-03-06 20:13:30 +01:00
Jakob Buchgraber
9fb235459e Add ZLIB_OR_NONE support to JdkZlibDecoder [#2016] 2014-03-03 06:37:47 +01:00
Trustin Lee
8837afddf8 Enable a user specify an arbitrary information with ReferenceCounted.touch()
- Related: #2163
- Add ResourceLeakHint to allow a user to provide a meaningful information about the leak when touching it
- DefaultChannelHandlerContext now implements ResourceLeakHint to tell where the message is going.
- Cleaner resource leak report by excluding noisy stack trace elements
2014-02-13 18:16:25 -08:00
Trustin Lee
45e70d9935 Add ReferenceCounted.touch() / Add missing retain() overrides
- Fixes #2163
- Inspector warnings
2014-02-13 18:10:11 -08:00
Mirko Caserta
ee8571824b CDATA support 2014-02-13 17:13:49 -08:00
Mirko Caserta
086dbd1ba1 Fixed the XML decoder 2014-02-13 17:13:39 -08:00
Trustin Lee
2d5a3b5898 Add XML decoder
- based on @mcaserta's work at https://github.com/netty/netty/pull/1121
- not ready for a merge yet
2014-02-13 17:13:29 -08:00
Trustin Lee
499033d44f Add a shortcut method for collision-free naming 2014-02-13 15:17:09 -08:00
Norman Maurer
573b54a93d [#1907] LengthFieldPrepender should better extend MessageToMessageEncoder for less memory copies 2014-02-13 14:52:12 -08:00
Vladimir Schafer
3d531231fe #2183 Fix for releasing of the internal cumulation buffer in ByteToMessageDecoder 2014-02-06 20:07:56 +01:00
Norman Maurer
85a997d5ec Correctly calculate checksum when using GZIP. Part of [#2168] 2014-02-06 10:47:59 +01:00
Norman Maurer
0f7379157a [#2168] Eliminate unnessary memory copy for heap buffers in JdkZlibEncoder
* Also adjust tests so it test with direct and heap buffers
2014-01-30 07:02:14 +01:00
Norman Maurer
faf8becf2e Make use of ByteBufProcessor for extract initial line and headers
This gives some nice performance boost as readByte() is quite expensive because of the index / replay checks.
2014-01-16 20:14:08 +01:00
Trustin Lee
3f7b674db8 Fix bugs in ZLIB codec where they produce malformed stream or their streams are not flushed on time
- Fixes #2014
- Add the tests that mix JDK ZLIB codec and JZlib codecs
- Fix a bug where JdkZlibEncoder does not encode the GZIP header when nothing was written to te channel
- Fix a bug where the encoders do not consider the overhead of the wrapper format when calculating the estimated compressed output size.
- Fix a bug where the decoders do not discard the received data after the compressed stream is finished
2013-11-29 18:09:04 +09:00
Trustin Lee
54db9ec725 Use StringUtil.simpleClassName(..) instead of Class.getSimpleName() where necessary
- Class.getSimpleName() doesn't render anonymous classes very well
- + some minor cleanup
2013-11-04 19:46:15 +09:00
Trustin Lee
4f5c52572b Allow a user to define his/her own length field decoder
- Fixes #1960
2013-11-02 14:29:51 +01:00
Luke Wood
31c1513229 [#1958] Use CRC32-C in Snappy as per the spec 2013-11-01 20:54:55 +01:00
Trustin Lee
1c2352e6a0 Replace constructor calls on UniqueName and its subtypes with valueOf() wherever possible 2013-10-25 20:58:53 +09:00
Norman Maurer
8930cefab8 [#1812] Rework ByteToMessageDecoder.channelRead(..) method to allow for inlining 2013-10-25 13:32:33 +02:00
Norman Maurer
d946659520 [#1906] Use a ByteBuf allocator from the ByteBufAllocator when encode Strings 2013-10-09 21:18:08 +02:00
Norman Maurer
00f99dbff6 [#1893] Fix bug in JdkZlibDecoder which did not let it handle large data 2013-10-07 07:31:21 +02:00
Norman Maurer
aedd631dbb [#1798] Correctly handle skipping of too big frames 2013-08-29 13:51:50 +02:00
Norman Maurer
7a38163743 [#1788] Correctly decode CRC32 and ISIZE when using JdkZlibDecoder
Because of incorrect decoding a CompressionException was thrown before
2013-08-28 11:00:47 +02:00
Derek Troy-West
6a4131c10f Currently the system variable to chose which type of zlib decoder is being negated 2013-08-27 08:27:34 +02:00
Norman Maurer
65d24a7647 [#1735] Disable usage of JdkZlibDecoder by default, will be enabled in 4.1.0.Final 2013-08-26 08:15:30 +02:00
Norman Maurer
39cabcd36c [#1481] Add a JdkZlibDecoder which has no dependencies 2013-08-14 07:41:28 +02:00
Norman Maurer
7d31943326 Remove some dead code 2013-08-11 21:37:43 +02:00
Norman Maurer
73755d1233 [#1729] Let StringEncoder extend MessageToMessageEncoder and so safe a memory copy 2013-08-11 21:24:02 +02:00
Norman Maurer
9a88c50ffb [#1728] Recycle list after done with it in ReplayingDecoder/ByteToMessageDecoder channelInactive(..) 2013-08-11 21:04:55 +02:00
Norman Maurer
26ddf0849f [#1714] Make sure ByteArrayDecoder also works correctly with pooled buffers 2013-08-09 08:40:43 +02:00
Norman Maurer
1547876e97 Tiny optimizations 2013-08-09 08:24:24 +02:00
Trustin Lee
87b17e195a Grammar 2013-08-05 17:40:45 +09:00
Norman Maurer
a06295fe0a Correctly fix problem in ByteToMessageDecoder and ReplayingDecoder which could let to have a released buffer passed to the decode methods.
This fixes #1664 and revert also the original commit which was meant to fix it 3b1881b523 . The problem with the original commit was that it could delay handlerRemove(..) calls and so mess up the order or forward bytes to late.
2013-08-01 09:54:07 +02:00
Norman Maurer
6ce8571df3 [#1668] Remove synchronized usage in JZlibEncoder and JdkZlibEncoder 2013-07-29 07:08:49 +02:00
Norman Maurer
6278d88d2f [#1656] Fix IndexOutOfBoundsException in MessageToMessageException 2013-07-26 11:46:26 +02:00
Norman Maurer
eb3283c59c [#1613] Allow to specify if direct buffers are prefered in ByteToMessageCodec 2013-07-19 09:39:06 +02:00
Norman Maurer
22be797054 Use Unpooled to allocate buffer in Base64 2013-07-16 07:24:45 +02:00
Norman Maurer
7522d530ac Use the allocator of the src ByteBuf to allocate the new ByteBuf 2013-07-15 10:08:44 +02:00
Norman Maurer
2f7759a264 Remove unused package private classes 2013-07-14 23:50:27 +02:00
Norman Maurer
3140282ae3 Use the ByteBufAllocator when possible 2013-07-14 23:34:36 +02:00
Norman Maurer
ecb215c12f Fix buffer leaks 2013-07-14 22:50:53 +02:00
Norman Maurer
a07abee55f Add and correct javadocs 2013-07-13 19:42:02 +02:00
Norman Maurer
910c5fd594 Hide RecyclableArrayList from sub-classes 2013-07-13 16:57:44 +02:00
Norman Maurer
a8af577423 Add javadocs to methods that user may override and make the others final 2013-07-12 13:49:54 +02:00
Trustin Lee
765d470500 Remove UnreplayableOperationException
- Can't find a reason to keep it.
2013-07-12 20:40:39 +09:00
Norman Maurer
edb6786a5c Make the cumulation field in ByteToMessageDecoder package private to hide it 2013-07-12 13:29:54 +02:00
Norman Maurer
b57d9f307f Allow per-write promises and disallow promises on flush()
- write() now accepts a ChannelPromise and returns ChannelFuture as most
  users expected.  It makes the user's life much easier because it is
  now much easier to get notified when a specific message has been
  written.
- flush() does not create a ChannelPromise nor returns ChannelFuture.
  It is now similar to what read() looks like.
2013-07-11 00:49:48 +09:00
Norman Maurer
da5c6add14 Rename CodecOutput to RecyclableArrayList and move it to internal package.
* Also reuse it in SslHandler
2013-07-10 07:50:26 +02:00
Norman Maurer
768152cf88 Correctly handle forward of remaining data on removal 2013-07-09 22:41:56 +02:00
Trustin Lee
26e9d70457 Remove channelReadSuspended / Rename messageReceived(Last) to channelRead(Complete)
- Remove channelReadSuspended because it's actually same with messageReceivedLast
- Rename messageReceived to channelRead
- Rename messageReceivedLast to channelReadComplete

We renamed messageReceivedLast to channelReadComplete because it
reflects what it really is for.  Also, we renamed messageReceived to
channelRead for consistency in method names.
2013-07-09 23:58:51 +09:00
Trustin Lee
cbd8817905 Remove MessageList from public API and change ChannelInbound/OutboundHandler accordingly
I must admit MesageList was pain in the ass.  Instead of forcing a
handler always loop over the list of messages, this commit splits
messageReceived(ctx, list) into two event handlers:

- messageReceived(ctx, msg)
- mmessageReceivedLast(ctx)

When Netty reads one or more messages, messageReceived(ctx, msg) event
is triggered for each message.  Once the current read operation is
finished, messageReceivedLast() is triggered to tell the handler that
the last messageReceived() was the last message in the current batch.

Similarly, for outbound, write(ctx, list) has been split into two:

- write(ctx, msg)
- flush(ctx, promise)

Instead of writing a list of message with a promise, a user is now
supposed to call write(msg) multiple times and then call flush() to
actually flush the buffered messages.

Please note that write() doesn't have a promise with it.  You must call
flush() to get notified on completion. (or you can use writeAndFlush())

Other changes:

- Because MessageList is completely hidden, codec framework uses
  List<Object> instead of MessageList as an output parameter.
2013-07-09 23:51:48 +09:00
Trustin Lee
65c2a6ed46 Make ByteBuf an abstract class rather than an interface
- 5% improvement in throughput (HelloWorldServer example)
- Made CompositeByteBuf a concrete class (renamed from DefaultCompositeByteBuf) because there's no multiple inheritance in Java

Fixes #1536
2013-07-08 14:59:52 +09:00
Trustin Lee
da5a5af520 Add Test case for truncated SSL packets
- Related issue: #1534
- Also made sure ByteToMessageDecoder does not wrap EncoderException.
2013-07-07 11:52:34 +09:00
Norman Maurer
7dda4b9ce4 [#1532] Remove @deprecated ByteBufIndexFinder and all methods that take it as argument 2013-07-06 20:14:53 +02:00
Trustin Lee
a6bdd7cda4 Fix compilation errors 2013-07-01 17:27:45 +09:00
Norman Maurer
4dab51b619 Remove un-used method parameter 2013-06-28 20:41:10 +02:00
Norman Maurer
e234abdbf7 Add javadocs 2013-06-28 20:28:29 +02:00
Trustin Lee
0dcf352f4c Vastly simplified ByteBufProcessor and MessageListProcessor
- Related: #1378
- They now accept only one argument.
- A user who wants to use a buffer for more complex use cases, he or she can always access the buffer directly via memoryAddress() and array()
2013-06-28 20:29:00 +09:00
Norman Maurer
094d01873b Throw IllegalStateException if implementations of ByteToMessageDecoder and ByteToMessageCodec use @Sharable 2013-06-27 22:38:35 +02:00
Trustin Lee
ac39cad5ff Split ByteBuf.forEachByte() into forEachByte() and forEachByteDesc()
- Related: #1378
- As suggested by @liqweed
2013-06-27 18:48:09 +09:00
Trustin Lee
b5bb36c087 Use (fromIndex, toIndex) instead of (index, length) for ByteBuf.forEachByte(...)
- Related: #1378
2013-06-27 17:30:19 +09:00
Trustin Lee
792edf631c Deprecate ByteBufIndexFinder
- Prefer ByteBufProcessor
- Related: #1378
2013-06-27 14:26:58 +09:00
Trustin Lee
4dd9b6ef2e Add ByteBufProcessor and ByteBuf.forEach(...)
- Fixes #1378
- Needs to provide optimized forEach implementations though.
2013-06-27 13:55:42 +09:00
Trustin Lee
96c5ab62e6 Fix IndexOutOfBoundsException raised by JZlibEncoder.encode()
- Fixes #1471
2013-06-25 11:07:13 +09:00
Norman Maurer
14f2e29af9 [#1450] Fix examples in ReplayDecoder javadocs 2013-06-16 16:53:03 +02:00
Trustin Lee
30bfb989c1 Fix memory leak 2013-06-14 10:44:22 +09:00
Trustin Lee
427d9c4bf2 Fix test failures and reported leaks 2013-06-13 15:18:11 +09:00
Trustin Lee
01d9f10af6 Remove the volatile modifiers where they are unnecessary 2013-06-13 14:43:39 +09:00
Trustin Lee
7eb0f6105d Fix memory leaks 2013-06-13 13:32:47 +09:00
Trustin Lee
175526b6bd Move ReferenceCounted and AbstractReferenceCounted to io.netty.util
- Fixes #1441
- Also move and rename IllegalBufferAccessException to ReferenceCountException
- Prettier reference count exception messages
2013-06-13 13:14:21 +09:00
Trustin Lee
283feda119 Reduce even more garbage by exposing ByteBuf.internalNioBuffer() 2013-06-13 12:40:26 +09:00
Norman Maurer
78c6925921 Clearify when you need to call retain() 2013-06-12 23:13:36 +02:00
Norman Maurer
68c737f0c0 Optimize the way messages are added from one MessageList to another one 2013-06-10 14:07:25 +02:00
Norman Maurer
3be25694d0 Add ChannelHandlerContext.isRemoved() to easily detect the removal of a ChannelHandler while in a method. 2013-06-10 11:16:05 +02:00
Trustin Lee
14158070bf Revamp the core API to reduce memory footprint and consumption
The API changes made so far turned out to increase the memory footprint
and consumption while our intention was actually decreasing them.

Memory consumption issue:

When there are many connections which does not exchange data frequently,
the old Netty 4 API spent a lot more memory than 3 because it always
allocates per-handler buffer for each connection unless otherwise
explicitly stated by a user.  In a usual real world load, a client
doesn't always send requests without pausing, so the idea of having a
buffer whose life cycle if bound to the life cycle of a connection
didn't work as expected.

Memory footprint issue:

The old Netty 4 API decreased overall memory footprint by a great deal
in many cases.  It was mainly because the old Netty 4 API did not
allocate a new buffer and event object for each read.  Instead, it
created a new buffer for each handler in a pipeline.  This works pretty
well as long as the number of handlers in a pipeline is only a few.
However, for a highly modular application with many handlers which
handles connections which lasts for relatively short period, it actually
makes the memory footprint issue much worse.

Changes:

All in all, this is about retaining all the good changes we made in 4 so
far such as better thread model and going back to the way how we dealt
with message events in 3.

To fix the memory consumption/footprint issue mentioned above, we made a
hard decision to break the backward compatibility again with the
following changes:

- Remove MessageBuf
- Merge Buf into ByteBuf
- Merge ChannelInboundByte/MessageHandler and ChannelStateHandler into ChannelInboundHandler
  - Similar changes were made to the adapter classes
- Merge ChannelOutboundByte/MessageHandler and ChannelOperationHandler into ChannelOutboundHandler
  - Similar changes were made to the adapter classes
- Introduce MessageList which is similar to `MessageEvent` in Netty 3
- Replace inboundBufferUpdated(ctx) with messageReceived(ctx, MessageList)
- Replace flush(ctx, promise) with write(ctx, MessageList, promise)
- Remove ByteToByteEncoder/Decoder/Codec
  - Replaced by MessageToByteEncoder<ByteBuf>, ByteToMessageDecoder<ByteBuf>, and ByteMessageCodec<ByteBuf>
- Merge EmbeddedByteChannel and EmbeddedMessageChannel into EmbeddedChannel
- Add SimpleChannelInboundHandler which is sometimes more useful than
  ChannelInboundHandlerAdapter
- Bring back Channel.isWritable() from Netty 3
- Add ChannelInboundHandler.channelWritabilityChanges() event
- Add RecvByteBufAllocator configuration property
  - Similar to ReceiveBufferSizePredictor in Netty 3
  - Some existing configuration properties such as
    DatagramChannelConfig.receivePacketSize is gone now.
- Remove suspend/resumeIntermediaryDeallocation() in ByteBuf

This change would have been impossible without @normanmaurer's help. He
fixed, ported, and improved many parts of the changes.
2013-06-10 16:10:39 +09:00
Luke Wood
c1cbe8bec3 Clarify SnappyFramedDecoder javadoc to make it as clear as possible regarding the default settings 2013-05-20 17:31:04 +01:00
Trustin Lee
e58af1bc94 Readability / Fix inception year 2013-05-17 17:59:13 +09:00
kxbmap
3a5aa12000 Add LengthFieldPrepender.lengthAdjustment parameter 2013-05-17 17:58:53 +09:00
Luke Wood
f841056752 Fix SnappyFramedDecoder checksum validation and add a pair of suitable unit tests 2013-05-16 14:10:54 +01:00
Trustin Lee
80f4c0b334 Merge SnappyChecksumUtil into Snappy and make calculateChecksum() public / Overall clean up 2013-05-09 15:23:56 +09:00
Norman Maurer
108c7d9b44 No need to pass the next BufType to the constructor 2013-05-06 07:46:26 +02:00
Norman Maurer
73c35aef4e [#1309] Make sure ReplayDecoder respect isSingleDecode()
* This could cause for example corrupt WebSocketFrame's if they was written from the server
  to the client directly after it send the handshake response.
2013-04-25 08:32:07 +02:00
Trustin Lee
7884574c7b Remove freeInboundBuffer() and freeOutboundBuffer() which has no value
- Fixes #1308

freeInboundBuffer() and freeOutboundBuffer() were introduced in the early days of the new API when we did not have reference counting mechanism in the buffer. A user did not want Netty to free the handler buffers had to override these methods.

However, now that we have reference counting mechanism built into the buffer, a user who wants to retain the buffers beyond handler's life cycle can simply return the buffer whose reference count is greater than 1 in newInbound/OutboundBuffer().
2013-04-25 09:15:55 +09:00
Trustin Lee
b5989e2449 Reduce exception instantiation overhead in SslHandler / Reduce unnecessary empty array creation
- Added EmptyArrays as an internal utility class
2013-04-24 09:32:53 +09:00
Trustin Lee
8e2e22c270 Change the thread model slightly for new/freeInbound/OutboundBuffer() for future improvement
- Related: #1283
- Make ReplayingDecoder work with the modified thread model
2013-04-23 13:06:27 +09:00
Norman Maurer
ca5554dfe7 [#1236] Fix problem where adding a new ChannelHandler could block the eventloop
This change also introduce a few other changes which was needed:
 * ChannelHandler.beforeAdd(...) and ChannelHandler.beforeRemove(...) were removed
 * ChannelHandler.afterAdd(...) -> handlerAdded(...)
 * ChannelHandler.afterRemoved(...) -> handlerRemoved(...)
 * SslHandler.handshake() -> SslHandler.hanshakeFuture() as the handshake is triggered automatically after
   the Channel becomes active
2013-04-19 07:00:50 +02:00
Trustin Lee
4a5dc32224 Fix a bug where LineBasedFrameDecoder does not handle too long lines correctly
- Related: #1287
2013-04-19 13:04:20 +09:00
Trustin Lee
6dfa455f9e Fix compiler warnings 2013-04-04 16:31:41 +09:00
Trustin Lee
c03179c81c Clean up try/catch blocks / Prefer 'throw' to 'fireExceptionCaught' 2013-04-04 15:00:07 +09:00
Trustin Lee
3b9994455a Saner field order 2013-04-04 14:53:32 +09:00
Trustin Lee
03955428bc Ensure BufUtil.release() is called even if writeBytes() fails in writeAndRelease() 2013-04-04 14:50:25 +09:00
Trustin Lee
015e60b00f Clean up catch blocks
- No need to have nested try blocks because the catch block catches everything and rethrows it
- No need to do instanceof-checks
2013-04-04 14:48:30 +09:00
Trustin Lee
c25fd78ca0 No need to have nested try blocks
.. because the catch block catches everything and rethrows it
2013-04-04 14:44:52 +09:00
Trustin Lee
788b88b7af Make sure to release the buffer when it's written to the next byte buffer
.. otherwise it will leak.
2013-04-04 14:37:54 +09:00
Trustin Lee
46540578fc Optimize OutputMessageBuf.drainToNextInbound/Outbound()
- Return early when the buffer is empty
- Keep only the number of byte buffers
- Remove unnecessary null check in the loop (because we know buffer is not empty at certain point)
2013-04-04 14:32:48 +09:00
Trustin Lee
8a672c2800 Proper assertion instead of blindly clearing the OutputMessageBuf 2013-04-04 14:11:24 +09:00
Norman Maurer
7ee2adb587 Move drain logic to OutputMessageBuf and optimize it as far as possible 2013-04-03 18:52:57 +02:00
Norman Maurer
1675e61f5b Share same ThreadLocal for all decoder/encoders to minimize memory usage 2013-04-03 18:19:35 +02:00
Norman Maurer
94ef7dc1b9 Optimize to minimize volatile reads to access next buffer in codec framework 2013-04-03 18:03:55 +02:00
Norman Maurer
9828267165 Fix performance regression which was caused by calling the fireInboundBufferUpdated() a way to often
* Also use a ThreadLocal for the decoder buffer to safe space as it is cheap
2013-04-03 17:07:52 +02:00
Norman Maurer
af4b71a00e Remove special handling of Object[] in codec framework (a.k.a unfolding)
- Fixes #1229
- Primarily written by @normanmaurer and revised by @trustin

This commit removes the notion of unfolding from the codec framework
completely.  Unfolding was introduced in Netty 3.x to work around the
shortcoming of the codec framework where encode() and decode() did not
allow generating multiple messages.

Such a shortcoming can be fixed by changing the signature of encode()
and decode() instead of introducing an obscure workaround like
unfolding.  Therefore, we changed the signature of them in 4.0.

The change is simple, but backward-incompatible.  encode() and decode()
do not return anything.  Instead, the codec framework will pass a
MessageBuf<Object> so encode() and decode() can add the generated
messages into the MessageBuf.
2013-04-03 21:44:54 +09:00
Norman Maurer
2e3b8f13ea Revert "Let ProtobufEncoder extend MessageToByteEncoder which makes more sense. Related to [#1222]"
This reverts commit 6a9d055f57.
2013-04-02 07:56:49 +02:00
Norman Maurer
6a9d055f57 Let ProtobufEncoder extend MessageToByteEncoder which makes more sense. Related to [#1222] 2013-04-02 07:17:45 +02:00
Norman Maurer
a0fca93128 [#1222] Correctly handle the case when a MessageToMessageEncoder encodes to ByteBuf and the next outbound buffer is of type ByteBuf 2013-04-02 07:17:10 +02:00
Trustin Lee
8b722d29a7 Add constructor parameters that do not perform type parameter auto-detection for the languages without type parameters
- Fixes #1177
- Add TypeParameterMatcher.get(parameterType)
- Add alternative constructors
2013-03-21 16:11:47 +09:00
Trustin Lee
97f2fa7341 Allow a user specify the byte order of the length field for LengthFieldBasedFrameDecoder
- Fixes #1164
2013-03-18 13:58:48 +09:00
Trustin Lee
e3d10ad493 Break the decode loop if decoder raises an exception to give a chance to close the connection to a user handler
- Fixes: #1161
2013-03-16 18:28:58 +09:00
Michael Nitschinger
b186342f52 Fixing 2 typos in codec documentation. 2013-03-16 17:18:21 +09:00
Trustin Lee
289d474a23 Make ByteArrayEncoder sharable 2013-03-14 17:44:32 +09:00
Trustin Lee
42e31a4445 Revert "[#1131] Codecs must not cache next buffer during processing"
This reverts commit b1775a3223.
2013-03-13 15:17:06 +09:00
Trustin Lee
c25513d5e1 Upgrade to protobuf 2.5 and take advantage of MessageLite.getParserFromType()
- also fall back to MessageBuilder if getParserFromType() is not available.
2013-03-12 16:25:35 +09:00
Trustin Lee
559b860ff6 Fix grammar 2013-03-12 15:23:09 +09:00
Trustin Lee
397830d238 Allow SingleOutboundMessageHandler.beginFlush() to reject the flush request by returning false / Replace PartialFlushException with IncompleteFlushException which is more correct. 2013-03-12 15:20:46 +09:00
Trustin Lee
c660002b4e Add DecoderResult.UNFINISHED to represent the case where a decoder generated a message that was not decoded completely / Remove partial failure in DecoderResult which is not very useful but confusing 2013-03-12 13:04:53 +09:00
Norman Maurer
806e9b1f8c Make sure we handle outbound messages of type ByteBuf special 2013-03-11 08:59:00 +09:00
Norman Maurer
b1775a3223 [#1131] Codecs must not cache next buffer during processing 2013-03-08 15:38:17 +01:00
Norman Maurer
fd3f923b52 Allow to specify the used buffer type for ChannelInboundByteBufHandler and ChannelOutboundByteBufHandler by configuration. As default it tries to use a direct ByteBuf 2013-03-08 08:20:46 +01:00
Trustin Lee
a9a29bdf3f Use I/O buffer whenever possible now that our direct buffers are as fast as heap buffers 2013-03-08 11:21:08 +09:00
Trustin Lee
88df53ec1a Fix infinite recursion when transferring data between different type of buffers / Add ByteBuf.hasMemoryAddress/memoryAddress()
- Fixes: #1109 and #1110
2013-03-06 18:22:16 +09:00
Trustin Lee
a8a7c4f576 Provide a way to implement an ChannelInbound/OutboundMessageHandler conveniently without extending an adapter class
- Add ChannelHandlerUtil and move the core logic of ChannelInbound/OutboundMessageHandler to ChannelHandlerUtil
- Add ChannelHandlerUtil.SingleInbound/OutboundMessageHandler and make ChannelInbound/OutboundMessageHandlerAdapter implement them.  This is a backward incompatible change because it forces all handler methods to be public (was protected previously)
- Fixes: #1119
2013-03-05 17:27:53 +09:00
Norman Maurer
42dad6d9d4 [#1106] Also handle FileRegion in ByteToByteEncoder and SslHandler 2013-02-28 19:40:34 +01:00
Trustin Lee
671f9d48d4 Use ConcurrentHashMapV8 wherever possible
- Fixes #1052
2013-02-26 15:54:51 -08:00
Luke Wood
e45db60b5e [#1069] Snappy decoding fixes
* Correct reading offset of 1-byte-offset copies
* Keep track of how much we've written so far in order to validate offsets
* Uncomment and reduce number of tests
2013-02-25 16:36:50 +00:00
Luke Wood
f6d88ca5bf Use Short.MAX_VALUE, as it accurately represents what we're trying to limit ourselves to 2013-02-22 15:23:47 +00:00
Luke Wood
0bfa9159e3 Do not attempt to compress trailing data that is less than the MIN_COMPRESSIBLE_LENGTH 2013-02-22 14:24:50 +00:00
Luke Wood
7a8e9d7993 Ensure that at least 4 bytes are remaining in input buffer when compiling our hash table in search of duplicates, and restrict encoded frame sizes to 32,767 bytes (2^15 - 1) 2013-02-22 14:24:50 +00:00
Atsuhiko Yamanaka
8fdf788cbd [#1012] Replace forked jzlib with official jzlib and add a test. 2013-02-20 12:49:05 +01:00
Trustin Lee
1011227b88 Remove apiviz tags - we are focusing on user guide instead and putting diagrams there 2013-02-14 12:09:16 -08:00
Norman Maurer
6983f704c0 Fix unreplayble error 2013-02-14 11:18:12 +01:00
Norman Maurer
5370573400 Change ReferenceCounted.retain* to return itself and so allow method chaining 2013-02-14 07:39:44 +01:00
Trustin Lee
53c27ef5ae More robust type parameter detection
- Also removed unnecessary constructors which were added due to incomplete type parameter detection logic
2013-02-13 19:02:55 -08:00
Luke Wood
c1783cc8d3 More fixes to snappy: * Make Snappy.decode stateful instead of relying on the uncompressed length being equal to the compressed length * Correctly handle copies where offset < length * Take copies from the output buffer in decoding * Make the maximum encoded chunk size 32kB for compressed data 2013-02-11 15:05:27 +01:00
Trustin Lee
b4f4b95739 Move io.netty.logging to io.netty.internal / Move Signal out of internal because we use it in Channel*MessageAdapters 2013-02-11 20:08:18 +09:00
Trustin Lee
a2e5cd94be Prettify APIviz / Tighten visibility / Move subclasses to top level / Remove unused UnknownSocksMessage 2013-02-11 19:42:23 +09:00
Trustin Lee
6c7bd6d174 Tighten visibility 2013-02-11 18:42:31 +09:00
Trustin Lee
d5105834d6 Hide SnappyChecksumUtil 2013-02-11 18:41:37 +09:00
Trustin Lee
2f1a0b0593 Remove freeInbound/OutboundMessage(), replaced by ReferenceCounted.retain/release()
- Related: #1029
2013-02-10 13:31:31 +09:00
Trustin Lee
b9996908b1 Implement reference counting
- Related: #1029
- Replace Freeable with ReferenceCounted
- Add AbstractReferenceCounted
- Add AbstractReferenceCountedByteBuf
- Add AbstractDerivedByteBuf
- Add EmptyByteBuf
2013-02-10 13:10:09 +09:00
Trustin Lee
bd87284829 Clean up SnappyIntegrationTest / Fix checksum encoding 2013-02-10 00:20:49 +09:00
Trustin Lee
8bd7ea2f93 Fix SnappyFramedEncoderTest & SnappyFramedEncoder
- chunkLength is 3 bytes
2013-02-10 00:03:47 +09:00
Trustin Lee
3268781607 Fix SnappyFramedDecoderTest 2013-02-09 23:57:14 +09:00
Trustin Lee
319b7fa69a Fix SnappyFramedEncoder/Decoder / Fix Snappy preamble encoding / Add test for #1002
- The new test still fails due to a bug in Snappy.encode/decode()
2013-02-09 23:39:33 +09:00
Trustin Lee
36f8630512 Make SnappyFramedDecoder more robust against corrupt frame 2013-02-09 20:58:55 +09:00
Trustin Lee
affd514b8c Rename ByteBufUtil to BufUtil and move ChannelHandlerUtil.freeMessage() there / Remove ChannelHandlerUtil 2013-02-08 23:23:26 +09:00
Trustin Lee
44ea0a116f Replace ChannelHandlerUtil.unfoldAndAdd() with MessageBuf.unfoldAndAdd() / Remove unused methods in ChannelHandlerUtil 2013-02-08 23:07:20 +09:00
Trustin Lee
ff5aec0c78 Replace TypeParameterFinder with TypeParameterMatcher
- We can avoid reflective matching using byte code generation.
 - Better matching performance when message type is Object
2013-02-08 18:28:06 +09:00
Trustin Lee
01e65a01c7 Make ReplayingDecoder.newInboundBuffer/discardInboundReadByte() final for safety 2013-02-08 17:41:43 +09:00
Trustin Lee
82c46180c9 Tighten access modifier of encode/decode() 2013-02-08 17:37:16 +09:00
Trustin Lee
76eb40a4d2 Make ChannelOutboundMessageHandlerAdapter similar to ChannelInboundMessageHandlerAdapter 2013-02-08 17:07:01 +09:00
Trustin Lee
1640b1fea6 Automatically detect the message types in MessageToMessageCodec 2013-02-08 16:12:32 +09:00
Trustin Lee
fa1b49de98 More robust automatic messageType detection for ChannelInboundMessageHandlerAdapter and MessageToMessageDecoder 2013-02-08 15:45:17 +09:00
Trustin Lee
f9eff51683 Fix missing ChannelHandlerUtil.acceptedMessageTypes() in ByteToMessageCodec constructor 2013-02-08 13:24:27 +09:00
Trustin Lee
d8c0bf3be2 Add the 'nextBufferType' parameter to ByteArrayEncoder like did to StringEncoder / Consistent parameter order 2013-02-08 01:36:41 +09:00
Trustin Lee
d4742bbe16 Clean up abstract ChannelHandler impls / Remove ChannelHandlerContext.hasNext*()
- Rename ChannelHandlerAdapter to ChannelDuplexHandler
- Add ChannelHandlerAdapter that implements only ChannelHandler
- Rename CombinedChannelHandler to CombinedChannelDuplexHandler and
  improve runtime validation
- Remove ChannelInbound/OutboundHandlerAdapter which are not useful
- Make ChannelOutboundByteHandlerAdapter similar to
  ChannelInboundByteHandlerAdapter
- Make the tail and head handler of DefaultChannelPipeline accept both
  bytes and messages.  ChannelHandlerContext.hasNext*() were removed
  because they always return true now.
- Removed various unnecessary null checks.
- Correct method/field names:
  inboundBufferSuspended -> channelReadSuspended
2013-02-07 23:47:45 +09:00
Luke Wood
bfe44180f9 Fix SnappyFramedDecoder issues
- Checksum header was being incorrectly read due to incorrect order of
  shift and masking operations.
- Length field of 1-byte copy was being incorrectly interpreted due to a
  typo in the binary mask used to extract it.
- Use ByteBuf.readUnsignedByte() instead of readByte() & 0xff
- Use bitwise-OR wherever possible
- Use EmbeddedByteChannel to test
- Use ByteBuf comparison instead of array comparison
- Work done by @lw346 and then revised by @trustin
2013-02-01 13:10:34 +09:00
Trustin Lee
2ec932798f Replace .readable() and .writable() to .isReadable() and .isWritable() 2013-01-31 18:24:33 +01:00
Trustin Lee
42c65cca3a Make MessageBuf bounded
- Move common methods from ByteBuf to Buf
- Rename ensureWritableBytes() to ensureWritable()
- Rename readable() to isReadable()
- Rename writable() to isWritable()
- Add isReadable(int) and isWritable(int)
- Add AbstractMessageBuf
- Rewrite DefaultMessageBuf and QueueBackedMessageBuf
  - based on Josh Bloch's public domain ArrayDeque impl
2013-01-31 18:11:06 +01:00
Norman Maurer
ec013bf2d3 [#983] Force the user to implement an actual ChannelInboundHandler or ChannelOutboundHandler
For this ChannelInboundHandler* and ChannelOutboundHandler* was made package private
2013-01-31 15:50:27 +01:00
Luke Wood
bd0339ce1a Fix buffer over-run in Snappy codec when emitting a copy that leaves < 4 bytes remaining 2013-01-30 12:15:57 +01:00
Norman Maurer
291293a6dc [#980] Automatically trigger a read operation if isAutoRead() == false but we only had a partial decode 2013-01-30 09:16:45 +01:00
Norman Maurer
d7bfd44e10 [#982] [#977] [#858] Allow to transfer the content a ChannelHandlers inbound/outbound buffer on removal/replacement
This changes the behavior of the ChannelPipeline.remove(..) and ChannelPipeline.replace(..) methods in that way
that after invocation it is not possible anymore to access any data in the inbound or outbound buffer. This is
because it empty it now to prevent side-effects. If a user want to preserve the content and forward it to the
next handler in the pipeline it is adviced to use one of the new methods which where introduced.

 - ChannelPipeline.removeAndForward(..)
 - ChannelPipeline.replaceAndForward(..)
2013-01-28 10:25:38 +01:00
Norman Maurer
b9aaf9a167 [#977] Stop processing messages/bytes in a loop when the handler was removed from the ChannelPipeline 2013-01-23 07:35:44 +01:00
Norman Maurer
f2d84f75d6 [#952] Allow to switch to single message decoding mode on the fly 2013-01-18 09:56:12 +01:00
Trustin Lee
5a82dccbc5 Tighten visibility 2013-01-17 14:43:59 +09:00
Trustin Lee
64ae8b6a37 Replace and merge DetectionUtil and DirectByteBufUtil into PlatformDependent and PlatformDependent0
PlatformDependent delegates the operations requires sun.misc.* to PlatformDependent0 to avoid runtime errors due to missing sun.misc.* classes.
2013-01-11 14:03:27 +09:00
Norman Maurer
b742dcc209 [#902] Remove usage of generics for output of Encoder/Decoder to make them more flexible again 2013-01-09 07:13:31 +01:00
Trustin Lee
5d2e0688ab Fix checkstyle 2013-01-09 15:12:25 +09:00
Trustin Lee
5c2be33cae Delegate to the actual encoder/decoder if possible 2013-01-09 15:09:20 +09:00
Trustin Lee
102563ec8f No need to override ByteToMessageCodec.decoder.discardInboundReadBytes() 2013-01-09 15:05:03 +09:00
Trustin Lee
ef692b0c38 Add missing decodeLast() 2013-01-09 14:58:44 +09:00
Trustin Lee
b58a8f0106 Add missing codec operations in ByteToMessageCodec 2013-01-09 14:56:07 +09:00
Trustin Lee
4a3d73724f Make all encoder/decoder methods overridable in MessageToMessageCodec 2013-01-09 14:44:19 +09:00
Trustin Lee
8d0785da36 Rename MessageToMessageEncoder.freeInboundMessage() to freeOutboundMessage() 2013-01-09 14:35:50 +09:00
Trustin Lee
8d5fd0839f Make ByteToMessageCodec.isEncodable() public like other similar methods 2013-01-09 14:28:42 +09:00
Trustin Lee
aa69d628f1 Add ByteToMessageCodec.isEncodable() 2013-01-09 14:26:15 +09:00
Trustin Lee
dd6b7969b7 Give a handler more control over how its buffers' read bytes are discarded.
This pull request adds two new handler methods: discardInboundReadBytes(ctx) and discardOutboundReadBytes(ctx) to ChannelInboundByteHandler and ChannelOutboundByteHandler respectively. They are called between every inboundBufferUpdated() and flush() respectively. Their default implementation is to call discardSomeReadBytes() on their buffers and a user can override this behavior easily. For example, ReplayingDecoder.discardInboundReadBytes() looks like the following:

    @Override
    public void discardInboundReadBytes(ChannelHandlerContext ctx) throws Exception {
        ByteBuf in = ctx.inboundByteBuffer();
        final int oldReaderIndex = in.readerIndex();
        super.discardInboundReadBytes(ctx);
        final int newReaderIndex = in.readerIndex();
        checkpoint -= oldReaderIndex - newReaderIndex;
    }

If a handler, which has its own buffer index variable, extends ReplayingDecoder or ByteToMessageDecoder, the handler can also override discardInboundReadBytes() and adjust its index variable accordingly.
2013-01-09 13:34:09 +09:00
kxbmap
cf2fbf7883 Fix a link in ZlibWrapper 2013-01-05 21:28:03 +01:00
Norman Maurer
5526153459 [#882] Add a PartialFlushException which will allow to notify the user that the flush/write was only partial succesful 2013-01-05 20:30:48 +01:00
Norman Maurer
ccb5409f58 [#884] Split SCTP transport into extra module 2013-01-03 22:19:06 +01:00
Norman Maurer
37a3f2e3b8 [#887] [#866] [#883] Add unified interface for Message oriented protocols and also use direct buffers for them 2013-01-03 18:15:53 +01:00
Norman Maurer
4e77bacdf7 [#873] [#868] Split ChannelFuture into ChannelFuture and ChannelPromise 2012-12-31 23:27:16 +09:00
Trustin Lee
0909878581 Read only when requested (read-on-demand)
This pull request introduces a new operation called read() that replaces the existing inbound traffic control method. EventLoop now performs socket reads only when the read() operation has been issued. Once the requested read() operation is actually performed, EventLoop triggers an inboundBufferSuspended event that tells the handlers that the requested read() operation has been performed and the inbound traffic has been suspended again. A handler can decide to continue reading or not.

Unlike other outbound operations, read() does not use ChannelFuture at all to avoid GC cost. If there's a good reason to create a new future per read at the GC cost, I'll change this.

This pull request consequently removes the readable property in ChannelHandlerContext, which means how the traffic control works changed significantly.

This pull request also adds a new configuration property ChannelOption.AUTO_READ whose default value is true. If true, Netty will call ctx.read() for you. If you need a close control over when read() is called, you can set it to false.

Another interesting fact is that non-terminal handlers do not really need to call read() at all. Only the last inbound handler will have to call it, and that's just enough. Actually, you don't even need to call it at the last handler in most cases because of the ChannelOption.AUTO_READ mentioned above.

There's no serious backward compatibility issue. If the compiler complains your handler does not implement the read() method, add the following:

public void read(ChannelHandlerContext ctx) throws Exception {
    ctx.read();
}

Note that this pull request certainly makes bounded inbound buffer support very easy, but itself does not add the bounded inbound buffer support.
2012-12-31 23:26:00 +09:00
Norman Maurer
64351ad7d6 [#861] Let SctpMessageCompletionHandler fire inboundMessageBufferUpdated() only if needed 2012-12-27 23:08:54 +01:00
Norman Maurer
a20aba87ab Remove get prefix from Sctp methods to be more inline with the rest 2012-12-27 22:53:42 +01:00
Norman Maurer
852f546b5b [#846] Tighten up visibility 2012-12-25 18:54:55 +01:00
Norman Maurer
fc4b205bc4 More javadocs 2012-12-22 15:53:01 +01:00
Norman Maurer
ff3dcffd49 Almost finish javadocs for codec module 2012-12-21 22:41:18 +01:00
Norman Maurer
b814033323 [#847] Let CompressionException extend CodecException 2012-12-21 22:36:51 +01:00
Norman Maurer
5b44fe6e65 More javadoc fixes 2012-12-21 22:32:16 +01:00
Norman Maurer
8a7bc2c606 Add a lot of javadocs to make usage more clear 2012-12-21 22:22:40 +01:00
Trustin Lee
e353540d47 Move snappy codec to netty-codec 2012-12-19 18:12:18 +09:00
Trustin Lee
def12a171c Rename ChannelBuf to Buf and ChannelBufType to BufType
- Fixes #825
2012-12-17 17:43:45 +09:00
Trustin Lee
03e68482bb Remove ChannelBuf/ByteBuf.Unsafe
- Fixes #826
Unsafe.isFreed(), free(), suspend/resumeIntermediaryAllocations() are not that dangerous. internalNioBuffer() and internalNioBuffers() are dangerous but it seems like nobody is using it even inside Netty. Removing those two methods also removes the necessity to keep Unsafe interface at all.
2012-12-17 17:41:21 +09:00
Trustin Lee
1f72e53af3 Remove redundant copyright headers added by IntelliJ 2012-12-14 12:23:24 +09:00
Trustin Lee
5a4a59406b Merge ByteBuf.hasNioBuffer() and hasNioBuffers()
- Fixes #797
2012-12-14 12:20:33 +09:00
Trustin Lee
8e1a6c6cf5 Add ByteBuf.maxWritableBytes()
- Fixes #806
2012-12-14 11:49:01 +09:00
Norman Maurer
bf8345999c Fix release of buffers in ByteToMessageCodec and MessageToMessageCodec 2012-12-08 12:59:32 +01:00
Shawn Silverman
8be43903e1 Added more Javadocs to the 'replace' methods, see Netty issue 756. 2012-12-07 06:31:21 +01:00
Norman Maurer
038bcfc4ff [#792] Implement freeInboundBuffer(...) and freeOutboundBuffer(...) methods in ByteToByteCodec 2012-12-06 20:57:08 +01:00
Trustin Lee
51e6519b67 Replace UnsafeByteBuf with ByteBuf.unsafe() again
* UnsafeByteBuf is gone. I added ByteBuf.unsafe() back.
* To avoid extra instantiation, all ByteBuf implementations implement the ByteBuf.Unsafe interface.
* To hide this implementation detail, all ByteBuf implementations are package-private.
* AbstractByteBuf and SwappedByteBuf are public and they do not implement ByteBuf.Unsafe because they don't need to.
* unwrap() is not an unsafe operation anymore.
* ChannelBuf also has unsafe() and Unsafe. ByteBuf.Unsafe extends ChannelBuf.unsafe(). ChannelBuf.unsafe() provides free() operation so that a user does not need to down-cast the buffer in freeInbound/OutboundBuffer().
2012-12-05 19:28:56 +09:00
Trustin Lee
33c0c89fef Remove unnecessary empty lines 2012-12-03 19:58:13 +09:00
Trustin Lee
5f9090a7f0 Fix invalid memory access in AIO writes
To perform writes in AioSocketChannel, we get a ByteBuffer view of the
outbound buffer and specify it as a parameter when we call
AsynchronousSocketChannel.write().

In most cases, the write() operation is finished immediately.  However,
sometimes, it is scheduled for later execution.  In such a case, there's
a chance for a user's handler to append more data to the outbound
buffer.

When more data is appended to the outbound buffer, the outbound buffer
can expand its capacity by itself.  Changing the capacity of a buffer is
basically made of the following steps:

1. Allocate a larger new internal memory region.
2. Copy the current content of the buffer to the new memory region.
3. Rewire the buffer so that it refers to the new region.
4. Deallocate the old memory region.

Because the old memory region is deallocated at the step 4, the write
operation scheduled later will access the deallocated region, leading
all sort of data corruption or even segfaults.

To prevent this situation, I added suspendIntermediaryDeallocations()
and resumeIntermediaryDeallocations() to UnsafeByteBuf.

AioSocketChannel.doFlushByteBuf() now calls suspendIntermediaryDealloc()
to defer the deallocation of the old memory regions until the completion
handler is notified.
2012-12-02 21:50:33 +09:00
Trustin Lee
bfe2a96505 Fix AssertionError from AsyncSocketChannel.beginRead()
An AssertionError is triggered by a ByteBuf when beginRead() attempts to
access the buffer which has been freed already.  This commit ensures the
buffer is not freed before performing an I/O operation.

To determine if the buffer has been freed, UnsafeByteBuf.isFreed() has
been added.
2012-12-02 20:17:53 +09:00
Trustin Lee
00c4b944e4 Fix more inspector warnings introduced by recent mergences 2012-12-01 00:10:42 +09:00
Trustin Lee
6208c62888 Fix inspector warnings introduced by recent mergences 2012-11-30 23:01:57 +09:00
Trustin Lee
089d022e03 Remove redundant exception message 2012-11-26 15:34:09 +09:00
Trustin Lee
40373526ce Remove unexpected bad text
Doh!
2012-11-26 14:41:12 +09:00
Trustin Lee
5b68f00382 Make CompatibleMarshallingDecoder discard the inbound buffer once TooLongFrameException is raised
(See #768)

Once too long object is received, CompatibleMarshallingDecoder has to
discard all input from now on, just like MarshallingDecoder does.
Otherwise, the decoder will raise more exceptions because the decoder
has no idea anymore where the object starts.

Before this fix, SerialThreadLocalCompatibleMarshallingDecoderTest
logged many additional exceptions raised by the decoder after test is
finished.
2012-11-26 14:40:18 +09:00
Norman Maurer
d7d8503f5c Add a line-based frame decoder with good performance.
Using DelimiterBasedFrameDecoder with Delimiters.lineDelimiter() has
quadratic performance in the size of the input buffer.  Needless to
say, the performance degrades pretty quickly as the size of the buffer
increases.  Larger MTUs or loopback connections can make it so bad that
it appears that the code is "busy waiting", when in fact it's spending
almost 100% of the CPU time in DelimiterBasedFrameDecoder.indexOf().

Add a new LineBasedFrameDecoder that decodes line-delimited frames
in O(n) instead of DelimiterBasedFrameDecoder's O(n^2) implementation.
In OpenTSDB's telnet-style protocol decoder this resulted in throughput
increases of an order of magnitude.

Change DelimiterBasedFrameDecoder to automatically detect when the
frames are delimited by line endings, and automatically switch to
using LineBasedFrameDecoder under the hood.  This means that all Netty
applications out there that using the combo DelimiterBasedFrameDecoder
with Delimiters.lineDelimiter() will automatically benefit from the
better performance of LineBasedFrameDecoder, without requiring a code
change.
2012-11-25 13:36:55 +01:00
Trustin Lee
81e2db10fa ByteBufAllocator API w/ ByteBuf perf improvements
This commit introduces a new API for ByteBuf allocation which fixes
issue #643 along with refactoring of ByteBuf for simplicity and better
performance. (see #62)

A user can configure the ByteBufAllocator of a Channel via
ChannelOption.ALLOCATOR or ChannelConfig.get/setAllocator().  The
default allocator is currently UnpooledByteBufAllocator.HEAP_BY_DEFAULT.

To allocate a buffer, do not use Unpooled anymore. do the following:

  ctx.alloc().buffer(...); // allocator chooses the buffer type.
  ctx.alloc().heapBuffer(...);
  ctx.alloc().directBuffer(...);

To deallocate a buffer, use the unsafe free() operation:

  ((UnsafeByteBuf) buf).free();

The following is the list of the relevant changes:

- Add ChannelInboundHandler.freeInboundBuffer() and
  ChannelOutboundHandler.freeOutboundBuffer() to let a user free the
  buffer he or she allocated. ChannelHandler adapter classes implement
  is already, so most users won't need to call free() by themselves.
  freeIn/OutboundBuffer() methods are invoked when a Channel is closed
  and deregistered.

- All ByteBuf by contract must implement UnsafeByteBuf. To access an
  unsafe operation: ((UnsafeByteBuf) buf).internalNioBuffer()

- Replace WrappedByteBuf and ByteBuf.Unsafe with UnsafeByteBuf to
  simplify overall class hierarchy and to avoid unnecesary instantiation
  of Unsafe instances on an unsafe operation.

- Remove buffer reference counting which is confusing

- Instantiate SwappedByteBuf lazily to avoid instantiation cost

- Rename ChannelFutureFactory to ChannelPropertyAccess and move common
  methods between Channel and ChannelHandlerContext there. Also made it
  package-private to hide it from a user.

- Remove unused unsafe operations such as newBuffer()

- Add DetectionUtil.canFreeDirectBuffer() so that an allocator decides
  which buffer type to use safely
2012-11-22 15:10:59 +09:00
Trustin Lee
aedf8790c3 Fix various Javadoc issues / Do not use argN parameter names 2012-11-12 12:26:19 +09:00
Trustin Lee
ea4a0e3535 Prefer {@code ...} to <code>...</code> / Fix deprecation warnings 2012-11-12 11:51:23 +09:00
Trustin Lee
aa7cd691df Remove redundant 'else' branches. 2012-11-12 09:31:40 +09:00
Trustin Lee
b4f796c5e3 Use 'x' over "x" wherever possible / String.equals("") -> isEmpty() 2012-11-10 08:03:52 +09:00
Trustin Lee
a9cfde97f0 Add missing final modifiers 2012-11-10 02:19:11 +09:00
Trustin Lee
957154c005 Remove redundant no-arg constructors 2012-11-10 01:08:18 +09:00
Norman Maurer
985fa97c9b Check if message is supported before cast. See #678 2012-10-24 07:03:02 +02:00
Norman Maurer
cbcabaf29b Add support for method chaining to ByteBuf 2012-10-18 08:57:23 +02:00
Trustin Lee
b76c39318d Make NoSuchBufferException message more useful 2012-10-16 14:23:02 -07:00
Norman Maurer
6143626427 Cleanup of sctp code based on comments in #633 2012-10-12 07:45:00 +02:00
Jestan Nirojan
6904b62c2f Resolved conflicts in SctpData/SctpMessage refactoring 2012-09-29 02:01:00 +08:00
Jestan Nirojan
64ebece730 Added SCTP Codec 2012-09-29 01:42:21 +08:00
Trustin Lee
94838ee274 [#441] Provide a better way to handle decoder failures
* Rename isPartial() to isPartialFailure()
* Add isCompleteFailure() and isFailure()
2012-09-28 15:46:17 +09:00