Commit Graph

609 Commits

Author SHA1 Message Date
Norman Maurer
0b1c82b254 Ensure read components can be discarded when COMPOSITE_CUMULATOR is used.
Motivation:

ByteToMessageDecoder must ensure that read components of the CompositeByteBuf can be discard by default when discardSomeReadBytes() is called. This may not be the case before as because of the default maxNumComponents that will cause consolidation.

Modifications:

Ensure we not do any consolidation to actually be abel to discard read components

Result:

Less memory usage and allocations.
2016-02-17 19:51:08 -08:00
Xiaoyan Lin
ccb0870600 Add methods with position independent FileChannel calls to ByteBuf
Motivation

See ##3229

Modifications:

Add methods with position independent FileChannel calls to ByteBuf and its subclasses.

Results:

The user can use these new methods to read/write ByteBuff without updating FileChannel's position.
2016-02-14 20:37:37 -08:00
Scott Mitchell
b112673554 ByteToMessageDecoder ChannelInputShutdownEvent support
Motivation:
b714297a44 introduced ChannelInputShutdownEvent support for HttpObjectDecoder. However this should have been added to the super class ByteToMessageDecoder, and ByteToMessageDecoder should not propegate a channelInactive event through the pipeline in this case.

Modifications:
- Move the ChannelInputShutdownEvent handling from HttpObjectDecoder to ByteToMessageDecoder
- ByteToMessageDecoder doesn't call ctx.fireChannelInactive() on ChannelInputShutdownEvent

Result:
Half closed events are treated more generically, and don't get translated into a channelInactive pipeline event.
2016-02-12 16:15:17 -08:00
Norman Maurer
65b3470456 [#4793] Correctly add newlines when encode base64
Motivation:

We not correctly added newlines if the src data needed to be padded. This regression was introduced by '63426fc3ed083513c07a58b45381f5c10dd47061'

Modifications:

- Correctly handling newlines
- Add unit test that proves the fix.

Result:

No more invalid base64 encoded data.
2016-02-06 09:56:21 +01:00
Norman Maurer
75a2ddd61c [maven-release-plugin] prepare for next development iteration 2016-02-04 16:51:44 +01:00
Norman Maurer
7eb3a60dba [maven-release-plugin] prepare release netty-4.1.0.CR2 2016-02-04 16:37:06 +01:00
Norman Maurer
a0758e7e60 [#4794] Support window size flag by default if ZlibCodecFactory supports it.
Motivation:

If the ZlibCodecFactory can support using a custom window size we should support it by default in the websocket extensions as well.

Modifications:

Detect if a custom window size can be handled by the ZlibCodecFactory and if so enable it by default for PerMessageDeflate*ExtensionHandshaker.

Result:

Support window size flag by default in most installations.
2016-02-04 14:01:40 +01:00
Norman Maurer
1c417e5f82 [maven-release-plugin] prepare for next development iteration 2016-01-21 15:35:55 +01:00
Norman Maurer
c681a40a78 [maven-release-plugin] prepare release netty-4.1.0.CR1 2016-01-21 15:28:21 +01:00
Xiaoyan Lin
d1ef33b8f4 Change 64 to 63 in Snappy.decodeLiteral
Motivation:

According to https://github.com/google/snappy/blob/master/format_description.txt#L55 , Snappy.decodeLiteral should handle the cases of 60, 61, 62 and 63. However right now it processes 64 instead of 63. I believe it's a typo since `tag >> 2 & 0x3F` must be less than 64.

Modifications:

Use the correct value 63.

Result:

Snappy.decodeLiteral handles the correct case.
2016-01-21 09:59:10 +01:00
Robert Borg
3785ca9311 added support for Protobuf codec nano runtime
Motivation:

Netty was missing support for Protobuf nano runtime targeted at
weaker systems such as Android devices.

Modifications:

Added ProtobufDecoderNano and ProtobufDecoderNano
in order to provide support for Nano runtime.

modified ProtobufVarint32FrameDecoder and
ProtobufLengthFieldPrepender in order to remove any
on either Nano or Lite runtime by copying the code
for handling Protobuf varint32 in from Protobuf
library.

modified Licenses and NOTICE in order to reflect the
changes i made.

added Protobuf Nano runtime as optional dependency

Result:

Netty now supports Protobuf Nano runtime.
2016-01-19 21:39:17 +01:00
Stephane Landelle
f7c83c8565 Don't enforce JZlibDecoder on JDK7+, close #4707
Motivation:

`JdkZlibDecoder` is available since Netty 4.0.8 and works with JDK7+.
However, `io.netty.noJdkZlibDecoder` System prop evaluation always defaults to
true, causing Netty to always use JZLib when decompressing on the
client side when the property insn't explictly set to `false`.

Modifications:

Default to `false` instead of `true` when JDK7+.

Result:

JZLib optional as expected on JDK7+.
2016-01-14 21:04:15 +01:00
Norman Maurer
4cdbe39284 [#4635] Stop decoding if decoder was removed
Motivation:

We need to check if this handler was removed before continuing with decoding.
If it was removed, it is not safe to continue to operate on the buffer.

Modifications:

Check if decoder was removed after fire messages through the pipeline.

Result:

No illegal buffer access when decoder was removed.
2016-01-05 11:01:24 +01:00
Xiaoyan Lin
a96d52fe66 Fix javadoc links and tags
Motivation:

There are some wrong links and tags in javadoc.

Modifications:

Fix the wrong links and tags in javadoc.

Result:

These links will work correctly in javadoc.
2015-12-26 08:34:31 +01:00
Norman Maurer
693633eeff Not use Unpooled to allocate buffers in Base64 but use a ByteBufAllocator
Motivation:

We should not use Unpooled to allocate buffers for performance reasons.

Modifications:

Allow to pass in ByteBufAllocate which is used to allocate buffers or use the allocate of the src buffer.

Result:

Better performance if the PooledByteBufAllocator is used.
2015-12-18 21:22:34 +01:00
Norman Maurer
63426fc3ed Prevent adding newline if Base64 buffer encoded ends directly on MAX_LINE_LENGTH
Motivation:

We need to ensure we not add a newline if the Base64 encoded buffer ends directly on the MAX_LINE_LENGTH. If we miss to do so this produce invalid data.
Because of this bug OpenSslServerContext and OpenSslClientContext may fail to load a cert.

Modifications:

- Only add NEW_LINE if we not are on the end of the dst buffer.
- Add unit test

Result:

Correct result in all cases
2015-12-18 20:57:59 +01:00
Norman Maurer
dc615ecaaf [#4212] Backport WebSocket Extension handlers for client and server.
Motivation:

We have websocket extension support (with compression) in old master. We should port this to 4.1

Modifications:

Backport relevant code.

Result:

websocket extension support (with compression) is now in 4.1.
2015-12-18 09:48:10 +01:00
Norman Maurer
d67fea606b Use ByteBuf.*LE methods for write and read LE
Motivation:

We recently added methods to ByteBuf to directly write and read LE values. We should use these in the Snappy implementation and so reduce duplication.

Modifications:

Replace manually swapping of values with LE write and read methods of ByteBuf.

Result:

Cleaner code with less duplication.
2015-12-10 09:34:27 +01:00
Alex Petrov
0f9492c9af Add first-class Little Endian support to ByteBuf and descendants
As discussed in	#3209, this PR adds Little Endian accessors
to ByteBuf and descendants.

Corresponding accessors were added to UnsafeByteBufUtil,
HeapByteBufferUtil to avoid calling `reverseBytes`.

Deprecate `order()`, `order(buf)` and `SwappedByteBuf`.
2015-11-26 20:30:24 +01:00
Scott Mitchell
cfcee5798d Adjustable size of DefaultHeaders array
Motivation:
DefaultHeaders creates an array of size 16 for all headers. This may waste a good deal of memory if applications only have a small number of headers. This memory may be critical when the number of connections grows large.

Modifications:
- Make the size of the array for DefaultHeaders configurable

Result:
Applications can control the size of the DefaultHeaders array and save memory.
2015-11-23 15:38:08 -08:00
Norman Maurer
7bee318fc7 Use OneTimeTask where possible to reduce object creation
Motivation:

We should use OneTimeTask where possible to reduce object creation.

Modifications:

Replace Runnable with OneTimeTask

Result:

Less object creation
2015-11-20 14:39:06 -08:00
Norman Maurer
2ecce8fa56 [maven-release-plugin] prepare for next development iteration 2015-11-10 22:59:33 +01:00
Norman Maurer
6a93f331d3 [maven-release-plugin] prepare release netty-4.1.0.Beta8 2015-11-10 22:50:57 +01:00
Louis Ryan
6e108cb96a Improve the performance of copying header sets when hashing and name validation are equivalent.
Motivation:
Headers and groups of headers are frequently copied and the current mechanism is slower than it needs to be.

Modifications:
Skip name validation and hash computation when they are not necessary.
Fix emergent bug in CombinedHttpHeaders identified with better testing
Fix memory leak in DefaultHttp2Headers when clearing
Added benchmarks

Result:
Faster header copying and some collateral bug fixes
2015-11-07 08:53:10 -08:00
Louis Ryan
3eb65797ed Make headers.set(self) a no-op instead of throwing. Makes it consistent with setAll
Motivation:

Makes the API contract of headers more consistent and simpler.

Modifications:

If self is passed to set then simply return

Result:

set and setAll will be consistent
2015-11-06 07:00:54 -08:00
Scott Mitchell
19658e9cd8 HTTP/2 Headers Type Updates
Motivation:
The HTTP/2 RFC (https://tools.ietf.org/html/rfc7540#section-8.1.2) indicates that header names consist of ASCII characters. We currently use ByteString to represent HTTP/2 header names. The HTTP/2 RFC (https://tools.ietf.org/html/rfc7540#section-10.3) also eludes to header values inheriting the same validity characteristics as HTTP/1.x. Using AsciiString for the value type of HTTP/2 headers would allow for re-use of predefined HTTP/1.x values, and make comparisons more intuitive. The Headers<T> interface could also be expanded to allow for easier use of header types which do not have the same Key and Value type.

Motivation:
- Change Headers<T> to Headers<K, V>
- Change Http2Headers<ByteString> to Http2Headers<CharSequence, CharSequence>
- Remove ByteString. Having AsciiString extend ByteString complicates equality comparisons when the hash code algorithm is no longer shared.

Result:
Http2Header types are more representative of the HTTP/2 RFC, and relationship between HTTP/2 header name/values more directly relates to HTTP/1.x header names/values.
2015-10-30 15:29:44 -07:00
Norman Maurer
11e8163aa9 [#4284] Forward decoded messages more frequently
Motivation:

At the moment we only forward decoded messages that were added the out List once the full decode loop was completed. This has the affect that resources may not be released as fast as possible and as an application may incounter higher latency if the user triggeres a writeAndFlush(...) as a result of the decoded messages.

Modifications:

- forward decoded messages after each decode call

Result:

Forwarding decoded messages through the pipeline in a more eager fashion.
2015-10-07 14:15:53 +02:00
Norman Maurer
2ff2806ada [maven-release-plugin] prepare for next development iteration 2015-10-02 09:03:29 +02:00
Norman Maurer
5a43de10f7 [maven-release-plugin] prepare release netty-4.1.0.Beta7 2015-10-02 09:02:58 +02:00
Norman Maurer
ca44436ce6 [#4265] Not allow to add/set DefaultHttpHeaders to itself.
Motivation:

We should prevent to add/set DefaultHttpHeaders to itself to prevent unexpected side-effects.

Modifications:

Throw IllegalArgumentException if user tries to pass the same instance to set/add.

Result:

No surprising side-effects.
2015-09-30 08:57:53 +02:00
Norman Maurer
c8a941d01e [#4275] Discard bytes after X reads to guard against OOME.
Motivation:

If a remote peer writes fast enough it may take a long time to have fireChannelReadComplete(...) triggered. Because of this we need to take special care and ensure we try to discard some bytes if channelRead(...) is called to often in ByteToMessageDecoder.

Modifications:

- Add ByteToMessageDecoder.setDiscardAfterReads(...) which allows to set the number of reads after which we try to discard the read bytes
- Use default value of 16 for max reads.

Result:

No risk of OOME.
2015-09-29 12:02:39 +02:00
Scott Mitchell
7adc1f9eb4 STOMP Headers Name Comparator
Motivation:
The HashingStrategy for DefaultStompHeaders was using the java .equals() method which would fail to compare String, AsciiString, and other CharSequence objects as equal.

Modification:
- Use AsciiString.CASE_SENSITIVE_HASHER for DefaultStompHeaders

Result:
DefaultStompHeaders work with all CharSequence objects.
Fixes https://github.com/netty/netty/issues/4247
2015-09-23 16:47:09 -07:00
Scott Mitchell
47726991b2 HTTP/2 Header Name Validation
Motivation:
The HTTP/2 header name validation was removed, and does not currently exist.

Modifications:
- Header name validation for HTTP/2 should be restored and set to the default mode of operation.

Result:
HTTP/2 header names are validated according to https://tools.ietf.org/html/rfc7540
2015-09-09 13:59:08 -07:00
Norman Maurer
34de2667c7 [maven-release-plugin] prepare for next development iteration 2015-09-02 11:45:20 +02:00
Norman Maurer
2eb444ec1d [maven-release-plugin] prepare release netty-4.1.0.Beta6 2015-09-02 11:36:11 +02:00
Norman Maurer
1fefe9affb [#4087] Correctly forward bytes when remove codec and handle channelInactive / channelReadComplete(...)
Motivation:

We missed to correctly implement the handlerRemoved(...) / channelInactive(...) and channelReadComplete(...) method, this leaded to multiple problems:

 - Missed to forward bytes when the codec is removed from the pipeline
 - Missed to call decodeLast(...) once the Channel goes in active
 - No correct handling of channelReadComplete that could lead to grow of cumulation buffer.

Modifications:

- Correctly implement methods and forward to the internal ByteToMessageDecoder
- Add unit test.

Result:

Correct behaviour
2015-08-21 18:26:32 +02:00
Scott Mitchell
a7135e8677 HttpObjectAggregator doesn't check content-length header
Motivation:
The HttpObjectAggregator always responds with a 100-continue response. It should check the Content-Length header to see if the content length is OK, and if not responds with a 417.

Modifications:
- HttpObjectAggregator checks the Content-Length header in the case of a 100-continue.

Result:
HttpObjectAggregator responds with 417 if content is known to be too big.
2015-08-17 09:26:50 -07:00
Scott Mitchell
ba6ce5449e Headers Performance Boost and Interface Simplification
Motivation:
A degradation in performance has been observed from the 4.0 branch as documented in https://github.com/netty/netty/issues/3962.

Modifications:
- Simplify Headers class hierarchy.
- Restore the DefaultHeaders to be based upon DefaultHttpHeaders from 4.0.
- Make various other modifications that are causing hot spots.

Result:
Performance is now on par with 4.0.
2015-08-17 08:50:11 -07:00
Jakob Buchgraber
6fd0a0c55f Faster and more memory efficient headers for HTTP, HTTP/2, STOMP and SPYD. Fixes #3600
Motivation:

We noticed that the headers implementation in Netty for HTTP/2 uses quite a lot of memory
and that also at least the performance of randomly accessing a header is quite poor. The main
concern however was memory usage, as profiling has shown that a DefaultHttp2Headers
not only use a lot of memory it also wastes a lot due to the underlying hashmaps having
to be resized potentially several times as new headers are being inserted.

This is tracked as issue #3600.

Modifications:
We redesigned the DefaultHeaders to simply take a Map object in its constructor and
reimplemented the class using only the Map primitives. That way the implementation
is very concise and hopefully easy to understand and it allows each concrete headers
implementation to provide its own map or to even use a different headers implementation
for processing requests and writing responses i.e. incoming headers need to provide
fast random access while outgoing headers need fast insertion and fast iteration. The
new implementation can support this with hardly any code changes. It also comes
with the advantage that if the Netty project decides to add a third party collections library
as a dependency, one can simply plug in one of those very fast and memory efficient map
implementations and get faster and smaller headers for free.

For now, we are using the JDK's TreeMap for HTTP and HTTP/2 default headers.

Result:

- Significantly fewer lines of code in the implementation. While the total commit is still
  roughly 400 lines less, the actual implementation is a lot less. I just added some more
  tests and microbenchmarks.

- Overall performance is up. The current implementation should be significantly faster
  for insertion and retrieval. However, it is slower when it comes to iteration. There is simply
  no way a TreeMap can have the same iteration performance as a linked list (as used in the
  current headers implementation). That's totally fine though, because when looking at the
  benchmark results @ejona86 pointed out that the performance of the headers is completely
  dominated by insertion, that is insertion is so significantly faster in the new implementation
  that it does make up for several times the iteration speed. You can't iterate what you haven't
  inserted. I am demonstrating that in this spreadsheet [1]. (Actually, iteration performance is
  only down for HTTP, it's significantly improved for HTTP/2).

- Memory is down. The implementation with TreeMap uses on avg ~30% less memory. It also does not
  produce any garbage while being resized. In load tests for GRPC we have seen a memory reduction
  of up to 1.2KB per RPC. I summarized the memory improvements in this spreadsheet [1]. The data
  was generated by [2] using JOL.

- While it was my original intend to only improve the memory usage for HTTP/2, it should be similarly
  improved for HTTP, SPDY and STOMP as they all share a common implementation.

[1] https://docs.google.com/spreadsheets/d/1ck3RQklyzEcCLlyJoqDXPCWRGVUuS-ArZf0etSXLVDQ/edit#gid=0
[2] https://gist.github.com/buchgr/4458a8bdb51dd58c82b4
2015-08-04 17:12:24 -07:00
tczerwinski
b33c7b12a4 XmlFrameDecoder is corrupt
Motivation:

Two problems:
1. Decoder assumption that as soon as it finds </ element it can decrement opened xml brackets counter. It can lead to bugs when closing bracket is not in byteBuf yet.
2. Not proper handling of more than two root elements in XML document. First element will be processed properly, second one not. It is caused by assumption that byteBuf readerIndex is 0 at the begging of decoding.

Modifications:

Both problems were resolved by fixes:
1. decrement opened brackets count only if </ > enclosing bracket is found
2. consider readerIndex higher than 0 when counting output frame length

Result:

Both problems were resolved
2015-07-29 18:49:26 +02:00
Norman Maurer
c53dbb748e Add ProtocolDetectionResult and use it in HAProxyMessageDecoder for allow detect HAProxy protocol.
Motivation:

Sometimes it is useful to detect if a ByteBuf contains a HAProxy header, for example if you want to write something like the PortUnification example.

Modifications:

- Add ProtocolDetectionResult which can be used as a return type for detecting different protocol.
- Add new method which allows to detect HA Proxy messages.

Result:

Easier to detect protocol.
2015-06-23 08:59:07 +02:00
Trustin Lee
950da2eae1 Allow MessageAggregator to disallow non-empty content
Motivation:

A user sometimes just want the aggregated message has no content at
all. (e.g. A user only wants HTTP GET requests.)

Modifications:

- Do not raise IllegalArgumentException even if a user specified
  the maxContentLength of 0

Result:

A user can disallow a message with non-empty content.
2015-06-10 12:06:27 +09:00
Norman Maurer
f23b7b4efd [maven-release-plugin] prepare for next development iteration 2015-05-07 14:21:08 -04:00
Norman Maurer
871ce43b1f [maven-release-plugin] prepare release netty-4.1.0.Beta5 2015-05-07 14:20:38 -04:00
Norman Maurer
891be30a28 Remove memory copy when extract frame in LengthFieldBasedFrameDecoder
Motivation:

We are currently doing a memory cop to extract the frame in LengthFieldBasedFrameDecoder which can be eliminated.

Modifications:

Use buffer.slice(...).retain() to eliminate the memory copy.

Result:

Better performance.
2015-04-29 08:43:06 +02:00
Alwayswithme
abccf18411 fix the discardedBytes counting on LineBasedFrameDecoder
Motivation:

The LineBasedFrameDecoder discardedBytes counting different compare to
DelimiterBasedFrameDecoder.

Modifications:

Add plus sign

Result:

DiscardedBytes counting correctly
2015-04-21 09:25:47 +02:00
Norman Maurer
62057f73d6 Fix handling of non-auto read for ByteToMessageDecoder and SslHandler
Motivation:

Our automatically handling of non-auto-read failed because it not detected the need of calling read again by itself if nothing was decoded. Beside this handling of non-auto-read never worked for SslHandler as it always triggered a read even if it decoded a message and auto-read was false.

This fixes [#3529] and [#3587].

Modifications:

- Implement handling of calling read when nothing was decoded (with non-auto-read) to ByteToMessageDecoder again
- Correctly respect non-auto-read by SslHandler

Result:

No more stales and correctly respecting of non-auto-read by SslHandler.
2015-04-20 09:11:02 +02:00
Norman Maurer
ccde870b38 Revert "Ensure channelReadComplete() is called only when necessary"
This reverts commit 27a25e29f7.
2015-04-20 09:10:41 +02:00
Norman Maurer
bb692816d2 Revert "Do not suppress channelReadComplete() when a handler was just added"
This reverts commit 720faa4df1.
2015-04-20 09:10:29 +02:00
Scott Mitchell
9a7a85dbe5 ByteString introduced as AsciiString super class
Motivation:
The usage and code within AsciiString has exceeded the original design scope for this class. Its usage as a binary string is confusing and on the verge of violating interface assumptions in some spots.

Modifications:
- ByteString will be created as a base class to AsciiString. All of the generic byte handling processing will live in ByteString and all the special character encoding will live in AsciiString.

Results:
The AsciiString interface will be clarified. Users of AsciiString can now be clear of the limitations the class imposes while users of the ByteString class don't have to live with those limitations.
2015-04-14 16:35:17 -07:00