Commit Graph

616 Commits

Author SHA1 Message Date
Xiaoyan Lin
01835fdf18 Add LineEncoder to append a line separator automatically
Motivation:

See #1811

Modifications:

Add LineEncoder and LineSeparator

Result:

The user can use LineEncoder to write a String with a line separator automatically
2016-03-16 20:31:01 +01:00
Xiaoyan Lin
4fb585965c Add DatagramPacketEncoder and DatagramPacketDecoder
Motivation:

UDP-oriented codec reusing the existing encoders and decoders would be helpful. See #1350

Modifications:

Add DatagramPacketEncoder and DatagramPacketDecoder to reuse the existing encoders and decoders.

Result:

People can use DatagramPacketEncoder and DatagramPacketDecoder to wrap existing encoders and decoders to create UDP-oriented codec.
2016-03-14 12:14:57 +01:00
Norman Maurer
9aac6dac2e [#4386] ByteToMessage.decodeLast(...) should not call decode(...) if buffer is empty.
Motivation:

If the input buffer is empty we should not have decodeLast(...) call decode(...) as the user may not expect this.

Modifications:

- Not call decode(...) in decodeLast(...) if the input buffer is empty.
- Add testcases.

Result:

decodeLast(...) will not call decode(...) if input buffer is empty.
2016-03-01 08:42:26 +01:00
Sergey Polovko
a2e113b987 Use ByteProcessor.FIND_LF to find end of line
Motivation:

Each call of ByteBuf.getByte(int) method does boundary checking. This can be eliminated by using ByteBuf.forEachByte(ByteProcessor) method and ByteProcessor.FIND_LF processor.

Modifications:

Find end of line with ByteProcessor.FIND_LF

Result:

A little better performance of LineBasedFrameDecoder.
2016-02-29 09:38:36 +01:00
Norman Maurer
0bea10b0b0 [#4892] Make Snappy class public
Motivation:

Some people may want to use the Snappy class directly to encode / decode ByteBufs.

Modifications:

Make the Snappy class public and final.

Result:

Easier for people to reuse parts of Netty.
2016-02-26 12:15:10 +01:00
Norman Maurer
ca443e42e0 [maven-release-plugin] prepare for next development iteration 2016-02-19 23:00:11 +01:00
Norman Maurer
f39eb9a6b2 [maven-release-plugin] prepare release netty-4.1.0.CR3 2016-02-19 22:59:52 +01:00
Norman Maurer
0b1c82b254 Ensure read components can be discarded when COMPOSITE_CUMULATOR is used.
Motivation:

ByteToMessageDecoder must ensure that read components of the CompositeByteBuf can be discard by default when discardSomeReadBytes() is called. This may not be the case before as because of the default maxNumComponents that will cause consolidation.

Modifications:

Ensure we not do any consolidation to actually be abel to discard read components

Result:

Less memory usage and allocations.
2016-02-17 19:51:08 -08:00
Xiaoyan Lin
ccb0870600 Add methods with position independent FileChannel calls to ByteBuf
Motivation

See ##3229

Modifications:

Add methods with position independent FileChannel calls to ByteBuf and its subclasses.

Results:

The user can use these new methods to read/write ByteBuff without updating FileChannel's position.
2016-02-14 20:37:37 -08:00
Scott Mitchell
b112673554 ByteToMessageDecoder ChannelInputShutdownEvent support
Motivation:
b714297a44 introduced ChannelInputShutdownEvent support for HttpObjectDecoder. However this should have been added to the super class ByteToMessageDecoder, and ByteToMessageDecoder should not propegate a channelInactive event through the pipeline in this case.

Modifications:
- Move the ChannelInputShutdownEvent handling from HttpObjectDecoder to ByteToMessageDecoder
- ByteToMessageDecoder doesn't call ctx.fireChannelInactive() on ChannelInputShutdownEvent

Result:
Half closed events are treated more generically, and don't get translated into a channelInactive pipeline event.
2016-02-12 16:15:17 -08:00
Norman Maurer
65b3470456 [#4793] Correctly add newlines when encode base64
Motivation:

We not correctly added newlines if the src data needed to be padded. This regression was introduced by '63426fc3ed083513c07a58b45381f5c10dd47061'

Modifications:

- Correctly handling newlines
- Add unit test that proves the fix.

Result:

No more invalid base64 encoded data.
2016-02-06 09:56:21 +01:00
Norman Maurer
75a2ddd61c [maven-release-plugin] prepare for next development iteration 2016-02-04 16:51:44 +01:00
Norman Maurer
7eb3a60dba [maven-release-plugin] prepare release netty-4.1.0.CR2 2016-02-04 16:37:06 +01:00
Norman Maurer
a0758e7e60 [#4794] Support window size flag by default if ZlibCodecFactory supports it.
Motivation:

If the ZlibCodecFactory can support using a custom window size we should support it by default in the websocket extensions as well.

Modifications:

Detect if a custom window size can be handled by the ZlibCodecFactory and if so enable it by default for PerMessageDeflate*ExtensionHandshaker.

Result:

Support window size flag by default in most installations.
2016-02-04 14:01:40 +01:00
Norman Maurer
1c417e5f82 [maven-release-plugin] prepare for next development iteration 2016-01-21 15:35:55 +01:00
Norman Maurer
c681a40a78 [maven-release-plugin] prepare release netty-4.1.0.CR1 2016-01-21 15:28:21 +01:00
Xiaoyan Lin
d1ef33b8f4 Change 64 to 63 in Snappy.decodeLiteral
Motivation:

According to https://github.com/google/snappy/blob/master/format_description.txt#L55 , Snappy.decodeLiteral should handle the cases of 60, 61, 62 and 63. However right now it processes 64 instead of 63. I believe it's a typo since `tag >> 2 & 0x3F` must be less than 64.

Modifications:

Use the correct value 63.

Result:

Snappy.decodeLiteral handles the correct case.
2016-01-21 09:59:10 +01:00
Robert Borg
3785ca9311 added support for Protobuf codec nano runtime
Motivation:

Netty was missing support for Protobuf nano runtime targeted at
weaker systems such as Android devices.

Modifications:

Added ProtobufDecoderNano and ProtobufDecoderNano
in order to provide support for Nano runtime.

modified ProtobufVarint32FrameDecoder and
ProtobufLengthFieldPrepender in order to remove any
on either Nano or Lite runtime by copying the code
for handling Protobuf varint32 in from Protobuf
library.

modified Licenses and NOTICE in order to reflect the
changes i made.

added Protobuf Nano runtime as optional dependency

Result:

Netty now supports Protobuf Nano runtime.
2016-01-19 21:39:17 +01:00
Stephane Landelle
f7c83c8565 Don't enforce JZlibDecoder on JDK7+, close #4707
Motivation:

`JdkZlibDecoder` is available since Netty 4.0.8 and works with JDK7+.
However, `io.netty.noJdkZlibDecoder` System prop evaluation always defaults to
true, causing Netty to always use JZLib when decompressing on the
client side when the property insn't explictly set to `false`.

Modifications:

Default to `false` instead of `true` when JDK7+.

Result:

JZLib optional as expected on JDK7+.
2016-01-14 21:04:15 +01:00
Norman Maurer
4cdbe39284 [#4635] Stop decoding if decoder was removed
Motivation:

We need to check if this handler was removed before continuing with decoding.
If it was removed, it is not safe to continue to operate on the buffer.

Modifications:

Check if decoder was removed after fire messages through the pipeline.

Result:

No illegal buffer access when decoder was removed.
2016-01-05 11:01:24 +01:00
Xiaoyan Lin
a96d52fe66 Fix javadoc links and tags
Motivation:

There are some wrong links and tags in javadoc.

Modifications:

Fix the wrong links and tags in javadoc.

Result:

These links will work correctly in javadoc.
2015-12-26 08:34:31 +01:00
Norman Maurer
693633eeff Not use Unpooled to allocate buffers in Base64 but use a ByteBufAllocator
Motivation:

We should not use Unpooled to allocate buffers for performance reasons.

Modifications:

Allow to pass in ByteBufAllocate which is used to allocate buffers or use the allocate of the src buffer.

Result:

Better performance if the PooledByteBufAllocator is used.
2015-12-18 21:22:34 +01:00
Norman Maurer
63426fc3ed Prevent adding newline if Base64 buffer encoded ends directly on MAX_LINE_LENGTH
Motivation:

We need to ensure we not add a newline if the Base64 encoded buffer ends directly on the MAX_LINE_LENGTH. If we miss to do so this produce invalid data.
Because of this bug OpenSslServerContext and OpenSslClientContext may fail to load a cert.

Modifications:

- Only add NEW_LINE if we not are on the end of the dst buffer.
- Add unit test

Result:

Correct result in all cases
2015-12-18 20:57:59 +01:00
Norman Maurer
dc615ecaaf [#4212] Backport WebSocket Extension handlers for client and server.
Motivation:

We have websocket extension support (with compression) in old master. We should port this to 4.1

Modifications:

Backport relevant code.

Result:

websocket extension support (with compression) is now in 4.1.
2015-12-18 09:48:10 +01:00
Norman Maurer
d67fea606b Use ByteBuf.*LE methods for write and read LE
Motivation:

We recently added methods to ByteBuf to directly write and read LE values. We should use these in the Snappy implementation and so reduce duplication.

Modifications:

Replace manually swapping of values with LE write and read methods of ByteBuf.

Result:

Cleaner code with less duplication.
2015-12-10 09:34:27 +01:00
Alex Petrov
0f9492c9af Add first-class Little Endian support to ByteBuf and descendants
As discussed in	#3209, this PR adds Little Endian accessors
to ByteBuf and descendants.

Corresponding accessors were added to UnsafeByteBufUtil,
HeapByteBufferUtil to avoid calling `reverseBytes`.

Deprecate `order()`, `order(buf)` and `SwappedByteBuf`.
2015-11-26 20:30:24 +01:00
Scott Mitchell
cfcee5798d Adjustable size of DefaultHeaders array
Motivation:
DefaultHeaders creates an array of size 16 for all headers. This may waste a good deal of memory if applications only have a small number of headers. This memory may be critical when the number of connections grows large.

Modifications:
- Make the size of the array for DefaultHeaders configurable

Result:
Applications can control the size of the DefaultHeaders array and save memory.
2015-11-23 15:38:08 -08:00
Norman Maurer
7bee318fc7 Use OneTimeTask where possible to reduce object creation
Motivation:

We should use OneTimeTask where possible to reduce object creation.

Modifications:

Replace Runnable with OneTimeTask

Result:

Less object creation
2015-11-20 14:39:06 -08:00
Norman Maurer
2ecce8fa56 [maven-release-plugin] prepare for next development iteration 2015-11-10 22:59:33 +01:00
Norman Maurer
6a93f331d3 [maven-release-plugin] prepare release netty-4.1.0.Beta8 2015-11-10 22:50:57 +01:00
Louis Ryan
6e108cb96a Improve the performance of copying header sets when hashing and name validation are equivalent.
Motivation:
Headers and groups of headers are frequently copied and the current mechanism is slower than it needs to be.

Modifications:
Skip name validation and hash computation when they are not necessary.
Fix emergent bug in CombinedHttpHeaders identified with better testing
Fix memory leak in DefaultHttp2Headers when clearing
Added benchmarks

Result:
Faster header copying and some collateral bug fixes
2015-11-07 08:53:10 -08:00
Louis Ryan
3eb65797ed Make headers.set(self) a no-op instead of throwing. Makes it consistent with setAll
Motivation:

Makes the API contract of headers more consistent and simpler.

Modifications:

If self is passed to set then simply return

Result:

set and setAll will be consistent
2015-11-06 07:00:54 -08:00
Scott Mitchell
19658e9cd8 HTTP/2 Headers Type Updates
Motivation:
The HTTP/2 RFC (https://tools.ietf.org/html/rfc7540#section-8.1.2) indicates that header names consist of ASCII characters. We currently use ByteString to represent HTTP/2 header names. The HTTP/2 RFC (https://tools.ietf.org/html/rfc7540#section-10.3) also eludes to header values inheriting the same validity characteristics as HTTP/1.x. Using AsciiString for the value type of HTTP/2 headers would allow for re-use of predefined HTTP/1.x values, and make comparisons more intuitive. The Headers<T> interface could also be expanded to allow for easier use of header types which do not have the same Key and Value type.

Motivation:
- Change Headers<T> to Headers<K, V>
- Change Http2Headers<ByteString> to Http2Headers<CharSequence, CharSequence>
- Remove ByteString. Having AsciiString extend ByteString complicates equality comparisons when the hash code algorithm is no longer shared.

Result:
Http2Header types are more representative of the HTTP/2 RFC, and relationship between HTTP/2 header name/values more directly relates to HTTP/1.x header names/values.
2015-10-30 15:29:44 -07:00
Norman Maurer
11e8163aa9 [#4284] Forward decoded messages more frequently
Motivation:

At the moment we only forward decoded messages that were added the out List once the full decode loop was completed. This has the affect that resources may not be released as fast as possible and as an application may incounter higher latency if the user triggeres a writeAndFlush(...) as a result of the decoded messages.

Modifications:

- forward decoded messages after each decode call

Result:

Forwarding decoded messages through the pipeline in a more eager fashion.
2015-10-07 14:15:53 +02:00
Norman Maurer
2ff2806ada [maven-release-plugin] prepare for next development iteration 2015-10-02 09:03:29 +02:00
Norman Maurer
5a43de10f7 [maven-release-plugin] prepare release netty-4.1.0.Beta7 2015-10-02 09:02:58 +02:00
Norman Maurer
ca44436ce6 [#4265] Not allow to add/set DefaultHttpHeaders to itself.
Motivation:

We should prevent to add/set DefaultHttpHeaders to itself to prevent unexpected side-effects.

Modifications:

Throw IllegalArgumentException if user tries to pass the same instance to set/add.

Result:

No surprising side-effects.
2015-09-30 08:57:53 +02:00
Norman Maurer
c8a941d01e [#4275] Discard bytes after X reads to guard against OOME.
Motivation:

If a remote peer writes fast enough it may take a long time to have fireChannelReadComplete(...) triggered. Because of this we need to take special care and ensure we try to discard some bytes if channelRead(...) is called to often in ByteToMessageDecoder.

Modifications:

- Add ByteToMessageDecoder.setDiscardAfterReads(...) which allows to set the number of reads after which we try to discard the read bytes
- Use default value of 16 for max reads.

Result:

No risk of OOME.
2015-09-29 12:02:39 +02:00
Scott Mitchell
7adc1f9eb4 STOMP Headers Name Comparator
Motivation:
The HashingStrategy for DefaultStompHeaders was using the java .equals() method which would fail to compare String, AsciiString, and other CharSequence objects as equal.

Modification:
- Use AsciiString.CASE_SENSITIVE_HASHER for DefaultStompHeaders

Result:
DefaultStompHeaders work with all CharSequence objects.
Fixes https://github.com/netty/netty/issues/4247
2015-09-23 16:47:09 -07:00
Scott Mitchell
47726991b2 HTTP/2 Header Name Validation
Motivation:
The HTTP/2 header name validation was removed, and does not currently exist.

Modifications:
- Header name validation for HTTP/2 should be restored and set to the default mode of operation.

Result:
HTTP/2 header names are validated according to https://tools.ietf.org/html/rfc7540
2015-09-09 13:59:08 -07:00
Norman Maurer
34de2667c7 [maven-release-plugin] prepare for next development iteration 2015-09-02 11:45:20 +02:00
Norman Maurer
2eb444ec1d [maven-release-plugin] prepare release netty-4.1.0.Beta6 2015-09-02 11:36:11 +02:00
Norman Maurer
1fefe9affb [#4087] Correctly forward bytes when remove codec and handle channelInactive / channelReadComplete(...)
Motivation:

We missed to correctly implement the handlerRemoved(...) / channelInactive(...) and channelReadComplete(...) method, this leaded to multiple problems:

 - Missed to forward bytes when the codec is removed from the pipeline
 - Missed to call decodeLast(...) once the Channel goes in active
 - No correct handling of channelReadComplete that could lead to grow of cumulation buffer.

Modifications:

- Correctly implement methods and forward to the internal ByteToMessageDecoder
- Add unit test.

Result:

Correct behaviour
2015-08-21 18:26:32 +02:00
Scott Mitchell
a7135e8677 HttpObjectAggregator doesn't check content-length header
Motivation:
The HttpObjectAggregator always responds with a 100-continue response. It should check the Content-Length header to see if the content length is OK, and if not responds with a 417.

Modifications:
- HttpObjectAggregator checks the Content-Length header in the case of a 100-continue.

Result:
HttpObjectAggregator responds with 417 if content is known to be too big.
2015-08-17 09:26:50 -07:00
Scott Mitchell
ba6ce5449e Headers Performance Boost and Interface Simplification
Motivation:
A degradation in performance has been observed from the 4.0 branch as documented in https://github.com/netty/netty/issues/3962.

Modifications:
- Simplify Headers class hierarchy.
- Restore the DefaultHeaders to be based upon DefaultHttpHeaders from 4.0.
- Make various other modifications that are causing hot spots.

Result:
Performance is now on par with 4.0.
2015-08-17 08:50:11 -07:00
Jakob Buchgraber
6fd0a0c55f Faster and more memory efficient headers for HTTP, HTTP/2, STOMP and SPYD. Fixes #3600
Motivation:

We noticed that the headers implementation in Netty for HTTP/2 uses quite a lot of memory
and that also at least the performance of randomly accessing a header is quite poor. The main
concern however was memory usage, as profiling has shown that a DefaultHttp2Headers
not only use a lot of memory it also wastes a lot due to the underlying hashmaps having
to be resized potentially several times as new headers are being inserted.

This is tracked as issue #3600.

Modifications:
We redesigned the DefaultHeaders to simply take a Map object in its constructor and
reimplemented the class using only the Map primitives. That way the implementation
is very concise and hopefully easy to understand and it allows each concrete headers
implementation to provide its own map or to even use a different headers implementation
for processing requests and writing responses i.e. incoming headers need to provide
fast random access while outgoing headers need fast insertion and fast iteration. The
new implementation can support this with hardly any code changes. It also comes
with the advantage that if the Netty project decides to add a third party collections library
as a dependency, one can simply plug in one of those very fast and memory efficient map
implementations and get faster and smaller headers for free.

For now, we are using the JDK's TreeMap for HTTP and HTTP/2 default headers.

Result:

- Significantly fewer lines of code in the implementation. While the total commit is still
  roughly 400 lines less, the actual implementation is a lot less. I just added some more
  tests and microbenchmarks.

- Overall performance is up. The current implementation should be significantly faster
  for insertion and retrieval. However, it is slower when it comes to iteration. There is simply
  no way a TreeMap can have the same iteration performance as a linked list (as used in the
  current headers implementation). That's totally fine though, because when looking at the
  benchmark results @ejona86 pointed out that the performance of the headers is completely
  dominated by insertion, that is insertion is so significantly faster in the new implementation
  that it does make up for several times the iteration speed. You can't iterate what you haven't
  inserted. I am demonstrating that in this spreadsheet [1]. (Actually, iteration performance is
  only down for HTTP, it's significantly improved for HTTP/2).

- Memory is down. The implementation with TreeMap uses on avg ~30% less memory. It also does not
  produce any garbage while being resized. In load tests for GRPC we have seen a memory reduction
  of up to 1.2KB per RPC. I summarized the memory improvements in this spreadsheet [1]. The data
  was generated by [2] using JOL.

- While it was my original intend to only improve the memory usage for HTTP/2, it should be similarly
  improved for HTTP, SPDY and STOMP as they all share a common implementation.

[1] https://docs.google.com/spreadsheets/d/1ck3RQklyzEcCLlyJoqDXPCWRGVUuS-ArZf0etSXLVDQ/edit#gid=0
[2] https://gist.github.com/buchgr/4458a8bdb51dd58c82b4
2015-08-04 17:12:24 -07:00
tczerwinski
b33c7b12a4 XmlFrameDecoder is corrupt
Motivation:

Two problems:
1. Decoder assumption that as soon as it finds </ element it can decrement opened xml brackets counter. It can lead to bugs when closing bracket is not in byteBuf yet.
2. Not proper handling of more than two root elements in XML document. First element will be processed properly, second one not. It is caused by assumption that byteBuf readerIndex is 0 at the begging of decoding.

Modifications:

Both problems were resolved by fixes:
1. decrement opened brackets count only if </ > enclosing bracket is found
2. consider readerIndex higher than 0 when counting output frame length

Result:

Both problems were resolved
2015-07-29 18:49:26 +02:00
Norman Maurer
c53dbb748e Add ProtocolDetectionResult and use it in HAProxyMessageDecoder for allow detect HAProxy protocol.
Motivation:

Sometimes it is useful to detect if a ByteBuf contains a HAProxy header, for example if you want to write something like the PortUnification example.

Modifications:

- Add ProtocolDetectionResult which can be used as a return type for detecting different protocol.
- Add new method which allows to detect HA Proxy messages.

Result:

Easier to detect protocol.
2015-06-23 08:59:07 +02:00
Trustin Lee
950da2eae1 Allow MessageAggregator to disallow non-empty content
Motivation:

A user sometimes just want the aggregated message has no content at
all. (e.g. A user only wants HTTP GET requests.)

Modifications:

- Do not raise IllegalArgumentException even if a user specified
  the maxContentLength of 0

Result:

A user can disallow a message with non-empty content.
2015-06-10 12:06:27 +09:00
Norman Maurer
f23b7b4efd [maven-release-plugin] prepare for next development iteration 2015-05-07 14:21:08 -04:00