Commit Graph

271 Commits

Author SHA1 Message Date
Norman Maurer
4beb075dd3 Ensure we only call debugData.release() if we called debugData.retain()
Motivation:

We need to ensure we only call debugData.release() if we called debugData.retain(), otherwise we my see an IllegalReferenceCountException.

Modifications:

Only call release() if we called retain().

Result:

No more IllegalReferenceCountException possible.
2016-09-12 06:47:38 -07:00
Norman Maurer
51629245d0 Call debugData.retain() before we forward the frame to the pipeline
Motivation:

We need to call debugData.retain() before we forward the frame to the pipeline as ByteToMessageDecoder will call release() on the buffer.

Modifications:

Correctly retain debugData and fix the unit test to test for it.

Result:

No more IllegalReferenceCountException when using the Http2FrameCodec.
2016-09-12 06:45:13 -07:00
Norman Maurer
d2389a9339 [#5762] HTTP/2: SETTINGS_HEADER_TABLE_SIZE should be an unsigned int
Motivation:

he HTTP/2 spec demands that the max value for SETTINGS_HEADER_TABLE_SIZE should be an unsigned 32-bit integer.

Modifications:

Change the limit to unsigned 32-bit integer and add tests.

Result:

Complient to rfc.
2016-09-09 13:20:32 +02:00
Norman Maurer
54b1a100f4 [maven-release-plugin] prepare for next development iteration 2016-08-26 10:06:32 +02:00
buchgr
515f8964b4 HTTP/2: Fix some errors reported by h2spec.
Motivation:

h2spec [1] reported 32 issues [2] with Netty's HTTP/2 implementation.

Modifications:

Fixed 11 of those issues. Mostly wrong error codes or added missing validation
code in the frame reader.

Result:

11 fewer errors. h2spec now reports: 70 tests, 48 passed, 1 skipped, 21 failed

[1] https://github.com/summerwind/h2spec
[2] https://github.com/netty/netty/issues/5761
2016-09-01 08:28:16 +02:00
Norman Maurer
1208b90f57 [maven-release-plugin] prepare release netty-4.1.5.Final 2016-08-26 04:59:35 +02:00
Trustin Lee
5b46cf25c1 Fulfill the write promise of empty HTTP/2 DATA frames sooner
Motivation:

DefaultHttp2ConnectionEncoder.FlowControlledData.write() does not
complete the write promise of empty HTTP/2 DATA frames until either a
non-DATA frame, a non-empty DATA frame or a DATA frame with endOfStream
set. This makes the write promise of the empty DATA frame is notified
much later than it could be.

Modifications:

- Notify the write promise of the empty DATA frames immediately is the
  queue contains empty DATA frames only

Result:

The write promise of an empty DATA frame is notified sooner.
2016-08-26 08:45:09 +09:00
Scott Mitchell
208893aac9 HTTP/2 Hpack Encoder Cleanup
Motivation:
The HTTP/2 HPACK Encoder class has some code which is only used for test purposes. This code can be removed to reduce complexity and member variable count.

Modifications:
- Remove test code and update unit tests
- Other minor cleanup

Result:
Test code is removed from operational code.
2016-08-25 09:08:46 -07:00
Norman Maurer
5fd239c29c Ensure we not log missleading errors if the promise was already failed due errors
Motivation:

In DefaultHttp2ConnectionEncoder we fail the promise in in the FlowControlledData.error(...) method but also add it the CoalescingBufferQueue. Which can lead to have the promise failed by error(...) before it can be failed in CoalescingBufferQueue.

This can lead to confusing and missleading errors in the log like:
    2016-08-12 09:47:43,716       nettyIoExecutorGroup-1-9 [WARN ] PromiseNotifier                - Failed to mark a promise as failure because it's done already: DefaultChannelPromise@374225e0(failure: javax.net.ssl.SSLException: SSLEngine closed already)
    javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-all-4.1.5.Final-SNAPSHOT.jar:?]

Modifications:

Ensure we only fail the queue (which will also fail the promise).

Result:

No more missleading logs.
2016-08-18 07:17:45 +02:00
buchgr
d568cfc14a HTTP/2: Treat MAX_CONCURRENT_STREAMS exceeded as a stream error.
Motivation:

As per the HTTP/2 spec, exceeding the MAX_CONCURRENT_STREAMS should be treated as a stream error as opposed to a connection error.

"An endpoint that receives a HEADERS frame that causes its advertised concurrent stream limit to be exceeded MUST treat this as a stream error (Section 5.4.2) of type PROTOCOL_ERROR or REFUSED_STREAM." http://httpwg.org/specs/rfc7540.html#rfc.section.5.1.2

Modifications:

Make the error a stream error.

Result:

It's a stream error.
2016-08-17 14:47:15 +02:00
Scott Mitchell
f73c4f24ee HTTP/2 HPACK Bounds Check Fix
Motivation:
21e8d84b79 changed the way bounds checking was done, but however a bounds check in the case of READ_LITERAL_HEADER_NAME_LENGTH_PREFIX was using an old value. This would delay when the bounds check would actually be done and potentially allow more allocation than necessary.

Modifications:
- Use the new length (index) in the bounds check instead of an old length (nameLength) which had not yet been assigned to the new value.

Result:
More correct bounds checking.
2016-08-13 11:04:26 -07:00
Scott Mitchell
21e8d84b79 HTTP/2 Simplify Headers Decode Bounds Checking
Motivation:
The HPACK decoder keeps state so that the decode method can be called multiple times with successive header fragments. This decoder also requires that a method is called to signify the decoding is complete. At this point status is returned to indicate if the max header size has been violated. Netty always accumulates the header fragments into a single buffer before attempting to HPACK decode process and so keeping state and delaying notification that bounds have been exceeded is not necessary.

Modifications:
- HPACK Decoder#decode(..) now must be called with a complete header block
- HPACK will terminate immediately if the maximum header length, or maximum number of headers is exceeded
- Reduce member variables in the HPACK Decoder class because they can now live in the decode(..) method

Result:
HPACK bounds checking is done earlier and less class state is needed.
2016-08-12 17:12:53 -07:00
Scott Mitchell
4d74bf3984 HTTP/2 MaxStreams cleanup
Motivation:
765e944d4d imposed a limit on the maximum number of stream in all states. However the default limit did not allow room for streams in addition to SETTINGS_MAX_CONCURRENT_STREAMS. This can mean streams in states outside the states which SETTINGS_MAX_CONCURRENT_STREAMS applies to may not be reliably created.

Modifications:
- The default limit should be larger than SETTINGS_MAX_CONCURRENT_STREAMS

Result:
More lenient limit is applied to maxStreams by default.
2016-08-11 09:38:58 -07:00
Scott Mitchell
765e944d4d HTTP/2 limit streams in all states
Motivation:
SETTINGS_MAX_CONCURRENT_STREAMS does not apply to idle streams and thus we do not apply any explicit limitations on how many idle streams can be created. This may allow a peer to consume an undesirable amount of resources.

Modifications:
- Each Endpoint should enforce a limit for streams in a any state. By default this limit will be the same as SETTINGS_MAX_CONCURRENT_STREAMS but can be overridden if necessary.

Result:
There is now a limit to how many IDLE streams can be created.
2016-08-11 09:01:37 -07:00
Scott Mitchell
a4ad68239e Http2ConnectionDecoder remove localSettings setter method
Motivation:
Http2ConnectionDecoder#localSettings(Http2Settings) is not used in codec-http2 and currently results in duplicated code.

Modifications:
- Remove Http2ConnectionDecoder#localSettings(Http2Settings)

Result:
Smaller interface and less duplicated code.
2016-08-10 12:23:03 -07:00
Norman Maurer
cb7cf4491c [maven-release-plugin] prepare for next development iteration 2016-07-27 13:29:56 +02:00
Norman Maurer
9466b32d05 [maven-release-plugin] prepare release netty-4.1.4.Final 2016-07-27 13:16:59 +02:00
buchgr
328510468c Complete ChannelPromise for Http2WindowUpdateFrames in Http2FrameCodec. Fixes #5530
Motivation:

The channel promise of a window update frame is not completed correctly,
depending on the failure or success of the operation.

Modification:

Succeed / Fail the promise if the window update succeeds / fails.

Result:

Correctly succeed / fail the promise.
2016-07-15 17:29:41 +02:00
Norman Maurer
047f6aed28 [maven-release-plugin] prepare for next development iteration 2016-07-15 09:09:13 +02:00
Norman Maurer
b2adea87a0 [maven-release-plugin] prepare release netty-4.1.3.Final 2016-07-15 09:08:53 +02:00
Oliver Gould
f2ce28bf18 Satisfy write promise when writing an Http2WindowUpdateFrame to Http2FrameCodec.
Motivation:

When writing an Http2WindowUpdateFrame to an Http2FrameCodec, the
ChannelPromise is never satisfied, so callers cannot generically rely on the
write future being satisfied on success.

Modifications:

When writing Http2WindowUpdateFrame, Http2FrameCodec now satisfies the
ChannelPromise immediately.

Result:

The write future is satisfied on successful writes.

Fixes netty/netty#5530.
2016-07-13 21:31:34 +02:00
Scott Mitchell
9de90d07a9 DefaultHttp2RemoteFlowController reentry infinite loop
Motivation:
DefaultHttp2RemoteFlowController.writePendingBytes does not support reentry but does not enforce this constraint. Reentry is possible if the channel transitions from Writable -> Not Writable -> Writable during the distribution phase. This can happen if the user flushes when notified of the channel transitioning to the not writable state, and may be done if the user wants to fill the channel outbound buffer before flushing.

Modifications:
- DefaultHttp2RemoteFlowController.writePendingBytes should protect against reentry

Result:
DefaultHttp2RemoteFlowController will not allocate unexpected amounts or enter an infinite loop.
2016-07-13 09:12:33 -07:00
Scott Mitchell
4baff691b4 DefaultPromise make listeners not volatile
Motivation:
DefaultPromise has a listeners member variable which is volatile to allow for an optimization which makes notification of listeners less expensive when there are no listeners to notify. However this change makes all other operations involving the listeners member variable more costly. This optimization which requires listeners to be volatile can be removed to avoid volatile writes/reads for every access on the listeners member variable.

Modifications:
- DefaultPromise listeners is made non-volatile and the null check optimization is removed

Result:
DefaultPromise.listeners is no longer volatile.
2016-07-07 12:53:03 -07:00
Norman Maurer
6492cb98b2 Revert "DefaultPromise make listeners not volatile"
This reverts commit 4d8132ff24 as I missed something I want to discuss first.
2016-07-07 08:37:41 +02:00
Scott Mitchell
4d8132ff24 DefaultPromise make listeners not volatile
Motivation:
DefaultPromise has a listeners member variable which is volatile to allow for an optimization which makes notification of listeners less expensive when there are no listeners to notify. However this change makes all other operations involving the listeners member variable more costly. This optimization which requires listeners to be volatile can be removed to avoid volatile writes/reads for every access on the listeners member variable.

Modifications:
- DefaultPromise listeners is made non-volatile and the null check optimization is removed

Result:
DefaultPromise.listeners is no longer volatile.
2016-07-07 08:01:25 +02:00
Norman Maurer
4676a2271c [maven-release-plugin] prepare for next development iteration 2016-07-01 10:33:32 +02:00
Norman Maurer
ad270c02b9 [maven-release-plugin] prepare release netty-4.1.2.Final 2016-07-01 09:07:40 +02:00
Scott Mitchell
6af56ffe76 HPACK Encoder headerFields improvements
Motivation:
HPACK Encoder has a data structure which is similar to a previous version of DefaultHeaders. Some of the same improvements can be made.

Motivation:
- Enforce the restriction that the Encoder's headerFields length must be a power of two so we can use masking instead of modulo
- Use AsciiString.hashCode which already has optimizations instead of having yet another hash code algorithm in Encoder

Result:
Fixes https://github.com/netty/netty/issues/5357
2016-06-30 09:00:12 -07:00
Scott Mitchell
a7f7d9c8e0 Remove unsafe char[] access in PlatformDependent
Motivation:
PlatformDependent attempts to use reflection to get the underlying char[] (or byte[]) from String objects. This is fragile as if the String implementation does not utilize the full array, and instead uses a subset of the array, this optimization is invalid. OpenJDK6 and some earlier versions of OpenJDK7 String have the capability to use a subsection of the underlying char[].

Modifications:
- PlatformDependent should not attempt to use the underlying array from String (or other data types) via reflection

Result:
PlatformDependent hash code generation for CharSequence does not depend upon specific JDK implementation details.
2016-06-30 08:58:28 -07:00
Scott Mitchell
df41be6fc8 HTTP/2 Decoder validate that GOAWAY lastStreamId doesn't increase
Motivation:
The HTTP/2 RFC states in https://tools.ietf.org/html/rfc7540#section-6.8 that Endpoints MUST NOT increase the value they send in the last stream identifier however we don't enforce this when decoding GOAWAY frames.

Modifications:
- Throw a connection error if the peer attempts to increase the lastStreamId in a GOAWAY frame

Result:
RFC is more strictly enforced.
2016-06-29 07:53:13 -07:00
buchgr
3613d15bca Split Http2MultiplexCodec into Frame- and MultiplexCodec + Tests. Fixes #4914
Motivation:

Quote from issue 4914:
"Http2MultiplexCodec currently does two things: mapping the existing h2 API to frames and managing the child channels.

It would be better if the two parts were separated. This would allow less-coupled development of the HTTP/2 handlers (flow control could be its own handler, for instance) and allow applications to insert themselves between all streams and the codec, which permits custom logic and could be used, in part, to implement custom frame types.

It would also greatly ease testing, as the child channel could be tested by itself without dealing with how frames are encoded on the wire."

Modifications:

- Split the Http2MultiplexCodec into Http2FrameCodec and Http2MultiplexCodec. The Http2FrameCodec interacts with the existing HTTP/2 callback-based API, while the Http2MulitplexCodec is completely independent of it and simply multiplexes Http2StreamFrames to the child channels. Additionally, the Http2Codec handler is introduced, which is a convenience class that simply sets up the Http2FrameCodec and Http2MultiplexCodec in the channel pipeline and removes itself.

- Improved test coverage quite a bit.

Result:

- The original Http2MultiplexCodec is split into Http2FrameCodec and Http2MultiplexCodec.
- More tests for higher confidence in the code.
2016-06-29 07:22:09 +02:00
Scott Mitchell
70651cc58d HpackUtil.equals performance improvement
Motivation:
PR #5355 modified interfaces to reduce GC related to the HPACK code. However this came with an anticipated performance regression related to HpackUtil.equals due to AsciiString's increase cost of charAt(..). We should mitigate this performance regression.

Modifications:
- Introduce an equals method in PlatformDependent which doesn't leak timing information and use this in HpcakUtil.equals

Result:
Fixes https://github.com/netty/netty/issues/5436
2016-06-27 14:37:39 -07:00
buchgr
73c4ad5f0a http2: count pad length field toward flow control. Fixes #5434
Motivation:
The HTTP/2 specification requires the pad length field of DATA, HEADERS and PUSH_PROMISE frames to be counted towards the flow control window. The current implementation doesn't do so (See #5434).

Furthermore, it's currently not possible to add one byte padding, as this would add the one byte pad length field as well as append one padding byte to the end of the frame.

Modifications:
Include the one byte pad length field in the padding parameter of the API. Thereby extending the allowed value range by one byte to 256 (inclusive). On the wire, a one byte padding is encoded with a pad length field with value zero and a 256 byte padding is encoded with a pad length field with value 255 and 255 bytes append to the end of the frame.

Result:
More correct padding.
2016-06-25 09:51:36 -07:00
Norman Maurer
b4d4c0034d Optimize HPACK usage to align more with Netty types and remove heavy object creations. Related to [#3597]
Motivations:

The HPACK code was not really optimized and written with Netty types in mind. Because of this a lot of garbage was created due heavy object creation.

This was first reported in [#3597] and https://github.com/grpc/grpc-java/issues/1872 .

Modifications:

- Directly use ByteBuf as input and output
- Make use of ByteProcessor where possible
- Use AsciiString as this is the only thing we need for our http2 usage

Result:

Less garbage and better usage of Netty apis.
2016-06-22 14:26:05 +02:00
Tim Brooks
d964bf6f18 Remove usages of deprecated methods group() and childGroup().
Motivation:

These methods were recently deprecated. However, they remained in use in several locations in Netty's codebase.

Modifications:

Netty's code will now access the bootstrap config to get the group or child group.

Result:

No impact on functionality.
2016-06-21 14:06:57 +02:00
Scott Mitchell
6aa5f76d42 HTTP/2 DelegatingDecompressorFrameListener return bytes to flow control
Motivation:
If a single DATA frame ends up being decompressed into multiple frames by DelegatingDecompressorFrameListener the flow control accounting is delayed until all frames have been decompressed. However it is possible the user may want to return bytes to the flow controller which were not included in the onDataRead return value. In this case the amount of processed bytes has not been incremented and will lead to negative value for processed bytes.

Modifications:
- Http2Decompressor.incrementProcessedBytes should be called each time onDataRead is called to ensure all bytes are accounted for at the correct time

Result:
Fixes https://github.com/netty/netty/issues/5375
2016-06-20 14:24:09 -07:00
Norman Maurer
e845670043 Set some StackTraceElement on pre-instantiated static exceptions
Motivation:

We use pre-instantiated exceptions in various places for performance reasons. These exceptions don't include a stacktrace which makes it hard to know where the exception was thrown. This is especially true as we use the same exception type (for example ChannelClosedException) in different places. Setting some StackTraceElements will provide more context as to where these exceptions original and make debugging easier.

Modifications:

Set a generated StackTraceElement on these pre-instantiated exceptions which at least contains the origin class and method name. The filename and linenumber are specified as unkown (as stated in the javadocs of StackTraceElement).

Result:

Easier to find the origin of a pre-instantiated exception.
2016-06-20 11:33:05 +02:00
Guido Medina
c3abb9146e Use shaded dependency on JCTools instead of copy and paste
Motivation:
JCTools supports both non-unsafe, unsafe versions of queues and JDK6 which allows us to shade the library in netty-common allowing it to stay "zero dependency".

Modifications:
- Remove copy paste JCTools code and shade the library (dependencies that are shaded should be removed from the <dependencies> section of the generated POM).
- Remove usage of OneTimeTask and remove it all together.

Result:
Less code to maintain and easier to update JCTools and less GC pressure as the queue implementation nt creates so much garbage
2016-06-10 13:19:45 +02:00
Norman Maurer
4dec7f11b7 [maven-release-plugin] prepare for next development iteration 2016-06-07 18:52:34 +02:00
Norman Maurer
cf670fab75 [maven-release-plugin] prepare release netty-4.1.1.Final 2016-06-07 18:52:22 +02:00
Scott Mitchell
79f2e3604e HTTP/2 close only send GO_AWAY if one has not already been sent
Motivation:
Http2ConnectionHandler will always send a GO_AWAY when the channel is closed. This may cause problems if the user is attempting to control when GO_AWAY is sent and the content of the GO_AWAY.

Modifications:
- When the channel is closed Http2ConnectionHandler should only send a GO_AWAY if one has not already been sent

Result:
The user has more control over when GO_AWAY is sent
Fixes https://github.com/netty/netty/issues/5307
2016-06-06 11:18:30 -07:00
Norman Maurer
844976a0a2 Ensure the same ByteBufAllocator is used in the EmbeddedChannel when compress / decompress. Related to [#5294]
Motivation:

The user may specify to use a different allocator then the default. In this case we need to ensure it is shared when creating the EmbeddedChannel inside of a ChannelHandler

Modifications:

Use the config of the "original" Channel in the EmbeddedChannel and so share the same allocator etc.

Result:

Same type of buffers are used.
2016-05-31 09:08:33 +02:00
Norman Maurer
6ca49d1336 [maven-release-plugin] prepare for next development iteration 2016-05-25 19:16:44 +02:00
Norman Maurer
446b38db52 [maven-release-plugin] prepare release netty-4.1.0.Final 2016-05-25 19:14:15 +02:00
nmittler
79a06eaede Fix NPE in Http2ConnectionHandler.onHttpServerUpgrade
Motivation:

Performing a server upgrade with a new initial flow control window will cause an NPE in the DefaultHttp2RemoteFlowController. This is due to the fact that the monitor does not check whether or not the channel is writable.

Modifications:

Added a check for channel writability before calling `writePendingBytes`. Also fixed a unit test that was supposed to be testing this :).

Result:

Fixes #5301
2016-05-25 09:13:06 -07:00
Norman Maurer
7b25402e80 Add CompositeByteBuf.addComponent(boolean ...) method to simplify usage
Motivation:

At the moment the user is responsible to increase the writer index of the composite buffer when a new component is added. We should add some methods that handle this for the user as this is the most popular usage of the composite buffer.

Modifications:

Add new methods that autoamtically increase the writerIndex when buffers are added.

Result:

Easier usage of CompositeByteBuf.
2016-05-21 19:52:16 +02:00
Scott Mitchell
1cb706ac93 HTTP/2 HPACK Header Name Validation and Trailing Padding
Motivation:
The HPACK code currently disallows empty header names. This is not explicitly forbidden by the HPACK RFC https://tools.ietf.org/html/rfc7541. However the HTTP/1.x RFC https://tools.ietf.org/html/rfc7230#section-3.2 and thus HTTP/2 both disallow empty header names, and so this precondition check should be moved from the HPACK code to the protocol level.
HPACK also requires that string literals which are huffman encoded must be treated as an encoding error if the string has more than 7 trailing padding bits https://tools.ietf.org/html/rfc7541#section-5.2, but this is currently not enforced.

Result:
- HPACK to allow empty header names
- HTTP/1.x and HTTP/2 header validation should not allow empty header names
- Enforce max of 7 trailing padding bits

Result:
Code is more compliant with the above mentioned RFCs
Fixes https://github.com/netty/netty/issues/5228
2016-05-17 13:42:16 -07:00
Trustin Lee
3a9f472161 Make retained derived buffers recyclable
Related: #4333 #4421 #5128

Motivation:

slice(), duplicate() and readSlice() currently create a non-recyclable
derived buffer instance. Under heavy load, an application that creates a
lot of derived buffers can put the garbage collector under pressure.

Modifications:

- Add the following methods which creates a non-recyclable derived buffer
  - retainedSlice()
  - retainedDuplicate()
  - readRetainedSlice()
- Add the new recyclable derived buffer implementations, which has its
  own reference count value
- Add ByteBufHolder.retainedDuplicate()
- Add ByteBufHolder.replace(ByteBuf) so that..
  - a user can replace the content of the holder in a consistent way
  - copy/duplicate/retainedDuplicate() can delegate the holder
    construction to replace(ByteBuf)
- Use retainedDuplicate() and retainedSlice() wherever possible
- Miscellaneous:
  - Rename DuplicateByteBufTest to DuplicatedByteBufTest (missing 'D')
  - Make ReplayingDecoderByteBuf.reject() return an exception instead of
    throwing it so that its callers don't need to add dummy return
    statement

Result:

Derived buffers are now recycled when created via retainedSlice() and
retainedDuplicate() and derived from a pooled buffer
2016-05-17 11:16:13 +02:00
Norman Maurer
68cd670eb9 Remove ChannelHandlerInvoker
Motivation:

We tried to provide the ability for the user to change the semantics of the threading-model by delegate the invoking of the ChannelHandler to the ChannelHandlerInvoker. Unfortunually this not really worked out quite well and resulted in just more complexity and splitting of code that belongs together. We should remove the ChannelHandlerInvoker again and just do the same as in 4.0

Modifications:

Remove ChannelHandlerInvoker again and replace its usage in Http2MultiplexCodec

Result:

Easier code and less bad abstractions.
2016-05-17 11:14:00 +02:00
Norman Maurer
979dc3e3e4 Fix one more possible leak as a follow up of 341a235fea 2016-05-13 18:38:37 +02:00