Commit Graph

449 Commits

Author SHA1 Message Date
Norman Maurer
41ebb5fcca [maven-release-plugin] prepare release netty-4.1.21.Final 2018-02-05 12:02:19 +00:00
Norman Maurer
e72c197aa3 Reflective setAccessible(true) will produce scary warnings on the console when using java9+, dont do it
Motivation:

Reflective setAccessible(true) will produce scary warnings on the console when using java9+, while netty still works. That said users may feel uncomfortable with these warnings, we should not try to do it by default when using java9+.

Modifications:

Add io.netty.tryReflectionSetAccessible  system property which controls if setAccessible(...) will be used. By default it will bet set to false when using java9+.

Result:

Fixes [#7254].
2018-01-30 12:18:34 +01:00
Julien Hoarau
3e6b54bb59 Fix failing h2spec tests 8.1.2.1 related to pseudo-headers validation
Motivation:

According to the spec:
All pseudo-header fields MUST appear in the header block before regular
header fields. Any request or response that contains a pseudo-header
field that appears in a header block after
a regular header field MUST be treated as malformed (Section 8.1.2.6).

Pseudo-header fields are only valid in the context in which they are defined.
Pseudo-header fields defined for requests MUST NOT appear in responses;
pseudo-header fields defined for responses MUST NOT appear in requests.
Pseudo-header fields MUST NOT appear in trailers.
Endpoints MUST treat a request or response that contains undefined or
invalid pseudo-header fields as malformed (Section 8.1.2.6).

Clients MUST NOT accept a malformed response. Note that these requirements
are intended to protect against several types of common attacks against HTTP;
they are deliberately strict because being permissive can expose
implementations to these vulnerabilities.

Modifications:

- Introduce validation in HPackDecoder

Result:

- Requests with unknown pseudo-field headers are rejected
- Requests with containing response specific pseudo-headers are rejected
- Requests where pseudo-header appear after regular header are rejected
- h2spec 8.1.2.1 pass
2018-01-29 19:42:56 -08:00
Norman Maurer
c795e8897b Convert Http2Error.STREAM_CLOSED to ClosedChannelException when using child channels
Motivation:

We should convert Http2Exceptions that are produced because of STREAM_CLOSED to ClosedChannelException when hand-over to the child channel to make it more consistent with other transports.

Modifications:

- Check if STREAM_CLOSED is used and if so create a new ClosedChannelException (while preserve the original exception as cause) and use it in the child channel
- Ensure STREAM_CLOSED is used in DefaultHttp2RemoteFlowController when writes are failed because of a closed stream.
- Add testcase

Result:

More consistent and correct exception usage.
2018-01-29 17:50:29 -08:00
Norman Maurer
6e6edb59e7 Remove unused variable in DefaultHttp2StreamChannel
Motivation:

We should remove unused variable (was never read).

Modifications:

Remove unused variable (was never read).

Result:

Cleanup.
2018-01-29 11:32:22 +01:00
Thomas Segismont
7f23c34b55 ReadOnlyHttp2Headers.contains always ignores case for values
Motivation:

When checking if a value is present, ReadOnlyHttp2Headers always ignores
case for values.

RFC 7540 says: https://tools.ietf.org/html/rfc7540#section-8.1.2
"header field names are strings of ASCII characters that are compared in a case-insensitive fashion"

But there is no such constraint on header values

Modifications:

Updated ReadOnlyHttp2Headers.contains to compare header value in a
case-sensitive way.

Result:

ReadOnlyHttp2Headers compares header names in a case-insensitive way,
values in a case-sensitive way.
2018-01-27 20:29:40 +01:00
Thomas Segismont
0db652277f DefaultHttp2Headers#contains(CharSequence, CharSequence) does not work with String
Motivation:

If you test a header value providing a String, contains() returns false.
This is due to the implementation inherited from DefaultHeaders using
the JAVA_HASHER.

JAVA_HASHER.equals returns false because a is a String and b an
AsciiString.

Modifications:

DefaultHttp2Headers overrides contains and uses CASE_SENSITIVE_HASHER.

Result:

You can test a header value with any CharSequence implementation.
2018-01-27 20:27:50 +01:00
Scott Mitchell
614b9e0f25 Add tests for Http2MultiplexChannel close promise completion consistency with AbstractChannel
Motivation:
The completion order of promises in Http2MultiplexChannel#close should be consistent with that of AbstractChannel. Otherwise this may result in Future listeners seeing incorrect channel state.

Modifications:
Add tests cases.

Result:
Ensure consistent behavior between Http2MultiplexChannel and AbstractChannel.
2018-01-27 08:57:17 +01:00
Norman Maurer
b423a35783 Correctly handle multiple calls to DefaultHttp2StreamChannel.Unsafe.close(...)
Motivation:

Calling DefaultHttp2StreamChannel.Unsafe.close(...) multiple times should not fail.

Modification:

- Correctly handle multiple calls to DefaultHttp2StreamChannel.Unsafe.close(...)
- Complete closePromise and promise that is given to close(...) in the correct order.
- Add unit test

Result:

Fixes [#7628] and [#7641]
2018-01-27 08:55:35 +01:00
Norman Maurer
b1695fe17d Ensure async failures are correctly propagated to Http2LifecycleManager.onError(...) in all cases.
Motivation:

If DefaultHttp2ConnectionEncoder process outbound operation it sometimes missed to call Http2LifecycleManager.onError(...) when the operation was executed asynchronously.

Modifications:

Make best effort to update flags but still ensure failures are propageted to Http2LifecycleManager.onError(...) in all cases.

Result:

More consistent handling of errors.
2018-01-27 08:46:23 +01:00
Norman Maurer
c43dc3364b Cleanup Http2MultiplexCodec by removing out-dated TODO
Motivation:

Http2MultiplexCodec contains some TODO that is outdated.

Modifications:

Remove TODO which is outdated

Result:

Cleaner code.
2018-01-27 08:42:17 +01:00
Carl Mastrangelo
9ba942e59b Swap header check in ReadOnlyHttp2Headers
Motivation:
Pseudo headers are checked less frequently than normal headers, so
it is more efficient to check the latter first.

Modifications:
Swap the order of the check, and fix minor formatting

Result:
Possibly more efficient header checks
2018-01-26 19:01:02 -08:00
Norman Maurer
e975c5f6fe Reduce objects by directly implement interface in internal implementations of DefaultHttp2RemoteFlowController
Motivation:

We can just implement the interfaces directly and so reduce object creation in DefaultHttp2RemoteFlowController.

Modifications:

Directly implement the interfaces.

Result:

Less object creation.
2018-01-26 18:33:58 -08:00
Norman Maurer
49d1db4113 Http2MultiplexCodec.DefaultHttpStreamChannel.isOpen() / isActive() shoule be false when fireChannelActive() is called
Motivation:

When part of a HTTP/2 StreamChannel the Http2StreamChannel.isOpen() / isActive() should report false within a call to a ChannelInboundHandlers channelInactive() method.

Modifications:

Fullfill promise before call fireChannelInactive()

Result:

Correctly update state / promise before notify handlers. Fixes [#7638]
2018-01-26 17:34:37 -08:00
Thomas Segismont
bed74d8380 Method to check if a Http2 header is present and has a given value
Motivation:

With HTTP1, it's very easy to check if a header is present and has a
given value: you can simply invoke
io.netty.handler.codec.http.HttpHeaders#contains(java.lang.CharSequence, java.lang.CharSequence, boolean)

It is not possible to do the same with HTTP2. You have to get the list
of all headers (returned as String) and then iterate over it invoking
String#equals or String#equalsIgnoreCase

Modifications:

I've added io.netty.handler.codec.http2.Http2Headers#contains and
implemented it in DefaultHttp2Headers, EmptyHttp2Headers and ReadOnlyHttp2Headers.

Result:

You can use AsciiString constants to check if a header is present in a
consice and efficient manner.
2018-01-26 08:33:49 -08:00
Norman Maurer
1df5b02fd9 Do not fire outbound exception throught the pipeline when using Http2FrameCodec / Http2MultiplexCodec
Motivation:

Usually when using netty exceptions which happen for outbound operations should not be fired through the pipeline but only the ChannelPromise should be failed.

Modifications:

- Change Http2LifecycleManager.onError(...) to take also an boolean that indicate if the error was caused by an outbound operation
- Channel Http2ConnectionHandler.on*Error(...) methods to also take this boolean
- Change Http2FrameCodec to only fire exceptions through the pipeline if these are not outbound operations related
- Add unit test.

Result:

More consistent error handling when using Http2FrameCodec and Http2MultiplexCodec.
2018-01-25 13:42:28 -08:00
Bryce Anderson
8a095d0244 Http2MultiplexCodec should propagate unhandled Http2Frames down the pipeline
Motivation:

Http2MultiplexCodec swallows Http2PingFrames without releasing the payload, resulting in a memory leak.

Modification:

Send unhandled frames down the pipeline for consumption/disposal by another InboundChannelHandler.

Result:

Fixes #7607.
2018-01-25 13:26:45 -08:00
Scott Mitchell
4921f62c8a
HttpResponseStatus object allocation reduction
Motivation:
Usages of HttpResponseStatus may result in more object allocation then necessary due to not looking for cached objects and the AsciiString parsing method not being used due to CharSequence method being used instead.

Modifications:
- HttpResponseDecoder should attempt to get the HttpResponseStatus from cache instead of allocating a new object
- HttpResponseStatus#parseLine(CharSequence) should check if the type is AsciiString and redirect to the AsciiString parsing method which may not require an additional toString call
- HttpResponseStatus#parseLine(AsciiString) can be optimized and doesn't require and may not require object allocation

Result:
Less allocations when dealing with HttpResponseStatus.
2018-01-24 22:01:52 -08:00
Norman Maurer
b769ec0934 Reduce overhead of cancel flowcontrolled writes.
Motivation:

When we cancel the flowcontrolled writes we did create a new StreamException for each write that was enqueued. Creating Exceptions is very expensive due of filling the stacktrace.

Modifications:

Only create the StreamException once and reuse the same for all the flowcontrolled writes (per stream).

Result:

Less expensive to cancel flowcontrolled writes.
2018-01-24 19:42:13 +01:00
Scott Mitchell
46dac128f7
Http2FrameCodec WindowUpdate bug
Motivation:
Http2FrameCodec increases the initialWindowSize when the user attempts to increase the connection flow control window. The initialWindowSize should only be touched as a result of a SETTINGS frame, and otherwise may result in flow control getting out of sync with our peer.

Modifications:
- Http2FrameCodec shouldn't update the initialWindowSize when a WindowUpdateFrame is written on the connection channel

Result:
More correct WindowUpdate processing.
2018-01-22 11:17:56 -08:00
Norman Maurer
ea58dc7ac7 [maven-release-plugin] prepare for next development iteration 2018-01-21 12:53:51 +00:00
Norman Maurer
96c7132dee [maven-release-plugin] prepare release netty-4.1.20.Final 2018-01-21 12:53:34 +00:00
Scott Mitchell
322a062185 HpackDecoderTest cleanup
Motivation:
public ExpectedException fields should be final.
setUp method signature has a throws, but it isn't necessary
2018-01-19 17:26:06 +01:00
Julien Hoarau
61dba95091 Fix h2spec test 4.3 about invalid header block
Motivation:

HPackDecoder works on entire header block, we shouldn't encounter
incomplete header fields. If we do we should treat it as
a decoding error and according to the specification:

A decoding error in a header block MUST be treated as
a connection error (Section 5.4.1) of type COMPRESSION_ERROR.

Modifications:

* Check final state in HpackDecoder once we've decoded all the data.

Result:

* Throw a connection error if we receive incomplete header fields
* H2spec 4.3 tests all passes
2018-01-18 11:01:20 -08:00
Scott Mitchell
d6e600548b HTTP/2 Remove Http2FrameStream#CONNECTION_STREAM
Motivation:
Http2FrameStream#CONNECTION_STREAM is required to identify the
connection stream. However this leads to inconsistent usage from a user
perspective. When a user creates a Http2Frame for a non-connection
stream, the Http2MultiplexCodec automatically sets the stream, and the
user is never exposed to the Http2FrameStream object. However when the
user writes a Http2Frame for a connection stream they are required to
set the Http2FrameStream object. We can remove the Http2FrameStream#CONNECTION_STREAM
and keep the Http2FrameStream object internal, and therefore consistent
between the connection and non-connection use cases.

Modifications:
- Remove Http2FrameStream#CONNECTION_STREAM
- Update Http2FrameCodec to handle Http2Frame#stream() which returns
null

Result:
More consistent usage on http2 parent channel and http2 child channel.
2018-01-14 13:31:30 +01:00
Scott Mitchell
adf2596c36 Http2FrameCodecTest increase timeout
Motivation:
Http2FrameCodecTest#newOutboundStream has a timeout of 1 second and has been observed to timeout on CI servers.

Modifications:
- Increase the timeout to 5 seconds

Result:
Less false positive test failures on CI servers.
2017-12-22 19:32:18 +01:00
Scott Mitchell
336cee9b1f HpackDecoder#addHeader has an unused parameter
Motivation:
HpackDecoder#addHeader takes in the streamId as a parameter but no longer uses it.

Modifications:
- Remove the streamId parameter from HpackDecoder#addHeader

Result:
Less unused parameters in HpackDecoder.
2017-12-22 07:17:24 +01:00
Julien Hoarau
6b033c51a5 Add 32 bytes overhead per header entry when calculating headers length in HPackDecoder
Motivation:

According to the HTTP/2 Spec:

SETTINGS_MAX_HEADER_LIST_SIZE (0x6): This advisory setting informs a
peer of the maximum size of header list that the sender is
prepared to accept, in octets. The value is based on the
uncompressed size of header fields, including the length of the
name and value in octets plus an overhead of 32 octets for each
header field.

We were accounting for the 32 bytes when encoding in HpackEncoder,
but not when decoding in HPackDecoder.

Modifications:

- Add 32 bytes to the header list length for each entry when decoding
with HPackDecoder.

Result:

- We account for the 32 bytes overhead by header entry in HPackDecoder
2017-12-21 17:50:16 -08:00
Norman Maurer
942b993f2b Only enable validation of headers if original headers were validating as well.
Motiviation:

In our replace(...) methods we always used validation for the newly created headers while the original headers may not use validation at all.

Modifications:

- Only use validation if the original headers used validation as well.
- Ensure we create a copy of the headers in replace(...).

Result:

Fixes [#5226]
2017-12-21 07:32:29 +01:00
Scott Mitchell
144716f668
HTTP/2 support pending data larger than Integer.MAX_VALUE
Motivation:
Currently the remote flow controller limits the maximum amount of pending data to Integer.MAX_VALUE. The overflow handling is also not very graceful in that it may lead to infinite loops, or otherwise no progress being made.

Modifications:
- StreamByteDistributor and RemoteFlowController should support pending bytes of type long.

Result:
Fixes https://github.com/netty/netty/issues/4283
2017-12-20 08:55:15 -08:00
Moses Nakamura
94ab0dc442 codec-http2: Better keep track of nameLength in HpackDecoder.decode
Motivation:

http/2 counts header sizes somewhat inconsistently.  Sometimes, headers
which are substantively less than the header list size will be measured
as longer than the header list size.

Modifications:

Keep better track of the nameLength of a given name, so that we don't
accidentally end up reusing a nameLength.

Result:

More consistent measurement of header list size.

Fixes #7511.
2017-12-18 17:41:09 -08:00
Norman Maurer
264a5daa41 [maven-release-plugin] prepare for next development iteration 2017-12-15 13:10:54 +00:00
Norman Maurer
0786c4c8d9 [maven-release-plugin] prepare release netty-4.1.19.Final 2017-12-15 13:09:30 +00:00
Norman Maurer
b2bc6407ab [maven-release-plugin] prepare for next development iteration 2017-12-08 09:26:15 +00:00
Norman Maurer
96732f47d8 [maven-release-plugin] prepare release netty-4.1.18.Final 2017-12-08 09:25:56 +00:00
Moses Nakamura
0cac1a6c8c H2C upgrades should be ineligible for flow control (#7400)
H2C upgrades should be ineligible for flow control

Motivation:

When the h2c upgrade request is too big, the Http2FrameCodec complains
it's too big for flow control reasons, even though it's ineligible for
flow control.

Modifications:

Specially mark upgrade streams and make Http2FrameCodec know not to try
to flow control on those streams.

Result:

Servers won't barf when they receive an upgrade request with a fat
payload.

[Fixes #7280]
2017-12-07 16:46:16 -08:00
Scott Mitchell
7cced5576f Http2ConnectionHandler Http2ConnectionPrefaceAndSettingsFrameWrittenEvent propagation
Motivation:
Http2ConnectionHandler uses ctx.fireUserEvent to propagate the Http2ConnectionPrefaceAndSettingsFrameWrittenEvent through the pipeline. This will propagate the event to the next inbound handler in the pipeline. If the user extends Http2ConnectionHandler the Http2ConnectionPrefaceAndSettingsFrameWrittenEvent may be missed and initialization dependent upon this event will not be run.

Modifications:
- Http2ConnectionHandler should use userEventTriggered instead of ctx.fireUserEvent

Result:
Classes that extend Http2ConnectionHandler will see the Http2ConnectionPrefaceAndSettingsFrameWrittenEvent user event.
2017-12-02 08:23:28 +01:00
Tomasz Jędrzejewski
e8540c2b7a Adding stable JDK9 module names that follow reverse-DNS style
Automatic-Module-Name entry provides a stable JDK9 module name, when Netty is used in a modular JDK9 applications. More info: http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html

When Netty migrates to JDK9 in the future, the entry can be replaced by actual module-info descriptor.

Modification:

The POM-s are configured to put the correct module names to the manifest.

Result:

Fixes #7218.
2017-11-29 11:50:24 +01:00
Scott Mitchell
7d213240ca HttpConversionUtil TE filtering robustness
Motivation:
HttpConversionUtil#toHttp2Headers has special code to filter the TE header name. However this filtering code may result in adding the <TE, TRAILERS> tuple in scenarios that are not appropriate. For example if a value containing trailers is seen it will be added, but the value could not actually be equal to trailers. Also CSV values are not supported.

Modifications:
- Account for CSV header values
- Account for the value containing 'trailers' but not actually being equal to 'trailers'

Result:
More robust parsing of the TE header.
2017-11-22 08:45:11 +01:00
Scott Mitchell
a3e41ba6eb HttpConversionUtils avoid intermediate collection allocation
Modifications:
HttpConversionUtil#toLowercaseMap requires an intermediate List to be allocated. This can be avoided with the recently added value iterator methods.

Modifications:
- Use HttpHeaders#valueCharSequenceIterator instead of getAll

Result:
Less intermediate object allocation and copying.
2017-11-20 14:05:03 -08:00
Scott Mitchell
e6126215e0 DefaultHttp2FrameWriter reduce object allocation
Motivation:
DefaultHttp2FrameWriter#writeData allocates a DataFrameHeader for each write operation. DataFrameHeader maintains internal state and allocates multiple slices of a buffer which is a maximum of 30 bytes. This 30 byte buffer may not always be necessary and the additional slice operations can utilize retainedSlice to take advantage of pooled objects. We can also save computation and object allocations if there is no padding which is a common case in practice.

Modifications:
- Remove DataFrameHeader
- Add a fast path for padding == 0

Result:
Less object allocation in DefaultHttp2FrameWriter
2017-11-20 08:10:59 -08:00
Scott Mitchell
a8bb9dc180 AbstractByteBuf readSlice bound check bug
Motivation:
AbstractByteBuf#readSlice relied upon the bounds checking of the slice operation in order to detect index out of bounds conditions. However the slice bounds checking operation allows for the slice to go beyond the writer index, and this is out of bounds for a read operation.

Modifications:
- AbstractByteBuf#readSlice and AbstractByteBuf#readRetainedSlice should ensure the desired amount of bytes are readable before taking a slice

Result:
No reading of undefined data in AbstractByteBuf#readSlice and AbstractByteBuf#readRetainedSlice.
2017-11-18 09:03:42 +01:00
Norman Maurer
78522cf3f0 Remove unused allocation introduced by d976dc108d 2017-11-17 17:19:36 +01:00
Moses Nakamura
d976dc108d codec-http2: Improve h1 to h2 header conversion
Motivation:

Netty could handle "connection" or "te" headers more gently when
converting from http/1.1 to http/2 headers.  Http/2 headers don't
support single-hop headers, so when we convert from http/1.1 to http/2,
we should drop all single-hop headers.  This includes headers like
"transfer-encoding" and "connection", but also the headers that
"connection" points to, since "connection" can be used to designate
other headers as single-hop headers.  For the "te" header, we can more
permissively convert it by just dropping non-conforming headers (ie
non-"trailers" headers) which is what we do for all other headers when
we convert.

Modifications:

Add a new blacklist to the http/1.1 to http/2 conversion, which is
constructed from the values of the "connection" header, and stop
throwing an exception when a "te" header is passed with a non-"trailers"
value.  Instead, drop all values except for "trailers".  Add unit tests
for "connection" and "te" headers when converting from http/1.1 to http/2.

Result:

This will improve the h2c upgrade request, and also conversions from
http/1.1 to http/2.  This will simplify implementing spec-compliant
http/2 servers that want to share code between their http/1.1 and http/2
implementations.

[Fixes #7355]
2017-11-17 09:09:52 +01:00
Norman Maurer
188ea59c9d [maven-release-plugin] prepare for next development iteration 2017-11-08 22:36:53 +00:00
Norman Maurer
812354cf1f [maven-release-plugin] prepare release netty-4.1.17.Final 2017-11-08 22:36:33 +00:00
Ning Sun
73e8122fc1 Fix sharable check logic
Motivation:

There is an logic issue when checking if ChannelHandler is sharable.

Modification:

Corrected || to &&
2017-11-08 08:36:07 -08:00
Scott Mitchell
35b0cd58fb HTTP/2 write of released buffer should not write and should fail the promise
Motivation:
HTTP/2 allows writes of 0 length data frames. However in some cases EMPTY_BUFFER is used instead of the actual buffer that was written. This may mask writes of released buffers or otherwise invalid buffer objects. It is also possible that if the buffer is invalid AbstractCoalescingBufferQueue will not release the aggregated buffer nor fail the associated promise.

Modifications:
- DefaultHttp2FrameCodec should take care to fail the promise, even if releasing the data throws
- AbstractCoalescingBufferQueue should release any aggregated data and fail the associated promise if something goes wrong during aggregation

Result:
More correct handling of invalid buffers in HTTP/2 code.
2017-11-06 14:38:58 -08:00
Scott Mitchell
911b2acc50 HTTP/2 Child Channel reading and flushing
Motivation:
If a child channel's read is triggered outside the parent channel's read
loop then it is possible a WINDOW_UPDATE will be written, but not
flushed.
If a child channel's beginRead processes data from the inboundBuffer and
then readPending is set to false, which will result in data not being
delivered if in the parent's read loop and more data is attempted to be
delievered to that child channel.

Modifications:
- The child channel must force a flush if a frame is written as a result
of reading a frame, and this is not in the parent channel's read loop
- The child channel must allow a transition from dequeueing from
beginRead into the parent channel's read loop to deliver more data

Result:
The child channel flushes data when reading outside the parent's read
loop, and has frames delivered more reliably.
2017-10-26 10:06:22 -07:00
Lionel Li
424bb09d24 Http2StreamFrameToHttpObjectCodec should handle 100-Continue properly
Motivation:
Http2StreamFrameToHttpObjectCodec was not properly encoding and
decoding 100-Continue HttpResponse/Http2SettingsFrame properly. It was
encoding 100-Continue FullHttpResponse as an Http2SettingFrame with
endStream=true, causing the child channel to terminate. It was not
decoding 100-Continue Http2SettingsFrame (endStream=false) as
FullHttpResponse. This should be fixed as it would cause http2 child
stream to prematurely close, and could cause HttpObjectAggregator to
fail if it's in the pipeline.

Modification:
- Fixed encode() to properly encode 100-Continue FullHttpResponse as
  Http2SettingsFrame with endStream=false
- Reject 100-Continue HttpResponse that are NOT FullHttpResponse
- Fixed decode() to properly decode 100-Continue Http2SettingsFrame
  (endStream=false) as a FullHttpResponse
- made Http2StreamFrameToHttpObjectCodec sharable so that it can b used
  among child streams within the same Http2MultiplexCodec

Result:
Now Http2StreamFrameToHttpObjectCodec should be properly handling
100-Continue responses.
2017-10-25 06:56:02 +02:00