Commit Graph

547 Commits

Author SHA1 Message Date
Julien Hoarau
3e6b54bb59 Fix failing h2spec tests 8.1.2.1 related to pseudo-headers validation
Motivation:

According to the spec:
All pseudo-header fields MUST appear in the header block before regular
header fields. Any request or response that contains a pseudo-header
field that appears in a header block after
a regular header field MUST be treated as malformed (Section 8.1.2.6).

Pseudo-header fields are only valid in the context in which they are defined.
Pseudo-header fields defined for requests MUST NOT appear in responses;
pseudo-header fields defined for responses MUST NOT appear in requests.
Pseudo-header fields MUST NOT appear in trailers.
Endpoints MUST treat a request or response that contains undefined or
invalid pseudo-header fields as malformed (Section 8.1.2.6).

Clients MUST NOT accept a malformed response. Note that these requirements
are intended to protect against several types of common attacks against HTTP;
they are deliberately strict because being permissive can expose
implementations to these vulnerabilities.

Modifications:

- Introduce validation in HPackDecoder

Result:

- Requests with unknown pseudo-field headers are rejected
- Requests with containing response specific pseudo-headers are rejected
- Requests where pseudo-header appear after regular header are rejected
- h2spec 8.1.2.1 pass
2018-01-29 19:42:56 -08:00
Norman Maurer
c795e8897b Convert Http2Error.STREAM_CLOSED to ClosedChannelException when using child channels
Motivation:

We should convert Http2Exceptions that are produced because of STREAM_CLOSED to ClosedChannelException when hand-over to the child channel to make it more consistent with other transports.

Modifications:

- Check if STREAM_CLOSED is used and if so create a new ClosedChannelException (while preserve the original exception as cause) and use it in the child channel
- Ensure STREAM_CLOSED is used in DefaultHttp2RemoteFlowController when writes are failed because of a closed stream.
- Add testcase

Result:

More consistent and correct exception usage.
2018-01-29 17:50:29 -08:00
Norman Maurer
6e6edb59e7 Remove unused variable in DefaultHttp2StreamChannel
Motivation:

We should remove unused variable (was never read).

Modifications:

Remove unused variable (was never read).

Result:

Cleanup.
2018-01-29 11:32:22 +01:00
Thomas Segismont
7f23c34b55 ReadOnlyHttp2Headers.contains always ignores case for values
Motivation:

When checking if a value is present, ReadOnlyHttp2Headers always ignores
case for values.

RFC 7540 says: https://tools.ietf.org/html/rfc7540#section-8.1.2
"header field names are strings of ASCII characters that are compared in a case-insensitive fashion"

But there is no such constraint on header values

Modifications:

Updated ReadOnlyHttp2Headers.contains to compare header value in a
case-sensitive way.

Result:

ReadOnlyHttp2Headers compares header names in a case-insensitive way,
values in a case-sensitive way.
2018-01-27 20:29:40 +01:00
Thomas Segismont
0db652277f DefaultHttp2Headers#contains(CharSequence, CharSequence) does not work with String
Motivation:

If you test a header value providing a String, contains() returns false.
This is due to the implementation inherited from DefaultHeaders using
the JAVA_HASHER.

JAVA_HASHER.equals returns false because a is a String and b an
AsciiString.

Modifications:

DefaultHttp2Headers overrides contains and uses CASE_SENSITIVE_HASHER.

Result:

You can test a header value with any CharSequence implementation.
2018-01-27 20:27:50 +01:00
Scott Mitchell
614b9e0f25 Add tests for Http2MultiplexChannel close promise completion consistency with AbstractChannel
Motivation:
The completion order of promises in Http2MultiplexChannel#close should be consistent with that of AbstractChannel. Otherwise this may result in Future listeners seeing incorrect channel state.

Modifications:
Add tests cases.

Result:
Ensure consistent behavior between Http2MultiplexChannel and AbstractChannel.
2018-01-27 08:57:17 +01:00
Norman Maurer
b423a35783 Correctly handle multiple calls to DefaultHttp2StreamChannel.Unsafe.close(...)
Motivation:

Calling DefaultHttp2StreamChannel.Unsafe.close(...) multiple times should not fail.

Modification:

- Correctly handle multiple calls to DefaultHttp2StreamChannel.Unsafe.close(...)
- Complete closePromise and promise that is given to close(...) in the correct order.
- Add unit test

Result:

Fixes [#7628] and [#7641]
2018-01-27 08:55:35 +01:00
Norman Maurer
b1695fe17d Ensure async failures are correctly propagated to Http2LifecycleManager.onError(...) in all cases.
Motivation:

If DefaultHttp2ConnectionEncoder process outbound operation it sometimes missed to call Http2LifecycleManager.onError(...) when the operation was executed asynchronously.

Modifications:

Make best effort to update flags but still ensure failures are propageted to Http2LifecycleManager.onError(...) in all cases.

Result:

More consistent handling of errors.
2018-01-27 08:46:23 +01:00
Norman Maurer
c43dc3364b Cleanup Http2MultiplexCodec by removing out-dated TODO
Motivation:

Http2MultiplexCodec contains some TODO that is outdated.

Modifications:

Remove TODO which is outdated

Result:

Cleaner code.
2018-01-27 08:42:17 +01:00
Carl Mastrangelo
9ba942e59b Swap header check in ReadOnlyHttp2Headers
Motivation:
Pseudo headers are checked less frequently than normal headers, so
it is more efficient to check the latter first.

Modifications:
Swap the order of the check, and fix minor formatting

Result:
Possibly more efficient header checks
2018-01-26 19:01:02 -08:00
Norman Maurer
e975c5f6fe Reduce objects by directly implement interface in internal implementations of DefaultHttp2RemoteFlowController
Motivation:

We can just implement the interfaces directly and so reduce object creation in DefaultHttp2RemoteFlowController.

Modifications:

Directly implement the interfaces.

Result:

Less object creation.
2018-01-26 18:33:58 -08:00
Norman Maurer
49d1db4113 Http2MultiplexCodec.DefaultHttpStreamChannel.isOpen() / isActive() shoule be false when fireChannelActive() is called
Motivation:

When part of a HTTP/2 StreamChannel the Http2StreamChannel.isOpen() / isActive() should report false within a call to a ChannelInboundHandlers channelInactive() method.

Modifications:

Fullfill promise before call fireChannelInactive()

Result:

Correctly update state / promise before notify handlers. Fixes [#7638]
2018-01-26 17:34:37 -08:00
Thomas Segismont
bed74d8380 Method to check if a Http2 header is present and has a given value
Motivation:

With HTTP1, it's very easy to check if a header is present and has a
given value: you can simply invoke
io.netty.handler.codec.http.HttpHeaders#contains(java.lang.CharSequence, java.lang.CharSequence, boolean)

It is not possible to do the same with HTTP2. You have to get the list
of all headers (returned as String) and then iterate over it invoking
String#equals or String#equalsIgnoreCase

Modifications:

I've added io.netty.handler.codec.http2.Http2Headers#contains and
implemented it in DefaultHttp2Headers, EmptyHttp2Headers and ReadOnlyHttp2Headers.

Result:

You can use AsciiString constants to check if a header is present in a
consice and efficient manner.
2018-01-26 08:33:49 -08:00
Norman Maurer
1df5b02fd9 Do not fire outbound exception throught the pipeline when using Http2FrameCodec / Http2MultiplexCodec
Motivation:

Usually when using netty exceptions which happen for outbound operations should not be fired through the pipeline but only the ChannelPromise should be failed.

Modifications:

- Change Http2LifecycleManager.onError(...) to take also an boolean that indicate if the error was caused by an outbound operation
- Channel Http2ConnectionHandler.on*Error(...) methods to also take this boolean
- Change Http2FrameCodec to only fire exceptions through the pipeline if these are not outbound operations related
- Add unit test.

Result:

More consistent error handling when using Http2FrameCodec and Http2MultiplexCodec.
2018-01-25 13:42:28 -08:00
Bryce Anderson
8a095d0244 Http2MultiplexCodec should propagate unhandled Http2Frames down the pipeline
Motivation:

Http2MultiplexCodec swallows Http2PingFrames without releasing the payload, resulting in a memory leak.

Modification:

Send unhandled frames down the pipeline for consumption/disposal by another InboundChannelHandler.

Result:

Fixes #7607.
2018-01-25 13:26:45 -08:00
Scott Mitchell
4921f62c8a
HttpResponseStatus object allocation reduction
Motivation:
Usages of HttpResponseStatus may result in more object allocation then necessary due to not looking for cached objects and the AsciiString parsing method not being used due to CharSequence method being used instead.

Modifications:
- HttpResponseDecoder should attempt to get the HttpResponseStatus from cache instead of allocating a new object
- HttpResponseStatus#parseLine(CharSequence) should check if the type is AsciiString and redirect to the AsciiString parsing method which may not require an additional toString call
- HttpResponseStatus#parseLine(AsciiString) can be optimized and doesn't require and may not require object allocation

Result:
Less allocations when dealing with HttpResponseStatus.
2018-01-24 22:01:52 -08:00
Norman Maurer
b769ec0934 Reduce overhead of cancel flowcontrolled writes.
Motivation:

When we cancel the flowcontrolled writes we did create a new StreamException for each write that was enqueued. Creating Exceptions is very expensive due of filling the stacktrace.

Modifications:

Only create the StreamException once and reuse the same for all the flowcontrolled writes (per stream).

Result:

Less expensive to cancel flowcontrolled writes.
2018-01-24 19:42:13 +01:00
Scott Mitchell
46dac128f7
Http2FrameCodec WindowUpdate bug
Motivation:
Http2FrameCodec increases the initialWindowSize when the user attempts to increase the connection flow control window. The initialWindowSize should only be touched as a result of a SETTINGS frame, and otherwise may result in flow control getting out of sync with our peer.

Modifications:
- Http2FrameCodec shouldn't update the initialWindowSize when a WindowUpdateFrame is written on the connection channel

Result:
More correct WindowUpdate processing.
2018-01-22 11:17:56 -08:00
Norman Maurer
ea58dc7ac7 [maven-release-plugin] prepare for next development iteration 2018-01-21 12:53:51 +00:00
Norman Maurer
96c7132dee [maven-release-plugin] prepare release netty-4.1.20.Final 2018-01-21 12:53:34 +00:00
Scott Mitchell
322a062185 HpackDecoderTest cleanup
Motivation:
public ExpectedException fields should be final.
setUp method signature has a throws, but it isn't necessary
2018-01-19 17:26:06 +01:00
Julien Hoarau
61dba95091 Fix h2spec test 4.3 about invalid header block
Motivation:

HPackDecoder works on entire header block, we shouldn't encounter
incomplete header fields. If we do we should treat it as
a decoding error and according to the specification:

A decoding error in a header block MUST be treated as
a connection error (Section 5.4.1) of type COMPRESSION_ERROR.

Modifications:

* Check final state in HpackDecoder once we've decoded all the data.

Result:

* Throw a connection error if we receive incomplete header fields
* H2spec 4.3 tests all passes
2018-01-18 11:01:20 -08:00
Scott Mitchell
d6e600548b HTTP/2 Remove Http2FrameStream#CONNECTION_STREAM
Motivation:
Http2FrameStream#CONNECTION_STREAM is required to identify the
connection stream. However this leads to inconsistent usage from a user
perspective. When a user creates a Http2Frame for a non-connection
stream, the Http2MultiplexCodec automatically sets the stream, and the
user is never exposed to the Http2FrameStream object. However when the
user writes a Http2Frame for a connection stream they are required to
set the Http2FrameStream object. We can remove the Http2FrameStream#CONNECTION_STREAM
and keep the Http2FrameStream object internal, and therefore consistent
between the connection and non-connection use cases.

Modifications:
- Remove Http2FrameStream#CONNECTION_STREAM
- Update Http2FrameCodec to handle Http2Frame#stream() which returns
null

Result:
More consistent usage on http2 parent channel and http2 child channel.
2018-01-14 13:31:30 +01:00
Scott Mitchell
adf2596c36 Http2FrameCodecTest increase timeout
Motivation:
Http2FrameCodecTest#newOutboundStream has a timeout of 1 second and has been observed to timeout on CI servers.

Modifications:
- Increase the timeout to 5 seconds

Result:
Less false positive test failures on CI servers.
2017-12-22 19:32:18 +01:00
Scott Mitchell
336cee9b1f HpackDecoder#addHeader has an unused parameter
Motivation:
HpackDecoder#addHeader takes in the streamId as a parameter but no longer uses it.

Modifications:
- Remove the streamId parameter from HpackDecoder#addHeader

Result:
Less unused parameters in HpackDecoder.
2017-12-22 07:17:24 +01:00
Julien Hoarau
6b033c51a5 Add 32 bytes overhead per header entry when calculating headers length in HPackDecoder
Motivation:

According to the HTTP/2 Spec:

SETTINGS_MAX_HEADER_LIST_SIZE (0x6): This advisory setting informs a
peer of the maximum size of header list that the sender is
prepared to accept, in octets. The value is based on the
uncompressed size of header fields, including the length of the
name and value in octets plus an overhead of 32 octets for each
header field.

We were accounting for the 32 bytes when encoding in HpackEncoder,
but not when decoding in HPackDecoder.

Modifications:

- Add 32 bytes to the header list length for each entry when decoding
with HPackDecoder.

Result:

- We account for the 32 bytes overhead by header entry in HPackDecoder
2017-12-21 17:50:16 -08:00
Norman Maurer
942b993f2b Only enable validation of headers if original headers were validating as well.
Motiviation:

In our replace(...) methods we always used validation for the newly created headers while the original headers may not use validation at all.

Modifications:

- Only use validation if the original headers used validation as well.
- Ensure we create a copy of the headers in replace(...).

Result:

Fixes [#5226]
2017-12-21 07:32:29 +01:00
Scott Mitchell
144716f668
HTTP/2 support pending data larger than Integer.MAX_VALUE
Motivation:
Currently the remote flow controller limits the maximum amount of pending data to Integer.MAX_VALUE. The overflow handling is also not very graceful in that it may lead to infinite loops, or otherwise no progress being made.

Modifications:
- StreamByteDistributor and RemoteFlowController should support pending bytes of type long.

Result:
Fixes https://github.com/netty/netty/issues/4283
2017-12-20 08:55:15 -08:00
Moses Nakamura
94ab0dc442 codec-http2: Better keep track of nameLength in HpackDecoder.decode
Motivation:

http/2 counts header sizes somewhat inconsistently.  Sometimes, headers
which are substantively less than the header list size will be measured
as longer than the header list size.

Modifications:

Keep better track of the nameLength of a given name, so that we don't
accidentally end up reusing a nameLength.

Result:

More consistent measurement of header list size.

Fixes #7511.
2017-12-18 17:41:09 -08:00
Norman Maurer
264a5daa41 [maven-release-plugin] prepare for next development iteration 2017-12-15 13:10:54 +00:00
Norman Maurer
0786c4c8d9 [maven-release-plugin] prepare release netty-4.1.19.Final 2017-12-15 13:09:30 +00:00
Norman Maurer
b2bc6407ab [maven-release-plugin] prepare for next development iteration 2017-12-08 09:26:15 +00:00
Norman Maurer
96732f47d8 [maven-release-plugin] prepare release netty-4.1.18.Final 2017-12-08 09:25:56 +00:00
Moses Nakamura
0cac1a6c8c H2C upgrades should be ineligible for flow control (#7400)
H2C upgrades should be ineligible for flow control

Motivation:

When the h2c upgrade request is too big, the Http2FrameCodec complains
it's too big for flow control reasons, even though it's ineligible for
flow control.

Modifications:

Specially mark upgrade streams and make Http2FrameCodec know not to try
to flow control on those streams.

Result:

Servers won't barf when they receive an upgrade request with a fat
payload.

[Fixes #7280]
2017-12-07 16:46:16 -08:00
Scott Mitchell
7cced5576f Http2ConnectionHandler Http2ConnectionPrefaceAndSettingsFrameWrittenEvent propagation
Motivation:
Http2ConnectionHandler uses ctx.fireUserEvent to propagate the Http2ConnectionPrefaceAndSettingsFrameWrittenEvent through the pipeline. This will propagate the event to the next inbound handler in the pipeline. If the user extends Http2ConnectionHandler the Http2ConnectionPrefaceAndSettingsFrameWrittenEvent may be missed and initialization dependent upon this event will not be run.

Modifications:
- Http2ConnectionHandler should use userEventTriggered instead of ctx.fireUserEvent

Result:
Classes that extend Http2ConnectionHandler will see the Http2ConnectionPrefaceAndSettingsFrameWrittenEvent user event.
2017-12-02 08:23:28 +01:00
Tomasz Jędrzejewski
e8540c2b7a Adding stable JDK9 module names that follow reverse-DNS style
Automatic-Module-Name entry provides a stable JDK9 module name, when Netty is used in a modular JDK9 applications. More info: http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html

When Netty migrates to JDK9 in the future, the entry can be replaced by actual module-info descriptor.

Modification:

The POM-s are configured to put the correct module names to the manifest.

Result:

Fixes #7218.
2017-11-29 11:50:24 +01:00
Scott Mitchell
7d213240ca HttpConversionUtil TE filtering robustness
Motivation:
HttpConversionUtil#toHttp2Headers has special code to filter the TE header name. However this filtering code may result in adding the <TE, TRAILERS> tuple in scenarios that are not appropriate. For example if a value containing trailers is seen it will be added, but the value could not actually be equal to trailers. Also CSV values are not supported.

Modifications:
- Account for CSV header values
- Account for the value containing 'trailers' but not actually being equal to 'trailers'

Result:
More robust parsing of the TE header.
2017-11-22 08:45:11 +01:00
Scott Mitchell
a3e41ba6eb HttpConversionUtils avoid intermediate collection allocation
Modifications:
HttpConversionUtil#toLowercaseMap requires an intermediate List to be allocated. This can be avoided with the recently added value iterator methods.

Modifications:
- Use HttpHeaders#valueCharSequenceIterator instead of getAll

Result:
Less intermediate object allocation and copying.
2017-11-20 14:05:03 -08:00
Scott Mitchell
e6126215e0 DefaultHttp2FrameWriter reduce object allocation
Motivation:
DefaultHttp2FrameWriter#writeData allocates a DataFrameHeader for each write operation. DataFrameHeader maintains internal state and allocates multiple slices of a buffer which is a maximum of 30 bytes. This 30 byte buffer may not always be necessary and the additional slice operations can utilize retainedSlice to take advantage of pooled objects. We can also save computation and object allocations if there is no padding which is a common case in practice.

Modifications:
- Remove DataFrameHeader
- Add a fast path for padding == 0

Result:
Less object allocation in DefaultHttp2FrameWriter
2017-11-20 08:10:59 -08:00
Scott Mitchell
a8bb9dc180 AbstractByteBuf readSlice bound check bug
Motivation:
AbstractByteBuf#readSlice relied upon the bounds checking of the slice operation in order to detect index out of bounds conditions. However the slice bounds checking operation allows for the slice to go beyond the writer index, and this is out of bounds for a read operation.

Modifications:
- AbstractByteBuf#readSlice and AbstractByteBuf#readRetainedSlice should ensure the desired amount of bytes are readable before taking a slice

Result:
No reading of undefined data in AbstractByteBuf#readSlice and AbstractByteBuf#readRetainedSlice.
2017-11-18 09:03:42 +01:00
Norman Maurer
78522cf3f0 Remove unused allocation introduced by d976dc108d 2017-11-17 17:19:36 +01:00
Moses Nakamura
d976dc108d codec-http2: Improve h1 to h2 header conversion
Motivation:

Netty could handle "connection" or "te" headers more gently when
converting from http/1.1 to http/2 headers.  Http/2 headers don't
support single-hop headers, so when we convert from http/1.1 to http/2,
we should drop all single-hop headers.  This includes headers like
"transfer-encoding" and "connection", but also the headers that
"connection" points to, since "connection" can be used to designate
other headers as single-hop headers.  For the "te" header, we can more
permissively convert it by just dropping non-conforming headers (ie
non-"trailers" headers) which is what we do for all other headers when
we convert.

Modifications:

Add a new blacklist to the http/1.1 to http/2 conversion, which is
constructed from the values of the "connection" header, and stop
throwing an exception when a "te" header is passed with a non-"trailers"
value.  Instead, drop all values except for "trailers".  Add unit tests
for "connection" and "te" headers when converting from http/1.1 to http/2.

Result:

This will improve the h2c upgrade request, and also conversions from
http/1.1 to http/2.  This will simplify implementing spec-compliant
http/2 servers that want to share code between their http/1.1 and http/2
implementations.

[Fixes #7355]
2017-11-17 09:09:52 +01:00
Norman Maurer
188ea59c9d [maven-release-plugin] prepare for next development iteration 2017-11-08 22:36:53 +00:00
Norman Maurer
812354cf1f [maven-release-plugin] prepare release netty-4.1.17.Final 2017-11-08 22:36:33 +00:00
Ning Sun
73e8122fc1 Fix sharable check logic
Motivation:

There is an logic issue when checking if ChannelHandler is sharable.

Modification:

Corrected || to &&
2017-11-08 08:36:07 -08:00
Scott Mitchell
35b0cd58fb HTTP/2 write of released buffer should not write and should fail the promise
Motivation:
HTTP/2 allows writes of 0 length data frames. However in some cases EMPTY_BUFFER is used instead of the actual buffer that was written. This may mask writes of released buffers or otherwise invalid buffer objects. It is also possible that if the buffer is invalid AbstractCoalescingBufferQueue will not release the aggregated buffer nor fail the associated promise.

Modifications:
- DefaultHttp2FrameCodec should take care to fail the promise, even if releasing the data throws
- AbstractCoalescingBufferQueue should release any aggregated data and fail the associated promise if something goes wrong during aggregation

Result:
More correct handling of invalid buffers in HTTP/2 code.
2017-11-06 14:38:58 -08:00
Scott Mitchell
911b2acc50 HTTP/2 Child Channel reading and flushing
Motivation:
If a child channel's read is triggered outside the parent channel's read
loop then it is possible a WINDOW_UPDATE will be written, but not
flushed.
If a child channel's beginRead processes data from the inboundBuffer and
then readPending is set to false, which will result in data not being
delivered if in the parent's read loop and more data is attempted to be
delievered to that child channel.

Modifications:
- The child channel must force a flush if a frame is written as a result
of reading a frame, and this is not in the parent channel's read loop
- The child channel must allow a transition from dequeueing from
beginRead into the parent channel's read loop to deliver more data

Result:
The child channel flushes data when reading outside the parent's read
loop, and has frames delivered more reliably.
2017-10-26 10:06:22 -07:00
Lionel Li
424bb09d24 Http2StreamFrameToHttpObjectCodec should handle 100-Continue properly
Motivation:
Http2StreamFrameToHttpObjectCodec was not properly encoding and
decoding 100-Continue HttpResponse/Http2SettingsFrame properly. It was
encoding 100-Continue FullHttpResponse as an Http2SettingFrame with
endStream=true, causing the child channel to terminate. It was not
decoding 100-Continue Http2SettingsFrame (endStream=false) as
FullHttpResponse. This should be fixed as it would cause http2 child
stream to prematurely close, and could cause HttpObjectAggregator to
fail if it's in the pipeline.

Modification:
- Fixed encode() to properly encode 100-Continue FullHttpResponse as
  Http2SettingsFrame with endStream=false
- Reject 100-Continue HttpResponse that are NOT FullHttpResponse
- Fixed decode() to properly decode 100-Continue Http2SettingsFrame
  (endStream=false) as a FullHttpResponse
- made Http2StreamFrameToHttpObjectCodec sharable so that it can b used
  among child streams within the same Http2MultiplexCodec

Result:
Now Http2StreamFrameToHttpObjectCodec should be properly handling
100-Continue responses.
2017-10-25 06:56:02 +02:00
Lionel Li
baf273aea8 Trigger user event when H2 conn preface & SETTINGS frame are sent
Motivation:
Previously client Http2ConnectionHandler trigger a user event
immediately when the HTTP/2 connection preface is sent. Any attempt to
immediately send a new request could cause the server to terminate the
connection, as it might not have received the SETTINGS frame from the
client. Per RFC7540 Section 3.5, the preface "MUST be followed by a
SETTINGS frame (Section 6.5), which MAY be empty."
(https://tools.ietf.org/html/rfc7540#section-3.5)

This event could be made more meaningful if it also indicates that the
initial client SETTINGS frame has been sent to signal that the channel
is ready to send new requests.

Modification:
- Renamed event to Http2ConnectionPrefaceAndSettingsFrameWrittenEvent.
- Modified Http2ConnectionHandler to trigger the user event only if it
  is a client and it has sent both the preface and SETTINGS frame.

Result:
It is now safe to use the event as an indicator that the HTTP/2
connection is ready to send new requests.
2017-10-24 09:17:06 +02:00
Norman Maurer
55b501d0d4 Correctly update Channel writability when queueing data in SslHandler.
Motivation:

A regression was introduced in 86e653e which had the effect that the writability was not updated for a Channel while queueing data in the SslHandler.

Modifications:

- Factor out code that will increment / decrement pending bytes and use it in AbstractCoalescingBufferQueue and PendingWriteQueue
- Add test-case

Result:

Channel writability changes are triggered again.
2017-10-24 09:13:15 +02:00
Idel Pivnitskiy
4793daa589 Make Comparators Serializable
Motivation:

Objects of java.util.TreeMap or java.util.TreeSet will become
non-Serializable if instantiated with Comparators, which are not also
 Serializable. This can result in unexpected and difficult-to-diagnose
 bugs.

Modifications:

Implements Serializable for all classes, which implements Comparator.

Result:

Proper Comparators which will not force collections to
non-Serializable mode.
2017-10-22 03:40:28 +02:00
Idel Pivnitskiy
50a067a8f7 Make methods 'static' where it possible
Motivation:

Even if it's a super micro-optimization (most JVM could optimize such
 cases in runtime), in theory (and according to some perf tests) it
 may help a bit. It also makes a code more clear and allows you to
 access such methods in the test scope directly, without instance of
 the class.

Modifications:

Add 'static' modifier for all methods, where it possible. Mostly in
test scope.

Result:

Cleaner code with proper 'static' modifiers.
2017-10-21 14:59:26 +02:00
Idel Pivnitskiy
558097449c Add missed 'serialVersionUID' field for Serializable classes
Motivation:

Without a 'serialVersionUID' field, any change to a class will make
previously serialized versions unreadable.

Modifications:

Add missed 'serialVersionUID' field for all Serializable
classes.

Result:

Proper deserialization of previously serialized objects.
2017-10-21 14:41:18 +02:00
Cory Benfield
1b0a545921 Do not send Content-Length: 0 on 101 responses.
Motivation:

During code read of the Netty codebase I noticed that the Netty
HttpServerUpgradeHandler unconditionally sets a Content-Length: 0
header on 101 Switching Protocols responses. This explicitly
contravenes RFC 7230 Section 3.3.2 (Content-Length), which notes
that:

    A server MUST NOT send a Content-Length header field in any
    response with a status code of 1xx (Informational) or 204
    (No Content).

While it is unlikely that any client will ever be confused by
this behaviour, there is no reason to contravene this part of the
specification.

Modifications:

Removed the line of code setting the header field and changed the
only test that expected it to be there.

Result:

When performing the server portion of HTTP upgrade, the 101
Switching Protocols response will no longer contain a
Content-Length: 0 header field.
2017-10-21 14:36:19 +02:00
Norman Maurer
625a7426cd [maven-release-plugin] prepare for next development iteration 2017-09-25 06:12:32 +02:00
Norman Maurer
f57d8f00e1 [maven-release-plugin] prepare release netty-4.1.16.Final 2017-09-25 06:12:16 +02:00
Lionel Li
e069079aff Adapt Http2ServerDowngrader to work with clients
Motivation:
Http2ServerDowngrader is specifically built for server channels where
inbound Http2StreamFrames are converted into HttpRequests, and outbound
HttpResponses are converted into Http2StreamFrames. It can be easily
made to be more generic to work with client channels where inbound
Http2StreamFrames are converted into HttpResponses, and outbound
HttpRequests are converted into Http2StreamFrames.

Modification:
- Renamed Http2ServerDowngrader to a more general
  Http2StreamFrameToHttpObjectCodec
- Made it take in an "isServer" parameter to determine whether encoding
  inbound Http2StreamFrames should create HttpRequests (for server) or
  HttpResponses (for client)
- Norman fixed a leak in the unit test. Thanks! :-)

Result:
Now Http2StreamFrameToHttpObjectCodec can be used to translate
Http2StreamFrame to HttpObject for both server and client.
2017-09-22 11:04:43 -07:00
Moses Nakamura
1ff2e1fb5d Match Http2ClientUpgradeCodec to the new upgrade policy
Motivation:

We changed Http2ConnectionHandler to expect the upgrade method to be
called *after* we send the preface (ie add the handler to the pipeline)
but we forgot to change the Http2ClientUpgradeCodec to match the new
policy.  This meant that client-side h2c upgrades failed.

Modifications:

Reverse sending the preface and calling the upgrade method to match the
new policy.

Result:

Clients can initiate h2c upgrades successfully.
2017-09-20 12:42:43 -07:00
durigon
282aa35682 Fix NPE in InboundHttp2ToHttpAdapter
Motiviation:

At the moment an NPE is thrown if someone tries to use the InboundHttp2ToHttpAdapter.

Modifications:
- Ensure the status was null in "InboundHttp2ToHttpAdapter::onPushPromiseRead" before calling "HttpConversionUtil.parseStatus" methods.
- Fix setting status to OK in "InboundHttp2ToHttpAdapter::onPushPromiseRead".

Result:
Fixes [#7214].
2017-09-17 09:07:11 -07:00
Scott Mitchell
44bb3b6f3a DefaultHeaders value iterator
Motivation:
The Headers interface supports an interface to get all the headers values corresponding to a particular name. This API returns a List which requires intermediate storage and increases GC pressure.

Modifications:
- Add a method which returns an iterator over all the values for a specific name

Result:
Ability to iterator over values for a specific name with no intermediate collection.
2017-09-16 16:46:19 -07:00
Norman Maurer
bf0a53179a Correctly update writability state of Http2StreamChannel created by Http2MultiplexCodec.
Motivation:

We missed to mark the Http2StreamChannel as writable in some cases which could lead to the situation that a Channel never becomes writable. Also when a Http2StreamChannel was created we always marked it non-writable at the beginning which means if the user will only start writing once the Channel becomes writable it will never happen as it only became writable after the first header was written.

Modifications:

- Correctly handle updates for writability in all cases
- Change unit tests to cover this.

Result:

Fixes [#7179].
2017-09-14 08:25:31 -07:00
Norman Maurer
15611dadbb Fix NPE when using Http2ServerUpgradeCodec with Http2FrameCodec and Http2MultiplexCodec
Motiviation:

At the moment an NPE is thrown if someone tries to use the Http2ServerUpgradeCodec with Http2FrameCodec and Http2MultiplexCodec.

Modifications:

- Ensure the handler was added to the pipeline before calling on*Upgrade(...) methods.
- Add tests
- Fix adding of handlers after upgrade.

Result:

Fixes [#7173].
2017-09-14 08:23:53 -07:00
Nikolay Fedorovskikh
de9e666493 Fix hashCode() in Http2StreamChannelId
Motivation:
In `Http2StreamChannelId` a `hashCode()` is not consistent with `equals()`.

Modifications:
Make a `Http2StreamChannelId.hashCode()` consistent with `equals()`.

Result:
Faster hash map's operations where the Http2StreamChannelId as keys.
2017-09-08 10:38:16 +02:00
Nikolay Fedorovskikh
e500755086 Remove double comparing of content out of the DefaultHttp2GoAwayFrame.equals()
Motivation:
In `DefaultHttp2GoAwayFrame.equals()` a content compared twice: explicitly and in the `super` method.

Modifications:
Remove explicit content comparision.
Make `hashCode()` consistent with `equals()`.

Result:
A `DefaultHttp2GoAwayFrame.equals()` work faster.
2017-09-07 08:30:43 +02:00
Norman Maurer
870b5f5e4b Not add inboundStreamHandler for outbound streams created by Http2MultiplexCodec.
Motivation:

We must not add the inboundStreamHandler for outbound streams creates by Http2MultiplexCodec as the user will specify a handler via Http2StreamChannelBootstrap.

Modifications:

- Check if the stream is for outbound and if so not add the inboundStreamHandler to the pipeline
- Update tests so this is covered.

Result:

Fixes [#7178]
2017-09-06 08:37:29 +02:00
Norman Maurer
5c572f0f63 Ensure the tests complete on java7 and java9 as well.
Motivation:

379ac890f4 introduced the usage of the inline mock maker. This unfortunally not work on java7 and java9.

Modifications:

Just use reflection to create the event for now.

Result:

Netty tests pass again on java7 and java9 as well.
2017-09-04 20:20:04 +02:00
Norman Maurer
379ac890f4 Fix reference count issue when using Http2FrameCodec / Http2MultiplexCodec with HttpServerUpgradeHandler
Motivation:

When using  Http2FrameCodec / Http2MultiplexCodec with HttpServerUpgradeHandler reference count exception will be triggered.

Modifications:

- Correctly retain before calling InboundHttpToHttp2Adapter.handle
- Add unit test

Result:

Fixes [#7172].
2017-09-04 13:30:18 +02:00
Norman Maurer
b967805f32 [maven-release-plugin] prepare for next development iteration 2017-08-24 15:38:22 +02:00
Norman Maurer
da8e010a42 [maven-release-plugin] prepare release netty-4.1.15.Final 2017-08-24 15:37:59 +02:00
Nikolay Fedorovskikh
ba27456653 Use the index-based AsciiString constructor instead of substring()
Motivation:
The construction `new AsciiString(string.substring(...))` can be replaced with the `new AsciiString(string, start, length)` to avoid extra allocation.

Modifications:
Apply the described replacement in `HttpConversionUtil#setHttp2Authority`.

Result:
Less allocations.
2017-08-18 09:48:05 +02:00
Nikolay Fedorovskikh
4875a2aad4 Immediate caching the strings wrapped to AsciiString
Motivation:
The `AsciiString#toString` method calculate string value and cache it into field. If an `AsciiString` created from the `String` value, we can avoid rebuilding strings if we cache them immediately when creating `AsciiString`. It would be useful for constants strings, which already stored in the JVMs string table, or in cases where an unavoidable `#toString `method call is assumed.

Modifications:
- Add new static method `AsciiString#cache(String)` which save string value into cache field.
- Apply a "benign" data race in the `#hashCode` and `#toString` methods.

Result:
Less memory usage in some `AsciiString` use cases.
2017-08-15 06:22:14 +02:00
Norman Maurer
08284dbbcd Ensure we call promise.setUncancellable() before trying to process in DefaultHttp2StreamChannel.
Motivation:

We should call promise.setUncancellable() in DefaultHttp2StreamChannel.Unsafe impl to detect if the operation was cancelled.

Modifications:

Add promise.setUncancellable() calls

Result:

More correct handling of cancelled promises
2017-08-12 09:19:36 +02:00
Norman Maurer
74f24a5c19 Finish work on http2 child channel implementation and http2 frame api.
Motivation:

Our http2 child channel implementation was not 100 % complete and had a few bugs. Beside this the performance overhead was non-trivial.

Modifications:

There are a lot of modifications, the most important....
  * Http2FrameCodec extends Http2ConnectionHandler and Http2MultiplexCodec extends Http2FrameCodec to reduce performance heads and inter-dependencies on handlers in the pipeline
  * Correctly handle outbound flow control for child channels
  * Support unknow frame types in Http2FrameCodec and Http2MultiplexCodec
  * Use a consistent way how to create Http2ConnectionHandler, Http2FrameCodec and Http2MultiplexCodec (via a builder)
  * Remove Http2Codec and Http2CodecBuilder as the user should just use Http2MultipleCodec and Http2MultiplexCodecBuilder now
  * Smart handling of flushes from child channels to reduce overhead
  * Reduce object allocations
  * child channels always use the same EventLoop as the parent Channel to reduce overhead and simplify implementation.
  * Not extend AbstractChannel for the child channel implementation to reduce overhead in terms of performance and memory usage
  * Remove Http2FrameStream.managedState(...) as the user of the child channel api should just use Channel.attr(...)

Result:

Http2MultiplexCodec (and so child channels) and Http2FrameCodec are more correct, faster and more feature complete.
2017-08-11 12:41:28 +02:00
buchgr
3a2b462a67 Remove the concept of pending streams. The close future can only be accessed once a stream is active. 2017-08-11 12:41:28 +02:00
buchgr
5380c7c3e3 HTTP/2 Child Channel and FrameCodec Feature Parity.
Motivation:

This PR (unfortunately) does 4 things:
1) Add outbound flow control to the Http2MultiplexCodec:
   The HTTP/2 child channel API should interact with HTTP/2 outbound/remote flow control. That is,
   if a H2 stream used up all its flow control window, the corresponding child channel should be
   marked unwritable and a writability-changed event should be fired. Similarly, a unwritable
   child channel should be marked writable and a writability-event should be fired, once a
   WINDOW_UPDATE frame has been received. The changes are (mostly) contained in ChannelOutboundBuffer,
   AbstractHttp2StreamChannel and Http2MultiplexCodec.

2) Introduce a Http2Stream2 object, that is used instead of stream identifiers on stream frames. A
   Http2Stream2 object allows an application to attach state to it, and so a application handler
   no longer needs to maintain stream state (i.e. in a map(id -> state)) himself.

3) Remove stream state events, which are no longer necessary due to the introduction of Http2Stream2.
   Also those stream state events have been found hard and complex to work with, when porting gRPC
   to the Http2FrameCodec.

4) Add support for HTTP/2 frames that have not yet been implemented, like PING and SETTINGS. Also add
   a Http2FrameCodecBuilder that exposes options from the Http2ConnectionHandler API that couldn't else
   be used with the frame codec, like buffering outbound streams, window update ratio, frame logger, etc.

Modifications:

1) A child channel's writability and a H2 stream's outbound flow control window interact, as described
   in the motivation. A channel handler is free to ignore the channel's writability, in which case the
   parent channel is reponsible for buffering writes until a WINDOW_UPDATE is received.

   The connection-level flow control window is ignored for now. That is, a child channel's writability
   is only affected by the stream-level flow control window. So a child channel could be marked writable,
   even though the connection-level flow control window is zero.

2) Modify Http2StreamFrame and the Http2FrameCodec to take a Http2Stream2 object intstead of a primitive
   integer. Introduce a special Http2ChannelDuplexHandler that has newStream() and forEachActiveStream()
   methods. It's recommended for a user to extend from this handler, to use those advanced features.

3) As explained in the documentation, a new inbound stream active can be detected by checking if the
   Http2Stream2.managedState() of a Http2HeadersFrame is null. An outbound stream active can be detected
   by adding a listener to the ChannelPromise of the write of the first Http2HeadersFrame. A stream
   closed event can be listened to by adding a listener to the Http2Stream2.closeFuture().

4) Add a simple Http2FrameCodecBuilder and implement the missing frame types.

Result:

1) The Http2MultiplexCodec supports outbound flow control.
2) The Http2FrameCodec API makes it easy for a user to manage custom stream specific state and to create
   new outbound streams.
3) The Http2FrameCodec API is much cleaner and easier to work with. Hacks like the ChannelCarryingHeadersFrame
   are no longer necessary.
4) The Http2FrameCodec now also supports PING and SETTINGS frames. The Http2FrameCodecBuilder allows the Http2FrameCodec
   to use some of the rich features of the Http2ConnectionHandler API.
2017-08-11 12:41:28 +02:00
Norman Maurer
348745608f Fix incorrect javadocs in Http2RemoteFlowController
Motivation:

The javadocs of Http2RemoteFlowController.isWritable(...) are incorrect.

Modifications:

Update javadocs to reflect reality.

Result:

Correct javadocs.
2017-08-08 07:47:18 +02:00
Scott Mitchell
0f8ecdcad5 Http2FrameLogger frame labels incorrect
Motivation:
The labels identifying the frame types in Http2FrameLogger are not always correct.

Modification:
- Correct the string labels to indicate the right frame type in Http2FrameLogger

Result:
Logs are more correct.
2017-08-07 10:24:17 -07:00
Norman Maurer
d56f403c69 First call channelReadComplete(...) before flush(...) for better performance
Motivation:

In Http2ConnectionHandler we call flush(...) in channelReadComplete(...) to ensure we update the flow-controller and write stuff to the remote peer. We should better flip the order and so may be able to pick up more bytes.

Modifications:

Change order of calls.

Result:

Better performance
2017-08-04 11:55:35 +02:00
Norman Maurer
52f384b37f [maven-release-plugin] prepare for next development iteration 2017-08-02 12:55:10 +00:00
Norman Maurer
8cc1071881 [maven-release-plugin] prepare release netty-4.1.14.Final 2017-08-02 12:54:51 +00:00
chhsiao90
8320a45c15 Configures HTTP2 pipeline with more proper way
Motivation:

When we use pipeline.replace and we still had ongoing inbound, then
there will be some problem that inbound message would go to wrong
handlers. So we add handler first, and remove self after add, so that
the next handler will be the correct one.

Modifications:

Uses remove after addAfter instead of replace.

Result:

Fixed #6881
2017-08-02 06:58:55 +02:00
Norman Maurer
2988fb8eeb Ensure Http2FrameCodec uses Http2Settings.defaultSettings()
Motivation:

Http2FrameCodec should use Http2Settings.defaultSettings() when no Http2Settings were specified by the user.

Modifications:

Replace new Http2Settings() with Http2Settings.defaultSettings()

Result:

Use correct Http2Settings by default when using Http2FrameCodec in all cases.
2017-08-01 07:07:05 +02:00
Vladimir Gordiychuk
fe8ecea366 Http2FrameLogger avoid hex dump of the ByteBufs when log disabled
Motivation:

Currentry logger create hex dump even if log write will not apply.
It's unecessary GC overhead.

Modifications:

Restore optimization from #3492

Result:

Fixes #7025
2017-07-26 21:21:04 +02:00
Spencer Fang
732b145842 Http2ConnectionHandler: allow graceful shutdown to wait forever
Motivation:

There should be a way to allow graceful shutdown to wait for all open streams to close without a timeout. Using gracefulShutdownTimeoutMillis with a large value is a bit of a hack, and has a gotcha that sufficiently large values will overflow the long, resulting in a ClosingChannelFutureListener that executes immediately.

Modification:

Allow to use gracefulShutdownTimeoutMillis(-1) to express waiting until all streams are closed.

Result:

We can now shutdown the connection without a forced timeout.
2017-07-26 20:40:24 +02:00
Scott Mitchell
a91df58ca1 HTTP/2 enforce HTTP message flow
Motivation:
codec-http2 currently does not strictly enforce the HTTP/1.x semantics with respect to the number of headers defined in RFC 7540 Section 8.1 [1]. We currently don't validate the number of headers nor do we validate that the trailing headers should indicate EOS.

[1] https://tools.ietf.org/html/rfc7540#section-8.1

Modifications:
- DefaultHttp2ConnectionDecoder should only allow decoding of a single headers and a single trailers
- DefaultHttp2ConnectionEncoder should only allow encoding of a single headers and optionally a single trailers

Result:
Constraints of RFC 7540 restricting the number of headers/trailers is enforced.
2017-07-19 13:37:23 -07:00
Norman Maurer
dbd82e07b1 Let Http2ServerUpgradeCodec support Http2FrameCodec
Motivation:

Http2ServerUpgradeCodec should support Http2FrameCodec.

Modifications:

- Add support for Http2FrameCodec
- Add example that uses Http2FrameCodec

Result:

More flexible use of Http2ServerUpgradeCodec
2017-07-19 11:12:10 +02:00
Norman Maurer
2a376eeb1b [maven-release-plugin] prepare for next development iteration 2017-07-06 13:24:06 +02:00
Norman Maurer
c7f8168324 [maven-release-plugin] prepare release netty-4.1.13.Final 2017-07-06 13:23:51 +02:00
Scott Mitchell
449befa003 Workaround IBM's J9 JVM getSupportedCipherSuites() returning SSL_ prefix cipher names
Motivation:
IBM's J9 JVM utilizes a custom cipher naming scheme with SSL_ prefix [1] instead of the TLS_ prefix defined by TLS RFCs and the JSSE cihper suite names [2]. IBM's documentation says that the SSL_ prefix are "interchangeable" with cipher names with the TLS_ prefix [1]. To work around this issue we parse the supported cipher list and see an SSL_ prefix we can also add the same cipher with the TLS_ prefix. For more details see a discussion on IBM's forums [3] and IBM's issue tracker [4].

[1] https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/ciphersuites.html
[2] http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#ciphersuites
[3] https://www.ibm.com/developerworks/community/forums/html/topic?id=9b5a56a9-fa46-4031-b33b-df91e28d77c2
[4] https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=71770

Modifications:
- When parsing the supported cipher list to get the supported ciphers and we encounter a SSL_ prefix we should also add a TLS_ prefix cipher.
- Remove SSL_ prefix ciphers from Http2SecurityUtil.

Result:
Work around for IBM JVM's custom naming scheme covers more cases for supported cipher suites.
2017-07-05 09:05:42 -04:00
Scott Mitchell
bc46a99eaa DefaultHttp2ConnectionEncoder#writeHeaders shouldn't send GO_AWAY if stream is closed
Motivation:
DefaultHttp2ConnectionEncoder#writeHeaders attempts to find a stream object, and if one doesn't exist it tries to create one. However in the event that the local endpoint has received a RST_STREAM frame before writing the response headers we attempt to create a stream. Since this stream ID is for the incorrect endpoint we then generate a GO_AWAY for what appears to be a protocol error, but can instead be failed locally.

Modifications:
- Just fail the local promise in the above situation instead of sending a GO_AWAY

Result:
Less severe consequences if the server asynchronously sends headers after a RST_STREAM has been received.
Fixes https://github.com/netty/netty/issues/6906.
2017-06-28 12:06:00 -04:00
Scott Mitchell
d3c44ef985 Update Http2SecurityUtil cipher suites
Motivation:
Mozilla's Server Side cipher suite recommendations have been updated [1].

[1] https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility

Modifications:
- Update Http2SecurityUtil to exclude older ciphers.
- Remove support for DHE ciphersuites because they are now Intermediate and BoringSSL dropped support for these ciphers [2]

[2] https://boringssl.googlesource.com/boringssl/+/7e06de5d2d1b53c57c0c81e8d6ba4122b64cf626

Result:
Updated default ciphers for HTTP/2.
2017-06-28 11:40:28 -04:00
Nikolay Fedorovskikh
01eb428b39 Move methods for decode hex dump into StringUtil
Motivation:

PR #6811 introduced a public utility methods to decode hex dump and its parts, but they are not visible from netty-common.

Modifications:

1. Move the `decodeHexByte`, `decodeHexDump` and `decodeHexNibble` methods into `StringUtils`.
2. Apply these methods where applicable.
3. Remove similar methods from other locations (e.g. `HpackHex` test class).

Result:

Less code duplication.
2017-06-23 18:52:42 +02:00
Scott Mitchell
5934ae8fd2 Http2FrameLogger Updates
Motivation:
The Http2FrameLogger uses a custom format when logging events. We should use the more familiar format of 'channel event type: details' and single line logging for more consistent debugging.

Modifications:
- Http2FrameLogger should not use a StringBuilder and instead should directly use the Logger
- Http2FrameLogger should use the more consistent format defined above

Result:
Http2FrameLogger's logging formate is more consistent with other log events.
2017-06-21 17:12:38 -07:00
Nikolay Fedorovskikh
8f8be31226 Remove unnecessary conversions
Motivation:

In a `HttpConversionUtil#toHttp2Headers` a status code conversion can be replaced with using `HttpResponseStatus#codeAsText` method.

Modifications:

Apply `HttpResponseStatus#codeAsText` method.

Result:

Less allocations.
2017-06-21 21:53:48 +02:00
Norman Maurer
575baf5050 Use more aggressive expanding strategy in HpackHuffmanDecoder
Motivation:

Before we always expanded the buffer by the initialCapacity which by default is 32 bytes. This may lead to many expansions of the buffer until we finally reached the point that the buffer can fin everything.

Modifications:

Double the buffer size until the threshold of >= 1024 is hit. After this will grow it by the initialCapacity

Result:

Less expansion of the buffer (and so allocations / copies) when the intialCapacity is not big enough. Fixes [#6864].
2017-06-15 06:56:44 +02:00
Norman Maurer
3f0085c267 Do proper bounds-checking in HpackHuffmanDecoder to reduce overhead of IndexOutOfBoundsException creation
Motivation:

HpackHuffmanDecoder.Decoder did not do any bound-checking but just catched IndexOutOfBoundsException to detect if the array needs to grow. This can be very expensive because of fillInStackTrace()

Modifications:

Add proper bounds checking and grow the array if needed without catching IndexOutOfBoundsException.

Result:

Less overhead if the array needs to grow.
2017-06-13 07:43:29 +02:00
Scott Mitchell
f00638af52 AbstractHttp2ConnectionHandlerBuilder support for HPACK huffman decoder initial size
Motivation:
Depending on the use case it may make sense to increase or decrease the initial size of the buffer used during the HPACK huffman decode process. This is currently not exposed through the AbstractHttp2ConnectionHandlerBuilder.

Modifications:
- Add a method to AbstractHttp2ConnectionHandlerBuilder which allows the initial size of the buffer used during the HPACK huffman decode prcoess to be configured.

Result:
AbstractHttp2ConnectionHandlerBuilder provides more control of codec-http2 knobs.
2017-06-12 16:36:43 -07:00
Norman Maurer
fd67a2354d [maven-release-plugin] prepare for next development iteration 2017-06-08 21:06:24 +02:00
Norman Maurer
3acd5c68ea [maven-release-plugin] prepare release netty-4.1.12.Final 2017-06-08 21:06:01 +02:00
Norman Maurer
0db2901f4d [maven-release-plugin] prepare for next development iteration 2017-05-11 16:00:55 +02:00
Norman Maurer
f7a19d330c [maven-release-plugin] prepare release netty-4.1.11.Final 2017-05-11 16:00:16 +02:00
Nikolay Fedorovskikh
94e9448ae3 Simplify JUnit assertions
Motivation:

Some JUnit assert calls can be replaced by simpler.

Modifications:

Replacement with a more suitable methods.

Result:

More informative JUnit reports.
2017-05-09 20:19:10 +02:00
Moses Nakamura
cf26227c6c Supply a builder for Http2Codec
Motivation:

DefaultHttp2FrameWriter has constructors that it would be a hassle to
expose as configuration parameters on Http2Codec. We should instead
make a builder for Http2Codec.

Modifications:

Get rid of the public constructors on Http2Codec and instead make sure
you can always use the builder where you would have used the constructor
before.

Result:

Http2Codec can be configured more flexibly, and the SensitivityDetector
can be configured.
2017-05-05 09:32:46 -07:00
Norman Maurer
6915ec3bb9 [maven-release-plugin] prepare for next development iteration 2017-04-29 14:10:00 +02:00
Norman Maurer
f30f242fee [maven-release-plugin] prepare release netty-4.1.10.Final 2017-04-29 14:09:32 +02:00
Scott Mitchell
d21f2adb98 HTTP/2 StreamByteDistributor improve parameter validation
Motivation:
Each StreamByteDistributor may allow for priority in different ways, but there are certain characteristics which are invalid regardless of the distribution algorithm. We should validate these invalid characteristics at the flow controller level.

Modifications:
- Disallow negative stream IDs from being used. These streams may be accepted by the WeightedFairQueueByteDistributor and cause state for other valid streams to be evicted.
- Improve unit tests to verify limits are enforced.

Result:
Boundary conditions related to the priority parameters are validated more strictly.
2017-04-24 17:17:18 -07:00
Eric Anderson
799350c369 Fix HTTP/2 dependency tree corruption
Motivation:

Chrome was randomly getting stuck loading the tiles examples.
Investigation showed that the Netty flow controller thought it had
nothing to send for the connection even though some streams has queued
data and window available.

Modifications:

Fixed an accounting error where an implicitly created parent was not
being added to the dependency tree, thus it and all of its children were
orphaned from the connection's tree and would never have data written.

Result:

Fixes #6621
2017-04-22 08:26:47 -07:00
Eric Anderson
c6ad9338b3 Avoid infinite loop in HTTP/2 distributor toString()
Motivation:

Although effectively unused, the toString() of
WeightedFairQueueByteDistributor.State is useful for debugging. It
accidentally had an infinite loop, as it would recurse infinitely
between a parent and its child, which makes it less useful for
debugging.

Modifications:

Prune the infinite loop by using the parent's streamId instead of the
parent's toString().

Result:

Faster, less stack-overflowing toString()
2017-04-22 08:16:41 -07:00
Nikolay Fedorovskikh
0692bf1b6a fix the typos 2017-04-20 04:56:09 +02:00
Scott Mitchell
1bc5bc69e3 HTTP/2 Allow more time for EventLoopGroup to shutdown in test that use LocalChannel
Motivation:
The CI servers have reported leaks while building the HTTP/2 unit tests. The unit tests attempt to wait for the channels to be closed before exiting the test, but we should wait in case there are any tasks pending on the EventLoopGroup's task queues.

Modifications:
- Change the Future.sync() operations to Future.syncUninterruptibly()
- HTTP/2 unit tests which use local channel should wait for 5 seconds before shutting down the EventLoopGroups

Result:
More likely that any cleanup related tasks will execute before the unit tests are shutdown.
2017-04-06 17:56:21 -07:00
Scott Mitchell
225d10e1ad HTTP/2 Make DefaultHttp2HeadersDecoder's Http2Headers object creation extensible
Motivation:
It is generally useful to override DefaultHttp2HeadersDecoder's creation of a new Http2Headers object so more optimized versions can be substituted if the use case allows for it.

Modifications:
- DefaultHttp2HeadersDecoder should support an overridable method to generate the new Http2Headers object for each decode operation

Result:
DefaultHttp2HeadersDecoder is more extensible.
Fixes https://github.com/netty/netty/issues/6591.
2017-04-03 11:19:14 -07:00
Scott Mitchell
e8da5e5162 Revert "Expose HTTP/2 HpackDecoder (#6589)"
This reverts commit f4c635d30b.
2017-04-03 11:19:09 -07:00
Scott Mitchell
36c6a61d33 HTTP/2 remove unnecessary buffer operations
Motivation:
codec-http2 has some helper methods to write to ByteBuf in a big endian fashion. This is the default memory structure for ByteBuf so these helper methods are not necessary.

Modifications:
- remove writeUnsignedInt and writeUnsignedShort

Result:
codec-http2 has less ByteBuf helper methods which are not necessary.
2017-03-31 15:23:39 -07:00
chhsiao90
0ee36fef00 Accept two ways to start HTTP/2 over clear text
Motivation:

HTTP/2 support two ways to start on a no-tls tcp connection,
http/1.1 upgrade and prior knowlege methodology to start HTTP/2.
Currently, the http2-server from example only support
starting by upgrade. I think we can do a simple dispatch by peek first
bytes from inbound that match to prior knowledge preface or not and
determine which handlers to set into pipeline.

Modifications:

Add ClearTextHttp2ServerUpgradeHandler to support start HTTP/2 via clear
text with two approach. And update example/http2-server to support
this functionality.

Result:

netty HTTP/2 and the example http2-server accept for two ways to start
HTTP/2 over clear text.

Fixed memory leak problem

Update fields to final

Rename ClearText to cleartext

Addressed comments for code improvement

- Always prefer static, final, and private if possible
- Add UnstableApi annotation
- Used EmbeddedChannel.readInbound instead of unhandled inbound handler
- More assertion

Update javadoc for CleartextHttp2ServerUpgradeHandler

Rename ClearTextHttp2ServerUpgradeHandler to CleartextHttp2ServerUpgradeHandler

Removed redundant code about configure pipeline

nit: PriorKnowledgeHandler

Removed Mockito.spy, investigate conn state instead

Add Http2UpgradeEvent

Check null of the constructor arguments

Rename Http2UpgradeEvent to PriorKnowledgeUpgradeEvent

Update unit test
2017-03-31 15:21:48 -07:00
Nathan Mittler
f4c635d30b Expose HTTP/2 HpackDecoder (#6589)
Motivation:

gRPC (and potentially other libraries) has an optimized header processor that requires direct access to the HpackDecoder.

Modifications:

Make the HpackDecoder and its constructors public.

Result:

Fixes #6579
2017-03-31 10:51:01 -07:00
Norman Maurer
40bead56c4 Revert "http2: Http2StreamChannel now shares options of its parent channel"
This reverts commit 7467106630.
2017-03-31 07:45:56 +02:00
Vladimir Kostyukov
7467106630 http2: Http2StreamChannel now shares options of its parent channel
Motivation

Http2StreamChannel ignores options of its parent channel when being created. That leads to surprising results when, for example, unpooled allocator could be silently replaced with pooled allocator (default setting).

Modification

Copy parent channel's options over to the Http2StreamChannel.

Result

Channel options are now consistent between Http2StreamChannel and its parent channel. Newly added test passes on this branch and fails on master. Fixes #6551.
2017-03-23 21:25:54 -07:00
Norman Maurer
2b8c8e0805 [maven-release-plugin] prepare for next development iteration 2017-03-10 07:46:17 +01:00
Norman Maurer
1db58ea980 [maven-release-plugin] prepare release netty-4.1.9.Final 2017-03-10 07:45:28 +01:00
Norman Maurer
e12f504ac1 Remove deprecated usage of Mockito methods
Motivation:

We used some deprecated Mockito methods.

Modifications:

- Replace deprecated method usage
- Some cleanup

Result:

No more usage of deprecated Mockito methods. Fixes [#6482].
2017-03-09 20:59:54 +01:00
Scott Mitchell
01012fc5b7 HTTP/2 SETTINGS ACK sequencing issue
Motivation:
DefaultHttp2ConnectionDecoder#onSettingsRead processes the settings, and then sends a SETTINGS ACK to the remote peer. Processing the settings may result in frames which violate the previous settings being send to the remote peer. The remote peer will not apply the new settings until it has received the SETTINGS ACK, and therefore we may violate the settings from the remote peer's perspective and the connection will be shutdown.

Modifications:
- We should send the SETTINGS ACK before we process the settings to ensure the peer receives the SETTINGS ACK before other frames which assume the settings have already been applied

Result:
Fixes https://github.com/netty/netty/issues/6520.
2017-03-09 10:21:55 -08:00
Fabian Lange
a94b23df7d Support SSL_ prefixed cipher suites in addition to TLS_ prefixed ones.
Motivation:
Http2SecurityUtil currently lists HTTP/2 ciphers as documented by
JSSE docs [1] and the IANA [2] using the TLS_ prefix.
In some IBM J9 implementations the SSL_ prefix is used, which is also
covered by the JSSE.

[1] http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html
[2] http://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml

Modifications:
Add both variants of the cipher names (prefixed with SSL_ in additon to TLS_)

Result:
HTTP/2 connections can now be created using the SslProvider.JDK on IBM J9
and potentially other JVMs which use the SSL_ prefix.
2017-03-09 09:26:33 +01:00
Nikolay Fedorovskikh
2993760e92 Fix misordered 'assertEquals' arguments in tests
Motivation:

Wrong argument order in some 'assertEquals' applying.

Modifications:

Flip compared arguments.

Result:

Correct `assertEquals` usage.
2017-03-08 22:48:37 -08:00
Scott Mitchell
e18c85b768 Add missing methods to Http2ConnectionHandlerBuilder
Motivation:
A previous commit added methods to AbstractHttp2ConnectionHandlerBuilder but forgot to expose them in Http2ConnectionHandlerBuilder.

Modifications:
- expose the new methods in Http2ConnectionHandlerBuilder

Result:
Http2ConnectionHandlerBuilder supports the new configuration options.
2017-03-07 20:34:16 -08:00
Scott Mitchell
f9001b9fc0 HTTP/2 move internal HPACK classes to the http2 package
Motivation:
The internal.hpack classes are no longer exposed in our public APIs and can be made package private in the http2 package.

Modifications:
- Make the hpack classes package private in the http2 package

Result:
Less APIs exposed as public.
2017-03-02 07:42:41 -08:00
Christopher Exell
52aecabe43 Allow GOAWAY to be sent from handlers after the Http2MultiplexCodec so that app developers can shed load by issuing GOAWAY 2017-03-01 19:22:35 -08:00
Norman Maurer
7aff6b0330 Increase timeouts in Http2ConnectionRoundtripTest
Motivation:

The timeouts used in the Http2ConnectionRoundtripTest seems to be too low when leak-detection is enabled so we sometimes get failed tests due timeout.

Modifications:

Increase timeouts.

Result:

Fixes [#6442].
2017-02-24 07:58:59 +01:00
Nikolay Fedorovskikh
0623c6c533 Fix javadoc issues
Motivation:

Invalid javadoc in project

Modifications:

Fix it

Result:

More correct javadoc
2017-02-22 07:31:07 +01:00
chhsiao90
e148b53c67 Add unit test for DefaultHttp2FrameWriter
Motivation:

DefaultHttp2FrameWriter contains important logic for how http2
frame encode into binary, currently, netty had no unit test
just for DefaultHttp2FrameWriter.

Modifications:

Write DefaultHttp2FrameWriterTest to test DefaultHttp2FrameWriter

Result:

The coverage of DefaultHttp2FrameWriter is increased, and the
class was more reliable
2017-02-21 09:57:16 +01:00
chhsiao90
b1436e80ef Cleanup DefaultHttp2FrameReader about verifyUnknownFrame
Motivation:

In previous PR about handling unknwon frame in the middle of header
block, I didn't notice and re-use about checking is processing header
. And I added a redundant method for same functionality.
I think that the redundant method would lead to some misleading
situation.

Modifications:

Removed redundant code on DefaultHttp2FrameReader

Result:

The code is more readable
2017-02-20 22:12:20 -08:00
Scott Mitchell
08e0c612cf HTTP/2 Unit Test LocalChannel Leaks
Motivation:
Some unit HTTP/2 unit tests use LocalChannel. LocalChannel's doClose method will ensure any pending items in the queue will be released, but it may execute a Runnable on the peer's EventLoop to ensure the peer's queue is also cleaned up. The HTTP/2 unit tests close the event loop groups with no wait time so that unit tests will execute quickly, but if the doClose Runnable is in the EventLoop's queue it will not run and thus the items in the queue will not be released.

Modifications:
- Ensure all HTTP/2 unit tests which use LocalChannel wait for both client and server channels to be closed before closing the EventLoop.

Result:
Related to https://github.com/netty/netty/issues/5850.
2017-02-20 13:54:30 -08:00
fenik17
0cf3f54a8d Adding 'final' keyword for private fields where possible
Motivation

Missing 'final' keyword for fields

Modifications

Add 'final' for fields where possible

Result

More safe and consistent code
2017-02-14 08:29:15 +01:00
chhsiao90
72916b9960 Add unit test on DefaultHttp2FrameReader
Motivation:

DefaultHttp2FrameReader contains the logic for how it parsed the network
traffic from http2 client,
it also validate the content is legal or not.
So keep high coverage rate on it will increase the stability of api.

Modifications:

Add unit test on DefaultHttp2FrameReader

Result:

Coverage rate increased
2017-02-06 23:57:03 -08:00
Norman Maurer
a7c0ff665c Only use Mockito for mocking.
Motivation:

We used various mocking frameworks. We should only use one...

Modifications:

Make usage of mocking framework consistent by only using Mockito.

Result:

Less dependencies and more consistent mocking usage.
2017-02-07 08:47:22 +01:00
Dmitriy Dumanskiy
b9abd3c9fc Cleanup : for loops for arrays to make code easier to read and removed unnecessary toLowerCase() 2017-02-06 07:47:59 +01:00
Scott Mitchell
3482651e0c HTTP/2 Non Active Stream RFC Corrections
Motivation:
codec-http2 couples the dependency tree state with the remainder of the stream state (Http2Stream). This makes implementing constraints where stream state and dependency tree state diverge in the RFC challenging. For example the RFC recommends retaining dependency tree state after a stream transitions to closed [1]. Dependency tree state can be exchanged on streams in IDLE. In practice clients may use stream IDs for the purpose of establishing QoS classes and therefore retaining this dependency tree state can be important to client perceived performance. It is difficult to limit the total amount of state we retain when stream state and dependency tree state is combined.

Modifications:
- Remove dependency tree, priority, and weight related items from public facing Http2Connection and Http2Stream APIs. This information is optional to track and depends on the flow controller implementation.
- Move all dependency tree, priority, and weight related code from DefaultHttp2Connection to WeightedFairQueueByteDistributor. This is currently the only place which cares about priority. We can pull out the dependency tree related code in the future if it is generally useful to expose for other implementations.
- DefaultHttp2Connection should explicitly limit the number of reserved streams now that IDLE streams are no longer created.

Result:
More compliant with the HTTP/2 RFC.
Fixes https://github.com/netty/netty/issues/6206.

[1] https://tools.ietf.org/html/rfc7540#section-5.3.4
2017-02-01 10:34:27 -08:00
Scott Mitchell
6e5b25733f HTTP/2 Connection Preface User Event
Motivation:
If an HTTP/2 client writes data before the connection preface the peer will shutdown the socket. Depending on what is in the pipeline (SslHandler) may require different evaluation criteria to infer when the codec-http2 has written the connection preface on behalf of the client. This can lead to unnecessarily complexity and error prone/racy application code.

Modifications:
- Introduce a user event that is fired up the pipeline when codec-http2 writes the connection preface

Result:
Reliable mechanism for applications to use to know when connection preface has been written (related to https://github.com/netty/netty/issues/6272).
2017-02-01 10:10:42 -08:00
Carl Mastrangelo
ead9938980 Include Http 1 request in error message
Motivation:

When An HTTP server is listening in plaintext mode, it doesn't have
a chance to negotiate "h2" in the tls handshake.  HTTP 1 clients
that are not expecting an HTTP2 server will accidentally a request
that isn't an upgrade, which the HTTP/2 decoder will not
understand.  The decoder treats the bytes as hex and adds them to
the error message.

These error messages are hard to understand by humans, and result
in extra, manual work to decode.

Modification:

If the first bytes of the request are not the preface, the decoder
will now see if they are an HTTP/1 request first.  If so, the error
message will include the method and path of the original request in
the error message.

In case the path is long, the decoder will check up to the first
1024 bytes to see if it matches.  This could be a DoS vector if
tons of bad requests or other garbage come in.  A future optimization
would be to treat the first few bytes as an AsciiString and not do
any Charset decoding.  ByteBuf.toCharSequence alludes to such an
optimization.

The code has been left simple for the time being.

Result:

Faster identification of errant HTTP requests.
2017-01-30 09:46:38 -08:00
Norman Maurer
735d6dd636 [maven-release-plugin] prepare for next development iteration 2017-01-30 15:14:02 +01:00
Norman Maurer
76e22e63f3 [maven-release-plugin] prepare release netty-4.1.8.Final 2017-01-30 15:12:36 +01:00
Scott Mitchell
e13da218e9 HTTP/2 revert Http2FrameWriter throws API change
Motivation:
2fd42cfc6b fixed a bug related to encoding headers but it also introduced a throws statement onto the Http2FrameWriter methods which write headers. This throws statement makes the API more verbose and is not necessary because we can communicate the failure in the ChannelFuture that is returned by these methods.

Modifications:
- Remove throws from all Http2FrameWriter methods.

Result:
Http2FrameWriter APIs do not propagate checked exceptions.
2017-01-26 23:26:17 -08:00
chhsiao90
f4c2c1926f Fixed rfc violation about sending extension frame in the middle of headers
Motivation:

At rfc7540 5.5, it said that it's not permitted to send extension
frame in the middle of header block and need be treated as
protocol error

Modifications:

When received a extension frame, in netty it's called unknown frame,
will verify that is there an headersContinuation exists

Result:

When received a extension frame in the middle of header block,
will throw connection error and closed the connection
2017-01-26 11:25:15 -08:00
Christopher Exell
d509365974 Move location of where oversized headers that don't exceed the go away limit is done
so that the check occurs after all headers have been read
2017-01-24 15:19:46 -08:00
Christopher Exell
907726988d Update hpack Decoder CTOR to allow for overflow in maxHeaderList size, as we do when we apply our ack'ed settings
This prevents us from having the first request, that hasn't ack'ed the setting causing a GOAWAY when we'd would
be under the maxHeaderListSizeGoAway that would have been set after the settings ack.
2017-01-21 10:20:49 -08:00
Scott Mitchell
2fd42cfc6b HTTP/2 Max Header List Size Bug
Motivation:
If the HPACK Decoder detects that SETTINGS_MAX_HEADER_LIST_SIZE has been violated it aborts immediately and sends a RST_STREAM frame for what ever stream caused the issue. Because HPACK is stateful this means that the HPACK state may become out of sync between peers, and the issue won't be detected until the next headers frame. We should make a best effort to keep processing to keep the HPACK state in sync with our peer, or completely close the connection.
If the HPACK Encoder is configured to verify SETTINGS_MAX_HEADER_LIST_SIZE it checks the limit and encodes at the same time. This may result in modifying the HPACK local state but not sending the headers to the peer if SETTINGS_MAX_HEADER_LIST_SIZE is violated. This will also lead to an inconsistency in HPACK state that will be flagged at some later time.

Modifications:
- HPACK Decoder now has 2 levels of limits related to SETTINGS_MAX_HEADER_LIST_SIZE. The first will attempt to keep processing data and send a RST_STREAM after all data is processed. The second will send a GO_AWAY and close the entire connection.
- When the HPACK Encoder enforces SETTINGS_MAX_HEADER_LIST_SIZE it should not modify the HPACK state until the size has been checked.
- https://tools.ietf.org/html/rfc7540#section-6.5.2 states that the initial value of SETTINGS_MAX_HEADER_LIST_SIZE is "unlimited". We currently use 8k as a limit. We should honor the specifications default value so we don't unintentionally close a connection before the remote peer is aware of the local settings.
- Remove unnecessary object allocation in DefaultHttp2HeadersDecoder and DefaultHttp2HeadersEncoder.

Result:
Fixes https://github.com/netty/netty/issues/6209.
2017-01-19 10:42:43 -08:00
Scott Mitchell
b701da8d1c HTTP/2 HPACK Integer Encoding Bugs
Motivation:
- Decoder#decodeULE128 has a bounds bug and cannot decode Integer.MAX_VALUE
- Decoder#decodeULE128 doesn't support values greater than can be represented with Java's int data type. This is a problem because there are cases that require at least unsigned 32 bits (max header table size).
- Decoder#decodeULE128 treats overflowing the data type and invalid input the same. This can be misleading when inspecting the error that is thrown.
- Encoder#encodeInteger doesn't support values greater than can be represented with Java's int data type. This is a problem because there are cases that require at least unsigned 32 bits (max header table size).

Modifications:
- Correct the above issues and add unit tests.

Result:
Fixes https://github.com/netty/netty/issues/6210.
2017-01-18 18:36:47 -08:00
Scott Mitchell
c590e3bd63 HTTP/2 relax test timeouts
Motivation:
Build failures have been observed with 2 second timeouts on the CI servers. We should make the timeouts longer to reduce false positive test failures due to tests timing out prematurely.

Modifications:
- Increase timeouts from 2 and 3 seconds to 5 seconds.

Result:
Less false positive test failures.
2017-01-18 08:02:15 +01:00
Norman Maurer
7f01da8d0f [maven-release-plugin] prepare for next development iteration 2017-01-12 11:36:51 +01:00
Norman Maurer
7a21eb1178 [maven-release-plugin] prepare release netty-4.1.7.Final 2017-01-12 11:35:58 +01:00
Scott Mitchell
ec3d077e0d DefaultHttp2Connection modifying child map while iterating
Motivation:
When DefaultHttp2Connection removes a stream it iterates over all children and adds them as children to the parent of the stream being removed. This process may remove elements from the child map while iterating without using the iterator's remove() method. This is generally unsafe and may result in an undefined iteration.

Modifications:
- We should use the Iterator's remove() method while iterating over the child map

Result:
Fixes https://github.com/netty/netty/issues/6163
2017-01-11 11:07:18 -08:00