Commit Graph

469 Commits

Author SHA1 Message Date
Norman Maurer
52f384b37f [maven-release-plugin] prepare for next development iteration 2017-08-02 12:55:10 +00:00
Norman Maurer
8cc1071881 [maven-release-plugin] prepare release netty-4.1.14.Final 2017-08-02 12:54:51 +00:00
chhsiao90
8320a45c15 Configures HTTP2 pipeline with more proper way
Motivation:

When we use pipeline.replace and we still had ongoing inbound, then
there will be some problem that inbound message would go to wrong
handlers. So we add handler first, and remove self after add, so that
the next handler will be the correct one.

Modifications:

Uses remove after addAfter instead of replace.

Result:

Fixed #6881
2017-08-02 06:58:55 +02:00
Norman Maurer
2988fb8eeb Ensure Http2FrameCodec uses Http2Settings.defaultSettings()
Motivation:

Http2FrameCodec should use Http2Settings.defaultSettings() when no Http2Settings were specified by the user.

Modifications:

Replace new Http2Settings() with Http2Settings.defaultSettings()

Result:

Use correct Http2Settings by default when using Http2FrameCodec in all cases.
2017-08-01 07:07:05 +02:00
Vladimir Gordiychuk
fe8ecea366 Http2FrameLogger avoid hex dump of the ByteBufs when log disabled
Motivation:

Currentry logger create hex dump even if log write will not apply.
It's unecessary GC overhead.

Modifications:

Restore optimization from #3492

Result:

Fixes #7025
2017-07-26 21:21:04 +02:00
Spencer Fang
732b145842 Http2ConnectionHandler: allow graceful shutdown to wait forever
Motivation:

There should be a way to allow graceful shutdown to wait for all open streams to close without a timeout. Using gracefulShutdownTimeoutMillis with a large value is a bit of a hack, and has a gotcha that sufficiently large values will overflow the long, resulting in a ClosingChannelFutureListener that executes immediately.

Modification:

Allow to use gracefulShutdownTimeoutMillis(-1) to express waiting until all streams are closed.

Result:

We can now shutdown the connection without a forced timeout.
2017-07-26 20:40:24 +02:00
Scott Mitchell
a91df58ca1 HTTP/2 enforce HTTP message flow
Motivation:
codec-http2 currently does not strictly enforce the HTTP/1.x semantics with respect to the number of headers defined in RFC 7540 Section 8.1 [1]. We currently don't validate the number of headers nor do we validate that the trailing headers should indicate EOS.

[1] https://tools.ietf.org/html/rfc7540#section-8.1

Modifications:
- DefaultHttp2ConnectionDecoder should only allow decoding of a single headers and a single trailers
- DefaultHttp2ConnectionEncoder should only allow encoding of a single headers and optionally a single trailers

Result:
Constraints of RFC 7540 restricting the number of headers/trailers is enforced.
2017-07-19 13:37:23 -07:00
Norman Maurer
dbd82e07b1 Let Http2ServerUpgradeCodec support Http2FrameCodec
Motivation:

Http2ServerUpgradeCodec should support Http2FrameCodec.

Modifications:

- Add support for Http2FrameCodec
- Add example that uses Http2FrameCodec

Result:

More flexible use of Http2ServerUpgradeCodec
2017-07-19 11:12:10 +02:00
Norman Maurer
2a376eeb1b [maven-release-plugin] prepare for next development iteration 2017-07-06 13:24:06 +02:00
Norman Maurer
c7f8168324 [maven-release-plugin] prepare release netty-4.1.13.Final 2017-07-06 13:23:51 +02:00
Scott Mitchell
449befa003 Workaround IBM's J9 JVM getSupportedCipherSuites() returning SSL_ prefix cipher names
Motivation:
IBM's J9 JVM utilizes a custom cipher naming scheme with SSL_ prefix [1] instead of the TLS_ prefix defined by TLS RFCs and the JSSE cihper suite names [2]. IBM's documentation says that the SSL_ prefix are "interchangeable" with cipher names with the TLS_ prefix [1]. To work around this issue we parse the supported cipher list and see an SSL_ prefix we can also add the same cipher with the TLS_ prefix. For more details see a discussion on IBM's forums [3] and IBM's issue tracker [4].

[1] https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/ciphersuites.html
[2] http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#ciphersuites
[3] https://www.ibm.com/developerworks/community/forums/html/topic?id=9b5a56a9-fa46-4031-b33b-df91e28d77c2
[4] https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=71770

Modifications:
- When parsing the supported cipher list to get the supported ciphers and we encounter a SSL_ prefix we should also add a TLS_ prefix cipher.
- Remove SSL_ prefix ciphers from Http2SecurityUtil.

Result:
Work around for IBM JVM's custom naming scheme covers more cases for supported cipher suites.
2017-07-05 09:05:42 -04:00
Scott Mitchell
bc46a99eaa DefaultHttp2ConnectionEncoder#writeHeaders shouldn't send GO_AWAY if stream is closed
Motivation:
DefaultHttp2ConnectionEncoder#writeHeaders attempts to find a stream object, and if one doesn't exist it tries to create one. However in the event that the local endpoint has received a RST_STREAM frame before writing the response headers we attempt to create a stream. Since this stream ID is for the incorrect endpoint we then generate a GO_AWAY for what appears to be a protocol error, but can instead be failed locally.

Modifications:
- Just fail the local promise in the above situation instead of sending a GO_AWAY

Result:
Less severe consequences if the server asynchronously sends headers after a RST_STREAM has been received.
Fixes https://github.com/netty/netty/issues/6906.
2017-06-28 12:06:00 -04:00
Scott Mitchell
d3c44ef985 Update Http2SecurityUtil cipher suites
Motivation:
Mozilla's Server Side cipher suite recommendations have been updated [1].

[1] https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility

Modifications:
- Update Http2SecurityUtil to exclude older ciphers.
- Remove support for DHE ciphersuites because they are now Intermediate and BoringSSL dropped support for these ciphers [2]

[2] https://boringssl.googlesource.com/boringssl/+/7e06de5d2d1b53c57c0c81e8d6ba4122b64cf626

Result:
Updated default ciphers for HTTP/2.
2017-06-28 11:40:28 -04:00
Nikolay Fedorovskikh
01eb428b39 Move methods for decode hex dump into StringUtil
Motivation:

PR #6811 introduced a public utility methods to decode hex dump and its parts, but they are not visible from netty-common.

Modifications:

1. Move the `decodeHexByte`, `decodeHexDump` and `decodeHexNibble` methods into `StringUtils`.
2. Apply these methods where applicable.
3. Remove similar methods from other locations (e.g. `HpackHex` test class).

Result:

Less code duplication.
2017-06-23 18:52:42 +02:00
Scott Mitchell
5934ae8fd2 Http2FrameLogger Updates
Motivation:
The Http2FrameLogger uses a custom format when logging events. We should use the more familiar format of 'channel event type: details' and single line logging for more consistent debugging.

Modifications:
- Http2FrameLogger should not use a StringBuilder and instead should directly use the Logger
- Http2FrameLogger should use the more consistent format defined above

Result:
Http2FrameLogger's logging formate is more consistent with other log events.
2017-06-21 17:12:38 -07:00
Nikolay Fedorovskikh
8f8be31226 Remove unnecessary conversions
Motivation:

In a `HttpConversionUtil#toHttp2Headers` a status code conversion can be replaced with using `HttpResponseStatus#codeAsText` method.

Modifications:

Apply `HttpResponseStatus#codeAsText` method.

Result:

Less allocations.
2017-06-21 21:53:48 +02:00
Norman Maurer
575baf5050 Use more aggressive expanding strategy in HpackHuffmanDecoder
Motivation:

Before we always expanded the buffer by the initialCapacity which by default is 32 bytes. This may lead to many expansions of the buffer until we finally reached the point that the buffer can fin everything.

Modifications:

Double the buffer size until the threshold of >= 1024 is hit. After this will grow it by the initialCapacity

Result:

Less expansion of the buffer (and so allocations / copies) when the intialCapacity is not big enough. Fixes [#6864].
2017-06-15 06:56:44 +02:00
Norman Maurer
3f0085c267 Do proper bounds-checking in HpackHuffmanDecoder to reduce overhead of IndexOutOfBoundsException creation
Motivation:

HpackHuffmanDecoder.Decoder did not do any bound-checking but just catched IndexOutOfBoundsException to detect if the array needs to grow. This can be very expensive because of fillInStackTrace()

Modifications:

Add proper bounds checking and grow the array if needed without catching IndexOutOfBoundsException.

Result:

Less overhead if the array needs to grow.
2017-06-13 07:43:29 +02:00
Scott Mitchell
f00638af52 AbstractHttp2ConnectionHandlerBuilder support for HPACK huffman decoder initial size
Motivation:
Depending on the use case it may make sense to increase or decrease the initial size of the buffer used during the HPACK huffman decode process. This is currently not exposed through the AbstractHttp2ConnectionHandlerBuilder.

Modifications:
- Add a method to AbstractHttp2ConnectionHandlerBuilder which allows the initial size of the buffer used during the HPACK huffman decode prcoess to be configured.

Result:
AbstractHttp2ConnectionHandlerBuilder provides more control of codec-http2 knobs.
2017-06-12 16:36:43 -07:00
Norman Maurer
fd67a2354d [maven-release-plugin] prepare for next development iteration 2017-06-08 21:06:24 +02:00
Norman Maurer
3acd5c68ea [maven-release-plugin] prepare release netty-4.1.12.Final 2017-06-08 21:06:01 +02:00
Norman Maurer
0db2901f4d [maven-release-plugin] prepare for next development iteration 2017-05-11 16:00:55 +02:00
Norman Maurer
f7a19d330c [maven-release-plugin] prepare release netty-4.1.11.Final 2017-05-11 16:00:16 +02:00
Nikolay Fedorovskikh
94e9448ae3 Simplify JUnit assertions
Motivation:

Some JUnit assert calls can be replaced by simpler.

Modifications:

Replacement with a more suitable methods.

Result:

More informative JUnit reports.
2017-05-09 20:19:10 +02:00
Moses Nakamura
cf26227c6c Supply a builder for Http2Codec
Motivation:

DefaultHttp2FrameWriter has constructors that it would be a hassle to
expose as configuration parameters on Http2Codec. We should instead
make a builder for Http2Codec.

Modifications:

Get rid of the public constructors on Http2Codec and instead make sure
you can always use the builder where you would have used the constructor
before.

Result:

Http2Codec can be configured more flexibly, and the SensitivityDetector
can be configured.
2017-05-05 09:32:46 -07:00
Norman Maurer
6915ec3bb9 [maven-release-plugin] prepare for next development iteration 2017-04-29 14:10:00 +02:00
Norman Maurer
f30f242fee [maven-release-plugin] prepare release netty-4.1.10.Final 2017-04-29 14:09:32 +02:00
Scott Mitchell
d21f2adb98 HTTP/2 StreamByteDistributor improve parameter validation
Motivation:
Each StreamByteDistributor may allow for priority in different ways, but there are certain characteristics which are invalid regardless of the distribution algorithm. We should validate these invalid characteristics at the flow controller level.

Modifications:
- Disallow negative stream IDs from being used. These streams may be accepted by the WeightedFairQueueByteDistributor and cause state for other valid streams to be evicted.
- Improve unit tests to verify limits are enforced.

Result:
Boundary conditions related to the priority parameters are validated more strictly.
2017-04-24 17:17:18 -07:00
Eric Anderson
799350c369 Fix HTTP/2 dependency tree corruption
Motivation:

Chrome was randomly getting stuck loading the tiles examples.
Investigation showed that the Netty flow controller thought it had
nothing to send for the connection even though some streams has queued
data and window available.

Modifications:

Fixed an accounting error where an implicitly created parent was not
being added to the dependency tree, thus it and all of its children were
orphaned from the connection's tree and would never have data written.

Result:

Fixes #6621
2017-04-22 08:26:47 -07:00
Eric Anderson
c6ad9338b3 Avoid infinite loop in HTTP/2 distributor toString()
Motivation:

Although effectively unused, the toString() of
WeightedFairQueueByteDistributor.State is useful for debugging. It
accidentally had an infinite loop, as it would recurse infinitely
between a parent and its child, which makes it less useful for
debugging.

Modifications:

Prune the infinite loop by using the parent's streamId instead of the
parent's toString().

Result:

Faster, less stack-overflowing toString()
2017-04-22 08:16:41 -07:00
Nikolay Fedorovskikh
0692bf1b6a fix the typos 2017-04-20 04:56:09 +02:00
Scott Mitchell
1bc5bc69e3 HTTP/2 Allow more time for EventLoopGroup to shutdown in test that use LocalChannel
Motivation:
The CI servers have reported leaks while building the HTTP/2 unit tests. The unit tests attempt to wait for the channels to be closed before exiting the test, but we should wait in case there are any tasks pending on the EventLoopGroup's task queues.

Modifications:
- Change the Future.sync() operations to Future.syncUninterruptibly()
- HTTP/2 unit tests which use local channel should wait for 5 seconds before shutting down the EventLoopGroups

Result:
More likely that any cleanup related tasks will execute before the unit tests are shutdown.
2017-04-06 17:56:21 -07:00
Scott Mitchell
225d10e1ad HTTP/2 Make DefaultHttp2HeadersDecoder's Http2Headers object creation extensible
Motivation:
It is generally useful to override DefaultHttp2HeadersDecoder's creation of a new Http2Headers object so more optimized versions can be substituted if the use case allows for it.

Modifications:
- DefaultHttp2HeadersDecoder should support an overridable method to generate the new Http2Headers object for each decode operation

Result:
DefaultHttp2HeadersDecoder is more extensible.
Fixes https://github.com/netty/netty/issues/6591.
2017-04-03 11:19:14 -07:00
Scott Mitchell
e8da5e5162 Revert "Expose HTTP/2 HpackDecoder (#6589)"
This reverts commit f4c635d30b.
2017-04-03 11:19:09 -07:00
Scott Mitchell
36c6a61d33 HTTP/2 remove unnecessary buffer operations
Motivation:
codec-http2 has some helper methods to write to ByteBuf in a big endian fashion. This is the default memory structure for ByteBuf so these helper methods are not necessary.

Modifications:
- remove writeUnsignedInt and writeUnsignedShort

Result:
codec-http2 has less ByteBuf helper methods which are not necessary.
2017-03-31 15:23:39 -07:00
chhsiao90
0ee36fef00 Accept two ways to start HTTP/2 over clear text
Motivation:

HTTP/2 support two ways to start on a no-tls tcp connection,
http/1.1 upgrade and prior knowlege methodology to start HTTP/2.
Currently, the http2-server from example only support
starting by upgrade. I think we can do a simple dispatch by peek first
bytes from inbound that match to prior knowledge preface or not and
determine which handlers to set into pipeline.

Modifications:

Add ClearTextHttp2ServerUpgradeHandler to support start HTTP/2 via clear
text with two approach. And update example/http2-server to support
this functionality.

Result:

netty HTTP/2 and the example http2-server accept for two ways to start
HTTP/2 over clear text.

Fixed memory leak problem

Update fields to final

Rename ClearText to cleartext

Addressed comments for code improvement

- Always prefer static, final, and private if possible
- Add UnstableApi annotation
- Used EmbeddedChannel.readInbound instead of unhandled inbound handler
- More assertion

Update javadoc for CleartextHttp2ServerUpgradeHandler

Rename ClearTextHttp2ServerUpgradeHandler to CleartextHttp2ServerUpgradeHandler

Removed redundant code about configure pipeline

nit: PriorKnowledgeHandler

Removed Mockito.spy, investigate conn state instead

Add Http2UpgradeEvent

Check null of the constructor arguments

Rename Http2UpgradeEvent to PriorKnowledgeUpgradeEvent

Update unit test
2017-03-31 15:21:48 -07:00
Nathan Mittler
f4c635d30b Expose HTTP/2 HpackDecoder (#6589)
Motivation:

gRPC (and potentially other libraries) has an optimized header processor that requires direct access to the HpackDecoder.

Modifications:

Make the HpackDecoder and its constructors public.

Result:

Fixes #6579
2017-03-31 10:51:01 -07:00
Norman Maurer
40bead56c4 Revert "http2: Http2StreamChannel now shares options of its parent channel"
This reverts commit 7467106630.
2017-03-31 07:45:56 +02:00
Vladimir Kostyukov
7467106630 http2: Http2StreamChannel now shares options of its parent channel
Motivation

Http2StreamChannel ignores options of its parent channel when being created. That leads to surprising results when, for example, unpooled allocator could be silently replaced with pooled allocator (default setting).

Modification

Copy parent channel's options over to the Http2StreamChannel.

Result

Channel options are now consistent between Http2StreamChannel and its parent channel. Newly added test passes on this branch and fails on master. Fixes #6551.
2017-03-23 21:25:54 -07:00
Norman Maurer
2b8c8e0805 [maven-release-plugin] prepare for next development iteration 2017-03-10 07:46:17 +01:00
Norman Maurer
1db58ea980 [maven-release-plugin] prepare release netty-4.1.9.Final 2017-03-10 07:45:28 +01:00
Norman Maurer
e12f504ac1 Remove deprecated usage of Mockito methods
Motivation:

We used some deprecated Mockito methods.

Modifications:

- Replace deprecated method usage
- Some cleanup

Result:

No more usage of deprecated Mockito methods. Fixes [#6482].
2017-03-09 20:59:54 +01:00
Scott Mitchell
01012fc5b7 HTTP/2 SETTINGS ACK sequencing issue
Motivation:
DefaultHttp2ConnectionDecoder#onSettingsRead processes the settings, and then sends a SETTINGS ACK to the remote peer. Processing the settings may result in frames which violate the previous settings being send to the remote peer. The remote peer will not apply the new settings until it has received the SETTINGS ACK, and therefore we may violate the settings from the remote peer's perspective and the connection will be shutdown.

Modifications:
- We should send the SETTINGS ACK before we process the settings to ensure the peer receives the SETTINGS ACK before other frames which assume the settings have already been applied

Result:
Fixes https://github.com/netty/netty/issues/6520.
2017-03-09 10:21:55 -08:00
Fabian Lange
a94b23df7d Support SSL_ prefixed cipher suites in addition to TLS_ prefixed ones.
Motivation:
Http2SecurityUtil currently lists HTTP/2 ciphers as documented by
JSSE docs [1] and the IANA [2] using the TLS_ prefix.
In some IBM J9 implementations the SSL_ prefix is used, which is also
covered by the JSSE.

[1] http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html
[2] http://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml

Modifications:
Add both variants of the cipher names (prefixed with SSL_ in additon to TLS_)

Result:
HTTP/2 connections can now be created using the SslProvider.JDK on IBM J9
and potentially other JVMs which use the SSL_ prefix.
2017-03-09 09:26:33 +01:00
Nikolay Fedorovskikh
2993760e92 Fix misordered 'assertEquals' arguments in tests
Motivation:

Wrong argument order in some 'assertEquals' applying.

Modifications:

Flip compared arguments.

Result:

Correct `assertEquals` usage.
2017-03-08 22:48:37 -08:00
Scott Mitchell
e18c85b768 Add missing methods to Http2ConnectionHandlerBuilder
Motivation:
A previous commit added methods to AbstractHttp2ConnectionHandlerBuilder but forgot to expose them in Http2ConnectionHandlerBuilder.

Modifications:
- expose the new methods in Http2ConnectionHandlerBuilder

Result:
Http2ConnectionHandlerBuilder supports the new configuration options.
2017-03-07 20:34:16 -08:00
Scott Mitchell
f9001b9fc0 HTTP/2 move internal HPACK classes to the http2 package
Motivation:
The internal.hpack classes are no longer exposed in our public APIs and can be made package private in the http2 package.

Modifications:
- Make the hpack classes package private in the http2 package

Result:
Less APIs exposed as public.
2017-03-02 07:42:41 -08:00
Christopher Exell
52aecabe43 Allow GOAWAY to be sent from handlers after the Http2MultiplexCodec so that app developers can shed load by issuing GOAWAY 2017-03-01 19:22:35 -08:00
Norman Maurer
7aff6b0330 Increase timeouts in Http2ConnectionRoundtripTest
Motivation:

The timeouts used in the Http2ConnectionRoundtripTest seems to be too low when leak-detection is enabled so we sometimes get failed tests due timeout.

Modifications:

Increase timeouts.

Result:

Fixes [#6442].
2017-02-24 07:58:59 +01:00
Nikolay Fedorovskikh
0623c6c533 Fix javadoc issues
Motivation:

Invalid javadoc in project

Modifications:

Fix it

Result:

More correct javadoc
2017-02-22 07:31:07 +01:00
chhsiao90
e148b53c67 Add unit test for DefaultHttp2FrameWriter
Motivation:

DefaultHttp2FrameWriter contains important logic for how http2
frame encode into binary, currently, netty had no unit test
just for DefaultHttp2FrameWriter.

Modifications:

Write DefaultHttp2FrameWriterTest to test DefaultHttp2FrameWriter

Result:

The coverage of DefaultHttp2FrameWriter is increased, and the
class was more reliable
2017-02-21 09:57:16 +01:00
chhsiao90
b1436e80ef Cleanup DefaultHttp2FrameReader about verifyUnknownFrame
Motivation:

In previous PR about handling unknwon frame in the middle of header
block, I didn't notice and re-use about checking is processing header
. And I added a redundant method for same functionality.
I think that the redundant method would lead to some misleading
situation.

Modifications:

Removed redundant code on DefaultHttp2FrameReader

Result:

The code is more readable
2017-02-20 22:12:20 -08:00
Scott Mitchell
08e0c612cf HTTP/2 Unit Test LocalChannel Leaks
Motivation:
Some unit HTTP/2 unit tests use LocalChannel. LocalChannel's doClose method will ensure any pending items in the queue will be released, but it may execute a Runnable on the peer's EventLoop to ensure the peer's queue is also cleaned up. The HTTP/2 unit tests close the event loop groups with no wait time so that unit tests will execute quickly, but if the doClose Runnable is in the EventLoop's queue it will not run and thus the items in the queue will not be released.

Modifications:
- Ensure all HTTP/2 unit tests which use LocalChannel wait for both client and server channels to be closed before closing the EventLoop.

Result:
Related to https://github.com/netty/netty/issues/5850.
2017-02-20 13:54:30 -08:00
fenik17
0cf3f54a8d Adding 'final' keyword for private fields where possible
Motivation

Missing 'final' keyword for fields

Modifications

Add 'final' for fields where possible

Result

More safe and consistent code
2017-02-14 08:29:15 +01:00
chhsiao90
72916b9960 Add unit test on DefaultHttp2FrameReader
Motivation:

DefaultHttp2FrameReader contains the logic for how it parsed the network
traffic from http2 client,
it also validate the content is legal or not.
So keep high coverage rate on it will increase the stability of api.

Modifications:

Add unit test on DefaultHttp2FrameReader

Result:

Coverage rate increased
2017-02-06 23:57:03 -08:00
Norman Maurer
a7c0ff665c Only use Mockito for mocking.
Motivation:

We used various mocking frameworks. We should only use one...

Modifications:

Make usage of mocking framework consistent by only using Mockito.

Result:

Less dependencies and more consistent mocking usage.
2017-02-07 08:47:22 +01:00
Dmitriy Dumanskiy
b9abd3c9fc Cleanup : for loops for arrays to make code easier to read and removed unnecessary toLowerCase() 2017-02-06 07:47:59 +01:00
Scott Mitchell
3482651e0c HTTP/2 Non Active Stream RFC Corrections
Motivation:
codec-http2 couples the dependency tree state with the remainder of the stream state (Http2Stream). This makes implementing constraints where stream state and dependency tree state diverge in the RFC challenging. For example the RFC recommends retaining dependency tree state after a stream transitions to closed [1]. Dependency tree state can be exchanged on streams in IDLE. In practice clients may use stream IDs for the purpose of establishing QoS classes and therefore retaining this dependency tree state can be important to client perceived performance. It is difficult to limit the total amount of state we retain when stream state and dependency tree state is combined.

Modifications:
- Remove dependency tree, priority, and weight related items from public facing Http2Connection and Http2Stream APIs. This information is optional to track and depends on the flow controller implementation.
- Move all dependency tree, priority, and weight related code from DefaultHttp2Connection to WeightedFairQueueByteDistributor. This is currently the only place which cares about priority. We can pull out the dependency tree related code in the future if it is generally useful to expose for other implementations.
- DefaultHttp2Connection should explicitly limit the number of reserved streams now that IDLE streams are no longer created.

Result:
More compliant with the HTTP/2 RFC.
Fixes https://github.com/netty/netty/issues/6206.

[1] https://tools.ietf.org/html/rfc7540#section-5.3.4
2017-02-01 10:34:27 -08:00
Scott Mitchell
6e5b25733f HTTP/2 Connection Preface User Event
Motivation:
If an HTTP/2 client writes data before the connection preface the peer will shutdown the socket. Depending on what is in the pipeline (SslHandler) may require different evaluation criteria to infer when the codec-http2 has written the connection preface on behalf of the client. This can lead to unnecessarily complexity and error prone/racy application code.

Modifications:
- Introduce a user event that is fired up the pipeline when codec-http2 writes the connection preface

Result:
Reliable mechanism for applications to use to know when connection preface has been written (related to https://github.com/netty/netty/issues/6272).
2017-02-01 10:10:42 -08:00
Carl Mastrangelo
ead9938980 Include Http 1 request in error message
Motivation:

When An HTTP server is listening in plaintext mode, it doesn't have
a chance to negotiate "h2" in the tls handshake.  HTTP 1 clients
that are not expecting an HTTP2 server will accidentally a request
that isn't an upgrade, which the HTTP/2 decoder will not
understand.  The decoder treats the bytes as hex and adds them to
the error message.

These error messages are hard to understand by humans, and result
in extra, manual work to decode.

Modification:

If the first bytes of the request are not the preface, the decoder
will now see if they are an HTTP/1 request first.  If so, the error
message will include the method and path of the original request in
the error message.

In case the path is long, the decoder will check up to the first
1024 bytes to see if it matches.  This could be a DoS vector if
tons of bad requests or other garbage come in.  A future optimization
would be to treat the first few bytes as an AsciiString and not do
any Charset decoding.  ByteBuf.toCharSequence alludes to such an
optimization.

The code has been left simple for the time being.

Result:

Faster identification of errant HTTP requests.
2017-01-30 09:46:38 -08:00
Norman Maurer
735d6dd636 [maven-release-plugin] prepare for next development iteration 2017-01-30 15:14:02 +01:00
Norman Maurer
76e22e63f3 [maven-release-plugin] prepare release netty-4.1.8.Final 2017-01-30 15:12:36 +01:00
Scott Mitchell
e13da218e9 HTTP/2 revert Http2FrameWriter throws API change
Motivation:
2fd42cfc6b fixed a bug related to encoding headers but it also introduced a throws statement onto the Http2FrameWriter methods which write headers. This throws statement makes the API more verbose and is not necessary because we can communicate the failure in the ChannelFuture that is returned by these methods.

Modifications:
- Remove throws from all Http2FrameWriter methods.

Result:
Http2FrameWriter APIs do not propagate checked exceptions.
2017-01-26 23:26:17 -08:00
chhsiao90
f4c2c1926f Fixed rfc violation about sending extension frame in the middle of headers
Motivation:

At rfc7540 5.5, it said that it's not permitted to send extension
frame in the middle of header block and need be treated as
protocol error

Modifications:

When received a extension frame, in netty it's called unknown frame,
will verify that is there an headersContinuation exists

Result:

When received a extension frame in the middle of header block,
will throw connection error and closed the connection
2017-01-26 11:25:15 -08:00
Christopher Exell
d509365974 Move location of where oversized headers that don't exceed the go away limit is done
so that the check occurs after all headers have been read
2017-01-24 15:19:46 -08:00
Christopher Exell
907726988d Update hpack Decoder CTOR to allow for overflow in maxHeaderList size, as we do when we apply our ack'ed settings
This prevents us from having the first request, that hasn't ack'ed the setting causing a GOAWAY when we'd would
be under the maxHeaderListSizeGoAway that would have been set after the settings ack.
2017-01-21 10:20:49 -08:00
Scott Mitchell
2fd42cfc6b HTTP/2 Max Header List Size Bug
Motivation:
If the HPACK Decoder detects that SETTINGS_MAX_HEADER_LIST_SIZE has been violated it aborts immediately and sends a RST_STREAM frame for what ever stream caused the issue. Because HPACK is stateful this means that the HPACK state may become out of sync between peers, and the issue won't be detected until the next headers frame. We should make a best effort to keep processing to keep the HPACK state in sync with our peer, or completely close the connection.
If the HPACK Encoder is configured to verify SETTINGS_MAX_HEADER_LIST_SIZE it checks the limit and encodes at the same time. This may result in modifying the HPACK local state but not sending the headers to the peer if SETTINGS_MAX_HEADER_LIST_SIZE is violated. This will also lead to an inconsistency in HPACK state that will be flagged at some later time.

Modifications:
- HPACK Decoder now has 2 levels of limits related to SETTINGS_MAX_HEADER_LIST_SIZE. The first will attempt to keep processing data and send a RST_STREAM after all data is processed. The second will send a GO_AWAY and close the entire connection.
- When the HPACK Encoder enforces SETTINGS_MAX_HEADER_LIST_SIZE it should not modify the HPACK state until the size has been checked.
- https://tools.ietf.org/html/rfc7540#section-6.5.2 states that the initial value of SETTINGS_MAX_HEADER_LIST_SIZE is "unlimited". We currently use 8k as a limit. We should honor the specifications default value so we don't unintentionally close a connection before the remote peer is aware of the local settings.
- Remove unnecessary object allocation in DefaultHttp2HeadersDecoder and DefaultHttp2HeadersEncoder.

Result:
Fixes https://github.com/netty/netty/issues/6209.
2017-01-19 10:42:43 -08:00
Scott Mitchell
b701da8d1c HTTP/2 HPACK Integer Encoding Bugs
Motivation:
- Decoder#decodeULE128 has a bounds bug and cannot decode Integer.MAX_VALUE
- Decoder#decodeULE128 doesn't support values greater than can be represented with Java's int data type. This is a problem because there are cases that require at least unsigned 32 bits (max header table size).
- Decoder#decodeULE128 treats overflowing the data type and invalid input the same. This can be misleading when inspecting the error that is thrown.
- Encoder#encodeInteger doesn't support values greater than can be represented with Java's int data type. This is a problem because there are cases that require at least unsigned 32 bits (max header table size).

Modifications:
- Correct the above issues and add unit tests.

Result:
Fixes https://github.com/netty/netty/issues/6210.
2017-01-18 18:36:47 -08:00
Scott Mitchell
c590e3bd63 HTTP/2 relax test timeouts
Motivation:
Build failures have been observed with 2 second timeouts on the CI servers. We should make the timeouts longer to reduce false positive test failures due to tests timing out prematurely.

Modifications:
- Increase timeouts from 2 and 3 seconds to 5 seconds.

Result:
Less false positive test failures.
2017-01-18 08:02:15 +01:00
Norman Maurer
7f01da8d0f [maven-release-plugin] prepare for next development iteration 2017-01-12 11:36:51 +01:00
Norman Maurer
7a21eb1178 [maven-release-plugin] prepare release netty-4.1.7.Final 2017-01-12 11:35:58 +01:00
Scott Mitchell
ec3d077e0d DefaultHttp2Connection modifying child map while iterating
Motivation:
When DefaultHttp2Connection removes a stream it iterates over all children and adds them as children to the parent of the stream being removed. This process may remove elements from the child map while iterating without using the iterator's remove() method. This is generally unsafe and may result in an undefined iteration.

Modifications:
- We should use the Iterator's remove() method while iterating over the child map

Result:
Fixes https://github.com/netty/netty/issues/6163
2017-01-11 11:07:18 -08:00
Scott Mitchell
06e7627b5f Read Only Http2Headers
Motivation:
A read only implementation of Http2Headers can allow for a more efficient usage of memory and more performant combined construction and iteration during serialization.

Modifications:
- Add a new ReadOnlyHttp2Headers class

Result:
ReadOnlyHttp2Headers exists and can be used for performance reasons when appropriate.

```
Benchmark                                            (headerCount)  Mode  Cnt    Score   Error  Units
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders               1  avgt   20   96.156 ± 1.902  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders               5  avgt   20  157.925 ± 3.847  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders              10  avgt   20  236.257 ± 2.663  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders              20  avgt   20  392.861 ± 3.932  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders               1  avgt   20   48.759 ± 0.466  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders               5  avgt   20  113.122 ± 0.948  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders              10  avgt   20  192.698 ± 1.936  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders              20  avgt   20  348.974 ± 3.111  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                    1  avgt   20   35.694 ± 0.271  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                    5  avgt   20   98.993 ± 2.933  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                   10  avgt   20  171.035 ± 5.068  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                   20  avgt   20  330.621 ± 3.381  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders              1  avgt   20   40.573 ± 0.474  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders              5  avgt   20   56.516 ± 0.660  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders             10  avgt   20   76.890 ± 0.776  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders             20  avgt   20  117.531 ± 1.393  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders              1  avgt   20   29.206 ± 0.264  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders              5  avgt   20   44.587 ± 0.312  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders             10  avgt   20   64.458 ± 1.169  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders             20  avgt   20  107.179 ± 0.881  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                   1  avgt   20   21.563 ± 0.202  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                   5  avgt   20   41.019 ± 0.440  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                  10  avgt   20   64.053 ± 0.785  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                  20  avgt   20  113.737 ± 4.433  ns/op
```
2016-12-18 09:32:24 -08:00
Nikolaj Hald Nielsen
cd458f10bc Server returns status 431 on header size errors
Motivation:

Currently clients attempting to send headers that are too large recieve
a RST frame. This makes it harder than needed for implementations on top
of netty to handle this in a graceful way.

Modifications:

When the Decoder throws a StreamError of type FRAME_SIZE_ERROR, the
Http2ConnectionHandler will now attempt to send an Http2Header with
status 431 and endOfStream=true

Result:

Implementations now do not have to subclass parts of netty to handle
431s
2016-12-15 18:06:24 -08:00
Scott Mitchell
4639d56596 HttpToHttp2ConnectionHandlerTest increase setup timeout
Motivation:
The 2 second timeout to bootstrap the test can timeout on the build servers. We should increase the timeout so it is less likely under powered or over worked machines are less likely to generate false failures.

Modifications:
- HttpToHttp2ConnectionHandlerTest setup timeout changed from 2 to 5 seconds

Result:
Less false build failures.
2016-12-08 10:51:42 -08:00
Stephane Landelle
ba95c401a7 Misc clean up
Motivation:
IntelliJ issues several warnings.

Modifications:

* `ClientCookieDecoder` and `ServerCookieDecoder`:
  * `nameEnd`, `valueBegin` and `valueEnd` don't need to be initialized
  * `keyValLoop` loop doesn't been to be labelled, as it's the most inner one (same thing for labelled breaks)
  * Remove `if (i != headerLen)` as condition is always true
* `ClientCookieEncoder` javadoc still mention old logic
* `DefaultCookie`, `ServerCookieEncoder` and `DefaultHttpHeaders` use ternary ops that can be turned into simple boolean ones
* `DefaultHeaders` uses a for(int) loop over an array. It can be turned into a foreach one as javac doesn't allocate an iterator to iterate over arrays
* `DefaultHttp2Headers` and `AbstractByteBuf` `equal` can be turned into a single boolean statement
Result:

Cleaner code
2016-11-22 15:17:05 -08:00
Norman Maurer
e5070b7266 [#6023] Ensure LastInboundHandler correctly handle different events / data
Motivation:

LastInboundHandler maintains 2 queues which may contain the same data and tries to match these up when you read elements out of it. Because of this it can happen that you remove an element only out of one queue and so double free stuff later.

Modifications:

Just use one "queue" to store things.

Result:

Not possible to only remove things from one queue and so get into trouble later when release everything that sits in the handler.
2016-11-21 07:49:13 +01:00
Scott Mitchell
782b7bcf4a Fix HTTP/2 test cleanup with LocalChannel
Motivation:
When a LocalChannel is closed it is responsible to ensure all queued objects are released. When a LocalChannel is closed it will also close its peer channel. However in HTTP/2 unit tests we may not wait until all channels have completed the shutdown process before destroying the threads and exiting the test. This may mean buffers are GCed before they are released and be reported as a leak.

Modifications:
- In HTTP/2 tests when we use LocalChannel we should wait for all channels to close before exiting the test and cleaning up the associated EventLoopGroups.

Result:
More correct usage of LocalChannel in HTTP/2 tests.
2016-11-19 09:25:52 -08:00
Norman Maurer
0bc30a123e Eliminate usage of releaseLater(...) to reduce memory usage during tests
Motiviation:

We used ReferenceCountUtil.releaseLater(...) in our tests which simplifies a bit the releasing of ReferenceCounted objects. The problem with this is that while it simplifies stuff it increase memory usage a lot as memory may not be freed up in a timely manner.

Modifications:

- Deprecate releaseLater(...)
- Remove usage of releaseLater(...) in tests.

Result:

Less memory needed to build netty while running the tests.
2016-11-18 09:34:11 +01:00
Scott Mitchell
c4e96d010e HTTP/2 WeightedFairQueueByteDistributor Bug
Motivation:
If a stream is not able to send any data (flow control window for the stream is exhausted) but has descendants who can send data then WeightedFairQueueByteDistributor may incorrectly modify the pseudo time and also double add the associated state to the parent's priority queue. The pseudo time should only be modified if a node is moved in the priority tree, and not if there happens to be no active streams in its descendent tree and a descendent is moved (e.g. removed from the tree because it wrote all data and the last data frame was EOS). Also the state objects for WeightedFairQueueByteDistributor should only appear once in any queue. If this condition is violated the pseudo time accounting would be biased at and assumptions in WeightedFairQueueByteDistributor would be invalidated.

Modifications:
- WeightedFairQueueByteDistributor#isActiveCountChangeForTree should not allow re-adding to the priority queue if we are currently processing a node in the distribution algorithm. The distribution algorithm will re-evaluate if the node should be re-added on the tail end of the recursion.

Result:
Fixes https://github.com/netty/netty/issues/5980
2016-11-14 16:40:13 -08:00
Nikolaj Hald Nielsen
5b5f6a6031 Allow non-default Http/2 server initial settings
Motivation:

Currently it is not possible to have an Http/2 server send non default
initial settings to clients when doing the initial connection handshake

Modifications:

Add additional constructors to Http2Codec allowing users to specify the
initial settings to send to the client and apply locally

Result:

You can now specify non default initial settings
2016-11-09 08:31:26 +01:00
Scott Mitchell
efca9d180e HTTP/2 HPACK Encoder remove unsued import 2016-10-17 09:25:49 -07:00
buchgr
c9918de37b http2: Make MAX_HEADER_LIST_SIZE exceeded a stream error when encoding.
Motivation:

The SETTINGS_MAX_HEADER_LIST_SIZE limit, as enforced by the HPACK Encoder, should be a stream error and not apply to the whole connection.

Modifications:

Made the necessary changes for the exception to be of type StreamException.

Result:

A HEADERS frame exceeding the limit, only affects a specific stream.
2016-10-17 09:24:06 -07:00
Norman Maurer
5f533b7358 [maven-release-plugin] prepare for next development iteration 2016-10-14 13:20:41 +02:00
Norman Maurer
35fb0babe2 [maven-release-plugin] prepare release netty-4.1.6.Final 2016-10-14 12:47:19 +02:00
buchgr
af8ef3e40c Correctly release debugData and notify the promise when goAway was already sent
Motivation:

Commit 908464f161 also introduced a change to guard against re-entrance but failed to correctly handle the debugData and promise.

Modifications:

Release debugData and correctly notify the promise.

Result:

No more buffer leak and promise is always notified.
2016-10-14 10:57:18 +02:00
Norman Maurer
be8b6c6f24 Correctly add STREAM_WEIGHT header in InboundHttp2ToHttpAdapter.onPushPromiseRead(...)
Motivation:

The weight header with the default value is not set but it should be (rfc7540#5.3.5: …Pushed streams initially depend on their associated stream … are assigned a default weight of 16).

Modifications:

Add STREAM_WEIGHT header.

Result:

Correctly add headers.
2016-10-10 10:38:35 +02:00
Scott Mitchell
540c26bb56 HTTP/2 Ensure default settings are correctly enforced and interfaces clarified
Motivation:
The responsibility for retaining the settings values and enforcing the settings constraints is spread out in different areas of the code and may be initialized with different values than the default specified in the RFC. This should not be allowed by default and interfaces which are responsible for maintaining/enforcing settings state should clearly indicate the restrictions that they should only be set by the codec upon receipt of a SETTINGS ACK frame.

Modifications:
- Encoder, Decoder, and the Headers Encoder/Decoder no longer expose public constructors that allow the default settings to be changed.
- Http2HeadersDecoder#maxHeaderSize() exists to provide some bound when headers/continuation frames are being aggregated. However this is roughly the same as SETTINGS_MAX_HEADER_LIST_SIZE (besides the 32 byte octet for each header field) and can be used instead of attempting to keep the two independent values in sync.
- Encoding headers now enforces SETTINGS_MAX_HEADER_LIST_SIZE at the octect level. Previously the header encoder compared the number of header key/value pairs against SETTINGS_MAX_HEADER_LIST_SIZE instead of the number of octets (plus 32 bytes overhead).
- DefaultHttp2ConnectionDecoder#onData calls shouldIgnoreHeadersOrDataFrame but may swallow exceptions from this method. This means a STREAM_RST frame may not be sent when it should for an unknown stream and thus violate the RFC. The exception is no longer swallowed.

Result:
Default settings state is enforced and interfaces related to settings state are clarified.
2016-10-07 13:00:45 -07:00
buchgr
f010033590 h2childchan: Ability to open outbound/local streams. Fixes #4913
Motivation:

The HTTP/2 child channel API does not allow to create local/outbound HTTP/2 streams.

Modifications:

Add a Http2StreamChannelBootstrap that allows to create outbound streams.

Result:

The HTTP/2 child channel API now supports outbound streams.
2016-10-06 16:12:19 -07:00
Norman Maurer
d1d954da35 [#5866] HttpToHttp2ConnectionHandler / InboundHttp2ToHttpAdapter ignores STREAM_DEPENDENCY_ID and STREAM_WEIGHT.
Motivation:

HttpToHttp2ConnectionHandler.write(ctx, msg, promise) ignores HttpConversionUtil.ExtensionHeaderNames.STREAM_DEPENDENCY_ID header in outbound message. Beside this InboundHttp2ToHttpAdapter also not add the STREAM_DEPENDENCY_ID and STREAM_WEIGHT headers.

Modifications:

Respect STREAM_DEPENDENCY_ID and STREAM_WEIGHT

Result:

Correctly respect STREAM_DEPENDENCY_ID and STREAM_WEIGHT.
2016-10-03 18:05:50 +02:00
radai-rosenblatt
15ac6c4a1f Clean-up unused imports
Motivation:

the build doesnt seem to enforce this, so they piled up

Modifications:

removed unused import lines

Result:

less unused imports

Signed-off-by: radai-rosenblatt <radai.rosenblatt@gmail.com>
2016-09-30 09:08:50 +02:00
Norman Maurer
3240e97420 [#5862] Http2EventAdapter#onUnknownFrame(...) should throw Http2Exception
Motivation:

Http2EventAdapter implements the Http2FrameListener interface but implements the #onUnknownFrame(...) method without the interface's throws Http2Exception.

Modifications:

Add throws Http2Exception.

Result:

More correct method signature.
2016-09-29 14:22:13 +02:00
buchgr
6a6d422702 Complete documentation of StreamBufferingEncoder.
Motivation:

The StreamBufferingEncoder is missing documentation of what happens
to buffered frames when it's closed.

Modifications:

Added this missing piece of documentation.

Result:

Improved documentation.
2016-09-23 16:39:31 -07:00
Moses Nakamura
0d2baaf6ae codec-http2: Let users configure per-frame logs
Motivation:

codec-http2 is really loud!

Modification:

Allow users to select how to log in the Http2Codec.

Result:

We can run Http2Codec and log however we like.
2016-09-23 14:58:23 -07:00
Scott Mitchell
4145fae919 HTTP/2 HPACK Decoder VarInt Improvement
Motivation:
HTTP/2 Decoder#decodeULE128 current will tolerate more bytes than necessary when attempted to detect overflow. The usage of this method also currently requires an additional overflow conditional.

Modifications:
- Integrate the first byte into Decoder#decodeULE128 which allows us to detect overflow reliably and avoid overflow checks outside of this method.

Result:
Less conditionals and earlier overflow detection in Decoder#decodeULE128
2016-09-22 00:01:54 -07:00
Norman Maurer
3c62a20519 Revert "Ensure we only call debugData.release() if we called debugData.retain()"
This reverts commit 4beb075dd3.
2016-09-16 08:51:11 -07:00
Moses Nakamura
da8734a6f9 codec-http2: Mark requests as chunked in Http2ServerDowngrader
Motivation:

Http2ServerDowngrader doesn't mark chunked requests as chunked, even
though the natural conversion from http/2 requests to http/1.1 requests
is to chunked ones.

Modifications:

Mark requests that aren't already complete as chunked.

Result:

Requests will be chunked, and can later be aggregated if necessary.
2016-09-15 15:58:40 -07:00
buchgr
908464f161 make the http2 codec void promise ready.
Motivation:

Void promises need special treatment, as they don't behave like normal promises. One
cannot add a listener to a void promise for example.

Modifications:

Inspected the code for addListener() calls, and added extra logic for void promises
where necessary. Essentially, when writing a frame with a void promise, any errors
are reported via the channel pipeline's exceptionCaught event.

Result:

The HTTP/2 codec has better support for void promises.
2016-09-15 09:19:57 -07:00
Norman Maurer
4beb075dd3 Ensure we only call debugData.release() if we called debugData.retain()
Motivation:

We need to ensure we only call debugData.release() if we called debugData.retain(), otherwise we my see an IllegalReferenceCountException.

Modifications:

Only call release() if we called retain().

Result:

No more IllegalReferenceCountException possible.
2016-09-12 06:47:38 -07:00
Norman Maurer
51629245d0 Call debugData.retain() before we forward the frame to the pipeline
Motivation:

We need to call debugData.retain() before we forward the frame to the pipeline as ByteToMessageDecoder will call release() on the buffer.

Modifications:

Correctly retain debugData and fix the unit test to test for it.

Result:

No more IllegalReferenceCountException when using the Http2FrameCodec.
2016-09-12 06:45:13 -07:00
Norman Maurer
d2389a9339 [#5762] HTTP/2: SETTINGS_HEADER_TABLE_SIZE should be an unsigned int
Motivation:

he HTTP/2 spec demands that the max value for SETTINGS_HEADER_TABLE_SIZE should be an unsigned 32-bit integer.

Modifications:

Change the limit to unsigned 32-bit integer and add tests.

Result:

Complient to rfc.
2016-09-09 13:20:32 +02:00
Norman Maurer
54b1a100f4 [maven-release-plugin] prepare for next development iteration 2016-08-26 10:06:32 +02:00
buchgr
515f8964b4 HTTP/2: Fix some errors reported by h2spec.
Motivation:

h2spec [1] reported 32 issues [2] with Netty's HTTP/2 implementation.

Modifications:

Fixed 11 of those issues. Mostly wrong error codes or added missing validation
code in the frame reader.

Result:

11 fewer errors. h2spec now reports: 70 tests, 48 passed, 1 skipped, 21 failed

[1] https://github.com/summerwind/h2spec
[2] https://github.com/netty/netty/issues/5761
2016-09-01 08:28:16 +02:00
Norman Maurer
1208b90f57 [maven-release-plugin] prepare release netty-4.1.5.Final 2016-08-26 04:59:35 +02:00
Trustin Lee
5b46cf25c1 Fulfill the write promise of empty HTTP/2 DATA frames sooner
Motivation:

DefaultHttp2ConnectionEncoder.FlowControlledData.write() does not
complete the write promise of empty HTTP/2 DATA frames until either a
non-DATA frame, a non-empty DATA frame or a DATA frame with endOfStream
set. This makes the write promise of the empty DATA frame is notified
much later than it could be.

Modifications:

- Notify the write promise of the empty DATA frames immediately is the
  queue contains empty DATA frames only

Result:

The write promise of an empty DATA frame is notified sooner.
2016-08-26 08:45:09 +09:00
Scott Mitchell
208893aac9 HTTP/2 Hpack Encoder Cleanup
Motivation:
The HTTP/2 HPACK Encoder class has some code which is only used for test purposes. This code can be removed to reduce complexity and member variable count.

Modifications:
- Remove test code and update unit tests
- Other minor cleanup

Result:
Test code is removed from operational code.
2016-08-25 09:08:46 -07:00
Norman Maurer
5fd239c29c Ensure we not log missleading errors if the promise was already failed due errors
Motivation:

In DefaultHttp2ConnectionEncoder we fail the promise in in the FlowControlledData.error(...) method but also add it the CoalescingBufferQueue. Which can lead to have the promise failed by error(...) before it can be failed in CoalescingBufferQueue.

This can lead to confusing and missleading errors in the log like:
    2016-08-12 09:47:43,716       nettyIoExecutorGroup-1-9 [WARN ] PromiseNotifier                - Failed to mark a promise as failure because it's done already: DefaultChannelPromise@374225e0(failure: javax.net.ssl.SSLException: SSLEngine closed already)
    javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-all-4.1.5.Final-SNAPSHOT.jar:?]

Modifications:

Ensure we only fail the queue (which will also fail the promise).

Result:

No more missleading logs.
2016-08-18 07:17:45 +02:00
buchgr
d568cfc14a HTTP/2: Treat MAX_CONCURRENT_STREAMS exceeded as a stream error.
Motivation:

As per the HTTP/2 spec, exceeding the MAX_CONCURRENT_STREAMS should be treated as a stream error as opposed to a connection error.

"An endpoint that receives a HEADERS frame that causes its advertised concurrent stream limit to be exceeded MUST treat this as a stream error (Section 5.4.2) of type PROTOCOL_ERROR or REFUSED_STREAM." http://httpwg.org/specs/rfc7540.html#rfc.section.5.1.2

Modifications:

Make the error a stream error.

Result:

It's a stream error.
2016-08-17 14:47:15 +02:00
Scott Mitchell
f73c4f24ee HTTP/2 HPACK Bounds Check Fix
Motivation:
21e8d84b79 changed the way bounds checking was done, but however a bounds check in the case of READ_LITERAL_HEADER_NAME_LENGTH_PREFIX was using an old value. This would delay when the bounds check would actually be done and potentially allow more allocation than necessary.

Modifications:
- Use the new length (index) in the bounds check instead of an old length (nameLength) which had not yet been assigned to the new value.

Result:
More correct bounds checking.
2016-08-13 11:04:26 -07:00
Scott Mitchell
21e8d84b79 HTTP/2 Simplify Headers Decode Bounds Checking
Motivation:
The HPACK decoder keeps state so that the decode method can be called multiple times with successive header fragments. This decoder also requires that a method is called to signify the decoding is complete. At this point status is returned to indicate if the max header size has been violated. Netty always accumulates the header fragments into a single buffer before attempting to HPACK decode process and so keeping state and delaying notification that bounds have been exceeded is not necessary.

Modifications:
- HPACK Decoder#decode(..) now must be called with a complete header block
- HPACK will terminate immediately if the maximum header length, or maximum number of headers is exceeded
- Reduce member variables in the HPACK Decoder class because they can now live in the decode(..) method

Result:
HPACK bounds checking is done earlier and less class state is needed.
2016-08-12 17:12:53 -07:00
Scott Mitchell
4d74bf3984 HTTP/2 MaxStreams cleanup
Motivation:
765e944d4d imposed a limit on the maximum number of stream in all states. However the default limit did not allow room for streams in addition to SETTINGS_MAX_CONCURRENT_STREAMS. This can mean streams in states outside the states which SETTINGS_MAX_CONCURRENT_STREAMS applies to may not be reliably created.

Modifications:
- The default limit should be larger than SETTINGS_MAX_CONCURRENT_STREAMS

Result:
More lenient limit is applied to maxStreams by default.
2016-08-11 09:38:58 -07:00
Scott Mitchell
765e944d4d HTTP/2 limit streams in all states
Motivation:
SETTINGS_MAX_CONCURRENT_STREAMS does not apply to idle streams and thus we do not apply any explicit limitations on how many idle streams can be created. This may allow a peer to consume an undesirable amount of resources.

Modifications:
- Each Endpoint should enforce a limit for streams in a any state. By default this limit will be the same as SETTINGS_MAX_CONCURRENT_STREAMS but can be overridden if necessary.

Result:
There is now a limit to how many IDLE streams can be created.
2016-08-11 09:01:37 -07:00
Scott Mitchell
a4ad68239e Http2ConnectionDecoder remove localSettings setter method
Motivation:
Http2ConnectionDecoder#localSettings(Http2Settings) is not used in codec-http2 and currently results in duplicated code.

Modifications:
- Remove Http2ConnectionDecoder#localSettings(Http2Settings)

Result:
Smaller interface and less duplicated code.
2016-08-10 12:23:03 -07:00
Norman Maurer
cb7cf4491c [maven-release-plugin] prepare for next development iteration 2016-07-27 13:29:56 +02:00
Norman Maurer
9466b32d05 [maven-release-plugin] prepare release netty-4.1.4.Final 2016-07-27 13:16:59 +02:00
buchgr
328510468c Complete ChannelPromise for Http2WindowUpdateFrames in Http2FrameCodec. Fixes #5530
Motivation:

The channel promise of a window update frame is not completed correctly,
depending on the failure or success of the operation.

Modification:

Succeed / Fail the promise if the window update succeeds / fails.

Result:

Correctly succeed / fail the promise.
2016-07-15 17:29:41 +02:00
Norman Maurer
047f6aed28 [maven-release-plugin] prepare for next development iteration 2016-07-15 09:09:13 +02:00
Norman Maurer
b2adea87a0 [maven-release-plugin] prepare release netty-4.1.3.Final 2016-07-15 09:08:53 +02:00
Oliver Gould
f2ce28bf18 Satisfy write promise when writing an Http2WindowUpdateFrame to Http2FrameCodec.
Motivation:

When writing an Http2WindowUpdateFrame to an Http2FrameCodec, the
ChannelPromise is never satisfied, so callers cannot generically rely on the
write future being satisfied on success.

Modifications:

When writing Http2WindowUpdateFrame, Http2FrameCodec now satisfies the
ChannelPromise immediately.

Result:

The write future is satisfied on successful writes.

Fixes netty/netty#5530.
2016-07-13 21:31:34 +02:00
Scott Mitchell
9de90d07a9 DefaultHttp2RemoteFlowController reentry infinite loop
Motivation:
DefaultHttp2RemoteFlowController.writePendingBytes does not support reentry but does not enforce this constraint. Reentry is possible if the channel transitions from Writable -> Not Writable -> Writable during the distribution phase. This can happen if the user flushes when notified of the channel transitioning to the not writable state, and may be done if the user wants to fill the channel outbound buffer before flushing.

Modifications:
- DefaultHttp2RemoteFlowController.writePendingBytes should protect against reentry

Result:
DefaultHttp2RemoteFlowController will not allocate unexpected amounts or enter an infinite loop.
2016-07-13 09:12:33 -07:00
Scott Mitchell
4baff691b4 DefaultPromise make listeners not volatile
Motivation:
DefaultPromise has a listeners member variable which is volatile to allow for an optimization which makes notification of listeners less expensive when there are no listeners to notify. However this change makes all other operations involving the listeners member variable more costly. This optimization which requires listeners to be volatile can be removed to avoid volatile writes/reads for every access on the listeners member variable.

Modifications:
- DefaultPromise listeners is made non-volatile and the null check optimization is removed

Result:
DefaultPromise.listeners is no longer volatile.
2016-07-07 12:53:03 -07:00
Norman Maurer
6492cb98b2 Revert "DefaultPromise make listeners not volatile"
This reverts commit 4d8132ff24 as I missed something I want to discuss first.
2016-07-07 08:37:41 +02:00
Scott Mitchell
4d8132ff24 DefaultPromise make listeners not volatile
Motivation:
DefaultPromise has a listeners member variable which is volatile to allow for an optimization which makes notification of listeners less expensive when there are no listeners to notify. However this change makes all other operations involving the listeners member variable more costly. This optimization which requires listeners to be volatile can be removed to avoid volatile writes/reads for every access on the listeners member variable.

Modifications:
- DefaultPromise listeners is made non-volatile and the null check optimization is removed

Result:
DefaultPromise.listeners is no longer volatile.
2016-07-07 08:01:25 +02:00
Norman Maurer
4676a2271c [maven-release-plugin] prepare for next development iteration 2016-07-01 10:33:32 +02:00
Norman Maurer
ad270c02b9 [maven-release-plugin] prepare release netty-4.1.2.Final 2016-07-01 09:07:40 +02:00
Scott Mitchell
6af56ffe76 HPACK Encoder headerFields improvements
Motivation:
HPACK Encoder has a data structure which is similar to a previous version of DefaultHeaders. Some of the same improvements can be made.

Motivation:
- Enforce the restriction that the Encoder's headerFields length must be a power of two so we can use masking instead of modulo
- Use AsciiString.hashCode which already has optimizations instead of having yet another hash code algorithm in Encoder

Result:
Fixes https://github.com/netty/netty/issues/5357
2016-06-30 09:00:12 -07:00
Scott Mitchell
a7f7d9c8e0 Remove unsafe char[] access in PlatformDependent
Motivation:
PlatformDependent attempts to use reflection to get the underlying char[] (or byte[]) from String objects. This is fragile as if the String implementation does not utilize the full array, and instead uses a subset of the array, this optimization is invalid. OpenJDK6 and some earlier versions of OpenJDK7 String have the capability to use a subsection of the underlying char[].

Modifications:
- PlatformDependent should not attempt to use the underlying array from String (or other data types) via reflection

Result:
PlatformDependent hash code generation for CharSequence does not depend upon specific JDK implementation details.
2016-06-30 08:58:28 -07:00
Scott Mitchell
df41be6fc8 HTTP/2 Decoder validate that GOAWAY lastStreamId doesn't increase
Motivation:
The HTTP/2 RFC states in https://tools.ietf.org/html/rfc7540#section-6.8 that Endpoints MUST NOT increase the value they send in the last stream identifier however we don't enforce this when decoding GOAWAY frames.

Modifications:
- Throw a connection error if the peer attempts to increase the lastStreamId in a GOAWAY frame

Result:
RFC is more strictly enforced.
2016-06-29 07:53:13 -07:00
buchgr
3613d15bca Split Http2MultiplexCodec into Frame- and MultiplexCodec + Tests. Fixes #4914
Motivation:

Quote from issue 4914:
"Http2MultiplexCodec currently does two things: mapping the existing h2 API to frames and managing the child channels.

It would be better if the two parts were separated. This would allow less-coupled development of the HTTP/2 handlers (flow control could be its own handler, for instance) and allow applications to insert themselves between all streams and the codec, which permits custom logic and could be used, in part, to implement custom frame types.

It would also greatly ease testing, as the child channel could be tested by itself without dealing with how frames are encoded on the wire."

Modifications:

- Split the Http2MultiplexCodec into Http2FrameCodec and Http2MultiplexCodec. The Http2FrameCodec interacts with the existing HTTP/2 callback-based API, while the Http2MulitplexCodec is completely independent of it and simply multiplexes Http2StreamFrames to the child channels. Additionally, the Http2Codec handler is introduced, which is a convenience class that simply sets up the Http2FrameCodec and Http2MultiplexCodec in the channel pipeline and removes itself.

- Improved test coverage quite a bit.

Result:

- The original Http2MultiplexCodec is split into Http2FrameCodec and Http2MultiplexCodec.
- More tests for higher confidence in the code.
2016-06-29 07:22:09 +02:00
Scott Mitchell
70651cc58d HpackUtil.equals performance improvement
Motivation:
PR #5355 modified interfaces to reduce GC related to the HPACK code. However this came with an anticipated performance regression related to HpackUtil.equals due to AsciiString's increase cost of charAt(..). We should mitigate this performance regression.

Modifications:
- Introduce an equals method in PlatformDependent which doesn't leak timing information and use this in HpcakUtil.equals

Result:
Fixes https://github.com/netty/netty/issues/5436
2016-06-27 14:37:39 -07:00
buchgr
73c4ad5f0a http2: count pad length field toward flow control. Fixes #5434
Motivation:
The HTTP/2 specification requires the pad length field of DATA, HEADERS and PUSH_PROMISE frames to be counted towards the flow control window. The current implementation doesn't do so (See #5434).

Furthermore, it's currently not possible to add one byte padding, as this would add the one byte pad length field as well as append one padding byte to the end of the frame.

Modifications:
Include the one byte pad length field in the padding parameter of the API. Thereby extending the allowed value range by one byte to 256 (inclusive). On the wire, a one byte padding is encoded with a pad length field with value zero and a 256 byte padding is encoded with a pad length field with value 255 and 255 bytes append to the end of the frame.

Result:
More correct padding.
2016-06-25 09:51:36 -07:00
Norman Maurer
b4d4c0034d Optimize HPACK usage to align more with Netty types and remove heavy object creations. Related to [#3597]
Motivations:

The HPACK code was not really optimized and written with Netty types in mind. Because of this a lot of garbage was created due heavy object creation.

This was first reported in [#3597] and https://github.com/grpc/grpc-java/issues/1872 .

Modifications:

- Directly use ByteBuf as input and output
- Make use of ByteProcessor where possible
- Use AsciiString as this is the only thing we need for our http2 usage

Result:

Less garbage and better usage of Netty apis.
2016-06-22 14:26:05 +02:00
Tim Brooks
d964bf6f18 Remove usages of deprecated methods group() and childGroup().
Motivation:

These methods were recently deprecated. However, they remained in use in several locations in Netty's codebase.

Modifications:

Netty's code will now access the bootstrap config to get the group or child group.

Result:

No impact on functionality.
2016-06-21 14:06:57 +02:00
Scott Mitchell
6aa5f76d42 HTTP/2 DelegatingDecompressorFrameListener return bytes to flow control
Motivation:
If a single DATA frame ends up being decompressed into multiple frames by DelegatingDecompressorFrameListener the flow control accounting is delayed until all frames have been decompressed. However it is possible the user may want to return bytes to the flow controller which were not included in the onDataRead return value. In this case the amount of processed bytes has not been incremented and will lead to negative value for processed bytes.

Modifications:
- Http2Decompressor.incrementProcessedBytes should be called each time onDataRead is called to ensure all bytes are accounted for at the correct time

Result:
Fixes https://github.com/netty/netty/issues/5375
2016-06-20 14:24:09 -07:00
Norman Maurer
e845670043 Set some StackTraceElement on pre-instantiated static exceptions
Motivation:

We use pre-instantiated exceptions in various places for performance reasons. These exceptions don't include a stacktrace which makes it hard to know where the exception was thrown. This is especially true as we use the same exception type (for example ChannelClosedException) in different places. Setting some StackTraceElements will provide more context as to where these exceptions original and make debugging easier.

Modifications:

Set a generated StackTraceElement on these pre-instantiated exceptions which at least contains the origin class and method name. The filename and linenumber are specified as unkown (as stated in the javadocs of StackTraceElement).

Result:

Easier to find the origin of a pre-instantiated exception.
2016-06-20 11:33:05 +02:00
Guido Medina
c3abb9146e Use shaded dependency on JCTools instead of copy and paste
Motivation:
JCTools supports both non-unsafe, unsafe versions of queues and JDK6 which allows us to shade the library in netty-common allowing it to stay "zero dependency".

Modifications:
- Remove copy paste JCTools code and shade the library (dependencies that are shaded should be removed from the <dependencies> section of the generated POM).
- Remove usage of OneTimeTask and remove it all together.

Result:
Less code to maintain and easier to update JCTools and less GC pressure as the queue implementation nt creates so much garbage
2016-06-10 13:19:45 +02:00
Norman Maurer
4dec7f11b7 [maven-release-plugin] prepare for next development iteration 2016-06-07 18:52:34 +02:00
Norman Maurer
cf670fab75 [maven-release-plugin] prepare release netty-4.1.1.Final 2016-06-07 18:52:22 +02:00
Scott Mitchell
79f2e3604e HTTP/2 close only send GO_AWAY if one has not already been sent
Motivation:
Http2ConnectionHandler will always send a GO_AWAY when the channel is closed. This may cause problems if the user is attempting to control when GO_AWAY is sent and the content of the GO_AWAY.

Modifications:
- When the channel is closed Http2ConnectionHandler should only send a GO_AWAY if one has not already been sent

Result:
The user has more control over when GO_AWAY is sent
Fixes https://github.com/netty/netty/issues/5307
2016-06-06 11:18:30 -07:00
Norman Maurer
844976a0a2 Ensure the same ByteBufAllocator is used in the EmbeddedChannel when compress / decompress. Related to [#5294]
Motivation:

The user may specify to use a different allocator then the default. In this case we need to ensure it is shared when creating the EmbeddedChannel inside of a ChannelHandler

Modifications:

Use the config of the "original" Channel in the EmbeddedChannel and so share the same allocator etc.

Result:

Same type of buffers are used.
2016-05-31 09:08:33 +02:00
Norman Maurer
6ca49d1336 [maven-release-plugin] prepare for next development iteration 2016-05-25 19:16:44 +02:00
Norman Maurer
446b38db52 [maven-release-plugin] prepare release netty-4.1.0.Final 2016-05-25 19:14:15 +02:00
nmittler
79a06eaede Fix NPE in Http2ConnectionHandler.onHttpServerUpgrade
Motivation:

Performing a server upgrade with a new initial flow control window will cause an NPE in the DefaultHttp2RemoteFlowController. This is due to the fact that the monitor does not check whether or not the channel is writable.

Modifications:

Added a check for channel writability before calling `writePendingBytes`. Also fixed a unit test that was supposed to be testing this :).

Result:

Fixes #5301
2016-05-25 09:13:06 -07:00
Norman Maurer
7b25402e80 Add CompositeByteBuf.addComponent(boolean ...) method to simplify usage
Motivation:

At the moment the user is responsible to increase the writer index of the composite buffer when a new component is added. We should add some methods that handle this for the user as this is the most popular usage of the composite buffer.

Modifications:

Add new methods that autoamtically increase the writerIndex when buffers are added.

Result:

Easier usage of CompositeByteBuf.
2016-05-21 19:52:16 +02:00
Scott Mitchell
1cb706ac93 HTTP/2 HPACK Header Name Validation and Trailing Padding
Motivation:
The HPACK code currently disallows empty header names. This is not explicitly forbidden by the HPACK RFC https://tools.ietf.org/html/rfc7541. However the HTTP/1.x RFC https://tools.ietf.org/html/rfc7230#section-3.2 and thus HTTP/2 both disallow empty header names, and so this precondition check should be moved from the HPACK code to the protocol level.
HPACK also requires that string literals which are huffman encoded must be treated as an encoding error if the string has more than 7 trailing padding bits https://tools.ietf.org/html/rfc7541#section-5.2, but this is currently not enforced.

Result:
- HPACK to allow empty header names
- HTTP/1.x and HTTP/2 header validation should not allow empty header names
- Enforce max of 7 trailing padding bits

Result:
Code is more compliant with the above mentioned RFCs
Fixes https://github.com/netty/netty/issues/5228
2016-05-17 13:42:16 -07:00
Trustin Lee
3a9f472161 Make retained derived buffers recyclable
Related: #4333 #4421 #5128

Motivation:

slice(), duplicate() and readSlice() currently create a non-recyclable
derived buffer instance. Under heavy load, an application that creates a
lot of derived buffers can put the garbage collector under pressure.

Modifications:

- Add the following methods which creates a non-recyclable derived buffer
  - retainedSlice()
  - retainedDuplicate()
  - readRetainedSlice()
- Add the new recyclable derived buffer implementations, which has its
  own reference count value
- Add ByteBufHolder.retainedDuplicate()
- Add ByteBufHolder.replace(ByteBuf) so that..
  - a user can replace the content of the holder in a consistent way
  - copy/duplicate/retainedDuplicate() can delegate the holder
    construction to replace(ByteBuf)
- Use retainedDuplicate() and retainedSlice() wherever possible
- Miscellaneous:
  - Rename DuplicateByteBufTest to DuplicatedByteBufTest (missing 'D')
  - Make ReplayingDecoderByteBuf.reject() return an exception instead of
    throwing it so that its callers don't need to add dummy return
    statement

Result:

Derived buffers are now recycled when created via retainedSlice() and
retainedDuplicate() and derived from a pooled buffer
2016-05-17 11:16:13 +02:00
Norman Maurer
68cd670eb9 Remove ChannelHandlerInvoker
Motivation:

We tried to provide the ability for the user to change the semantics of the threading-model by delegate the invoking of the ChannelHandler to the ChannelHandlerInvoker. Unfortunually this not really worked out quite well and resulted in just more complexity and splitting of code that belongs together. We should remove the ChannelHandlerInvoker again and just do the same as in 4.0

Modifications:

Remove ChannelHandlerInvoker again and replace its usage in Http2MultiplexCodec

Result:

Easier code and less bad abstractions.
2016-05-17 11:14:00 +02:00
Norman Maurer
979dc3e3e4 Fix one more possible leak as a follow up of 341a235fea 2016-05-13 18:38:37 +02:00
Norman Maurer
341a235fea Fix possible leaks in Http2ServerDowngraderTest
Motivation:

We need to ensure we release all ReferenceCounted objects during tests to not leak.

Modifications:

Fix possible leaks by calling release()

Result:

No more leaks in tests.
2016-05-13 17:38:26 +02:00
Norman Maurer
4b24b0aac8 Annotate Http2ServerDowngrader with @UnstableApi
Motivation:

Everything in the http2 package should be considered unstable for now

Modifications:

Add missing annotation on Http2ServerDowngrader

Result:

Clearly mark class as unstable.
2016-05-13 08:41:46 +02:00