Commit Graph

9461 Commits

Author SHA1 Message Date
Norman Maurer
6283a78e4f HTTP2: Guard against empty DATA frames (without end_of_stream flag) set (#9461)
Motivation:

It is possible for a remote peer to flood the server / client with empty DATA frames (without end_of_stream flag) set and so cause high CPU usage without the possibility to ever hit a limit. We need to guard against this.

See CVE-2019-9518

Modifications:

- Add a new config option to AbstractHttp2ConnectionBuilder and sub-classes which allows to set the max number of consecutive empty DATA frames (without end_of_stream flag). After this limit is hit we will close the connection. A limit of 10 is used by default.
- Add unit tests

Result:

Guards against CVE-2019-9518
2019-08-14 10:02:32 +02:00
Norman Maurer
c6c679597f HTTP2: Add protection against remote control frames that are triggered by a remote peer (#9460)
Motivation:

Due how http2 spec is defined it is possible by a remote peer to flood us with frames that will trigger control frames as response, the problem here is that the remote peer can also just stop reading these (while still produce more of these) and so may drive us to the pointer where we either run out of memory or burn all CPU. To protect against this we need to implement some kind of limit that will tear down connections that cause the above mentioned situation.

See CVE-2019-9512 / CVE-2019-9514 / CVE-2019-9515

Modifications:

- Add Http2ControlFrameLimitEncoder which limits the number of queued control frames that were caused because of the remote peer.
- Allow to insert ths Http2ControlFrameLimitEncoder by setting AbstractHttp2ConnectionBuilder.encoderEnforceMaxQueuedControlFrames(...) to a number higher then 0. The default is 10000 which provides some protection by default but will hopefully not cause too many false-positives.
- Add unit tests

Result:

Protect against DDOS due control frames. Fixes CVE-2019-9512 / CVE-2019-9514 / CVE-2019-9515 .
2019-08-14 10:02:24 +02:00
Nick Hill
bc22bfa320 Use alloc().heapBuffer(...) to allocate new heap buffer.
Motivation

Underlying array allocations in UnpooledHeapByteBuf are intended be done
via the protected allocateArray(int) method, so that they can be tracked
and/or overridden by subclasses, for example
UnpooledByteBufAllocator$InstrumentedUnpooledHeapByteBuf or #8015. But
it looks like an explicit allocation was missed in the copy(int,int)
method.

Modification

Just use alloc().heapBuffer(...) for the allocation

Result

No possibility of "missing" array allocations when ByteBuf#copy is used.
2019-08-13 10:52:52 +02:00
Norman Maurer
02408b306c Delay Http2ConnectionPrefaceAndSettingsFrameWrittenEvent by one EventLoop tick when using the Http2FrameCodec (#9442)
Motivation:

We should delay the firing of the Http2ConnectionPrefaceAndSettingsFrameWrittenEvent by one EventLoop tick when using the Http2FrameCodec to ensure all handlers are added to the pipeline before the event is passed through it.

This is needed to workaround a race that could happen when the preface is send in handlerAdded(...) but a later handler wants to act on the event.

Modifications:

Offload firing of the event to the EventExecutor.

Result:

Fixes https://github.com/netty/netty/issues/9432.
2019-08-13 10:50:40 +02:00
Norman Maurer
2f0c00187a Use delegated docker mount option to speedup builds (#9441)
Motivation:

As we use the docker files for the CI we should use the delegated mount option to speed up builds.

See https://docs.docker.com/docker-for-mac/osxfs-caching/#delegated

Modifications:

Use delegated mount option

Result:

Faster builds when using docker
2019-08-13 10:27:33 +02:00
Norman Maurer
16e290796d Always wrap X509ExtendedTrustManager when using OpenSSL and JDK < 11 (#9443)
Motivation:

When using OpenSSL and JDK < 11 is used we need to wrap the user provided X509ExtendedTrustManager to be able to support TLS1.3. We had a check in place that first tried to see if wrapping is needed at all which could lead to missleading calls of the user provided trustmanager. We should remove these calls and just always wrap if needed.

Modifications:

Always wrap if OpenSSL + JDK < 11 and TLS1.3 is supported

Result:

Less missleading calls to user provided trustmanager
2019-08-13 10:26:56 +02:00
Nico Kruber
d285623925 Try to load native linux libraries with matching classifier first (#9411)
Motivation:

Users' runtime systems may have incompatible dynamic libraries to the ones our
tcnative wrappers link to. Unfortunately, we cannot determine and catch these
scenarios (in which the JVM crashes) but we can make a more educated guess on
what library to load and try to find one that works better before crashing.

Modifications:

1) Build dynamically linked openSSL builds for more OSs (netty-tcnative)
2) Load native linux libraries with matching classifier (first)

Result:

More developers / users can use the dynamically-linked native libraries.
2019-08-12 08:48:58 +02:00
violetagg
a4c9fb7729 Do not cache local/remote address when creating EpollDatagramChannel with InternetProtocolFamily (#9436)
Motivation:

EpollDatagramChannel#localAddress returns wrong information when
EpollDatagramChannel is created with InternetProtocolFamily,
and EpollDatagramChannel#localAddress is invoked BEFORE the actual binding.

This is a regression caused by change
e17ce934da

Modifications:

EpollDatagramChannel() and EpollDatagramChannel(InternetProtocolFamily family)
do not cache local/remote address

Result:

Rebinding on the same address without "reuse port" works
EpollDatagramChannel#localAddress returns correct address
2019-08-11 09:09:26 +02:00
Andrey Mizurov
4340f6c116 Set the ORIGIN header from a custom headers if present (#9435)
Motivation:

Allow to set the ORIGIN header value from custom headers in WebSocketClientHandshaker

Modification:

Only override header if not present already

Result:

More flexible handshaker usage
2019-08-11 08:22:31 +02:00
Nick Hill
dcd145864a Fix ByteBufUtil#writeUtf8 subsequence split surrogate edge-case bug (#9437)
Motivation:

#9224 introduced overrides of ByteBufUtil#writeUtf8(...) and related
methods to operate on a sub-CharSequence directly to save having to
allocate substrings, but it missed an edge case where the subsequence
does not extend to the end of the CharSequence and the last char in the
sequence is a high surrogate.

Due to the catch-IndexOutOfBoundsException optimization that avoids an
additional bounds check, it would be possible to read past the specified
end char index and successfully decode a surrogate pair which would
otherwise result in a '?' byte being written.

Modifications:

- Check for end-of-subsequence before reading next char after a high
surrogate is encountered in the
writeUtf8(AbstractByteBuf,int,CharSequence,int,int) and
utf8BytesNonAscii methods
- Add unit test for this edge case

Result:

Bug is fixed.

This removes the bounds-check-avoidance optimization but it does not
appear to have a measurable impact on benchmark results, including when
the char sequence contains many surrogate pairs (which should be rare in
any case).
2019-08-10 20:54:43 +02:00
noSim
e8c2986cf1 Fix HttpUtil.getCharset to not throw illegal charset exception (#9439)
Motivation:

If the HttpUtil.getCharset method is called with an illegal charset like
"charset=!illegal!" it throws an IllegalCharsetNameException. But the javadoc
states, that defaultCharset is returned if incorrect header value. Since the
client sending the request sets the header value this should not crash.

Modification:

HttpUtil.getCharset catches the IllegalCharsetNameException and returns the
defualt value.

Result:

HttpUtil.getCharset does not throw IllegalCharsetNameException any more.
2019-08-10 19:16:12 +02:00
Nico Kruber
68e49af483 Fix native-build/target/lib wanted but build in native-build/target/lib64 (#9410)
Motivation:

On openSUSE (probably more), 64 bit builds use lib64, e.g. /usr/lib64, and
configure picks this up and builds the native library in
native-build/target/lib64 where maven is not looking.

Modifications:

Explicitly specify --libdir=${project.build.directory}/native-build/target/lib
during configuration.

Result:

Maven uses the correct lib directory.
2019-08-07 07:58:04 +00:00
Nico Kruber
c1c43af060 Update os-maven-plugin version to match netty-tcnative (#9409)
Motivation:

Since both projects (to some extend) rely on classifier parsing via the
os-maven-plugin, they should ideally use the same version in case the parsing
changed.

Modifications:

Upgrade os-maven-plugin from 1.6.0 to 1.6.2

Result:

Same os-maven-plugin with same parsing logic.
2019-08-03 10:39:27 +00:00
Dmitriy Dumanskiy
10ee697557 Allow to turn off Utf8FrameValidator creation for websocket with Bina… (#9417)
…ryWebSocketFrames

Motivation:

`Utf8FrameValidator` is always created and added to the pipeline in `WebSocketServerProtocolHandler.handlerAdded` method. However, for websocket connection with only `BinaryWebSocketFrame`'s UTF8 validator is unnecessary overhead. Adding of `Utf8FrameValidator` could be easily avoided by extending of `WebSocketDecoderConfig` with additional property.

Specification requires UTF-8 validation only for `TextWebSocketFrame`.

Modification:

Added `boolean WebSocketDecoderConfig.withUTF8Validator` that allows to avoid adding of `Utf8FrameValidator` during pipeline initialization.

Result:

Less overhead when using only `BinaryWebSocketFrame`within web socket.
2019-08-03 10:36:54 +00:00
Dmitriy Dumanskiy
ca2e8c3b69 Motivation:
Look like `EmbeddedChannelPipeline` should also override `onUnhandledInboundMessage(ChannelHandlerContext ctx, Object msg)` in order to do not print "Discarded message pipeline" because in case of `EmbeddedChannelPipeline` discarding actually not happens.

This fixes next warning in the latest netty version with websocket and `WebSocketServerCompressionHandler`:

```
13:36:36.231 DEBUG- Decoding WebSocket Frame opCode=2
13:36:36.231 DEBUG- Decoding WebSocket Frame length=5
13:36:36.231 DEBUG- Discarded message pipeline : [JdkZlibDecoder#0, DefaultChannelPipeline$TailContext#0]. Channel : [id: 0xembedded, L:embedded - R:embedded].
```

Modification:

Override correct method

Result:
Follow up fix after https://github.com/netty/netty/pull/9286
2019-08-03 10:31:48 +00:00
Dmitriy Dumanskiy
aea6c67cfb Decrease the level of logging in WebSocketFrameEncoder/Decoder (#9415)
Motivation:

Our QA servers are spammed with this messages:

13:57:51.560 DEBUG- Decoding WebSocket Frame opCode=1
13:57:51.560 DEBUG- Decoding WebSocket Frame length=4
I think this is too much info for debug level. It is better to move it to trace level.

Modification:

logger.debug changed to logger.trace for WebSocketFrameEncoder/Decoder

Result:

Less messages in Debug mode.
2019-08-03 10:17:52 +00:00
Per Lundberg
eef72b022a Future.java: Fix typos in Javadoc (#9391)
Motivation:

Docs should have no typos

Modifications:

Fix a few typos

Result:

More correct docs.
2019-07-24 07:38:08 +02:00
Norman Maurer
d6c05e9b31 HTTP/2: Ensure newStream() is called only once per connection upgrade and the correct handler is used (#9396)
Motivation:

306299323c introduced some code change to move the responsibility of creating the stream for the upgrade to Http2FrameCodec. Unfortunaly this lead to the situation of having newStream().setStreamAndProperty(...) be called twice. Because of this we only ever saw the channelActive(...) on Http2StreamChannel but no other events as the mapping was replaced on the second newStream().setStreamAndProperty(...) call.

Beside this we also did not use the correct handler for the upgrade stream in some cases

Modifications:

- Just remove the Http2FrameCodec.onHttpClientUpgrade() method and so let the base class handle all of it. The stream is created correctly as part of the ConnectionListener implementation of Http2FrameCodec already.
- Consolidate logic of creating stream channels
- Adjust unit test to capture the bug

Result:

Fixes https://github.com/netty/netty/issues/9395
2019-07-23 21:13:33 +02:00
YuanHu
90b788bef7 Recycler availableSharedCapacity will be slowly exhausted due missing reclaimSpace(...) call (#9394)
Motivation:

We did miss to call reclaimSpace(...) in one case which can lead to the situation of having the Recycler to not correctly reclaim space and so just create new objects when not needed.

Modifications:

Correctly call reclaimSpace(...)

Result:

Recycler correctly reclaims space in all situations.
2019-07-21 21:07:12 +02:00
Norman Maurer
a677e62dd0 HTTP/2 multiplex: Correctly process buffered inbound data even if autoRead is false (#9389)
Motivation:

When using the HTTP/2 multiplex implementation we need to ensure we correctly drain the buffered inbound data even if the RecvByteBufallocator.Handle tells us to stop reading in between.

Modifications:

Correctly loop through the buffered inbound data until the user does stop to request from it.

Result:

Fixes https://github.com/netty/netty/issues/9387.

Co-authored-by: Bryce Anderson <banderson@twitter.com>
2019-07-21 20:58:50 +02:00
Norman Maurer
f6005c5b2d Reuse Http2FrameStreamEvent instances to reduce GC pressure (#9392)
Motivation:

We can easily reuse the Http2FrameStreamEvent instances and so reduce GC pressure as there may be multiple events per streams over the life-time.

Modifications:

Reuse instances

Result:

Less allocations
2019-07-21 20:37:28 +02:00
Norman Maurer
e2525ced48 Update java versions (#9393)
Motivation:

There were new openjdk releases

Modifications:

Update releases to latest

Result:

Use latest openjdk versions on CI
2019-07-21 20:35:04 +02:00
Norman Maurer
e4995be33c Use allocator when constructing ByteBufHolder sub-types or use Unpool… (#9377)
Motivation:

In many places Netty uses Unpooled.buffer(0) while should use EMPTY_BUFFER. We can't change this due to back compatibility in the constructors but can use Unpooled.EMPTY_BUFFER in some cases to ensure we not allocate at all. In others we can directly use the allocator either from the Channel / ChannelHandlerContext or the request / response.

Modification:

- Use Unpooled.EMPTY_BUFFER where possible
- Use allocator where possible

Result:

Fixes #9345 for websockets and http package
2019-07-18 10:36:03 +02:00
Norman Maurer
82ac20bec0 Cache the ChannelHandlerContext used in Http2StreamChannelBootstrap (#9382)
Motivation:

At the moment we lookup the ChannelHandlerContext used in Http2StreamChannelBootstrap each time the open(...) method is invoked. This is not needed and we can just cache it for later usage.

Modifications:

Cache ChannelHandlerContext in volatile field.

Result:

Speed up open(...) method implementation when called multiple times
2019-07-18 10:29:14 +02:00
Norman Maurer
2ef4b16138 Add websocket encoder / decoder in correct order to the pipeline when HttpServerCodec is used (#9386)
Motivation:

We need to ensure we place the encoder before the decoder when doing the websockets upgrade as the decoder may produce a close frame when protocol violations are detected.

Modifications:

- Correctly place encoder before decoder
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/9300
2019-07-18 10:19:33 +02:00
Bryce Anderson
8e370fbbcd Fix an NPE in AbstractHttp2StreamChannel (#9379)
Motivation:

If a read triggers a AbstractHttp2StreamChannel to close we can
get an NPE in the read loop.

Modifications:

Make sure that the inboundBuffer isn't null before attempting to
continue the loop.

Result:

No NPE.
Fixes #9337
2019-07-17 20:13:09 +02:00
Norman Maurer
9ebb6b6b56 Add testcase to prove that ET semantics for eventFD are correct (#9385)
Motivation:

We recently made a change to use ET for the eventfd and not trigger a read each time. This testcase proves everything works as expected.

Modifications:

Add testcase that verifies thqat the wakeups happen correctly

Result:

More tests
2019-07-17 12:29:10 +02:00
Norman Maurer
4e167662cc Update to adopt@1.8.212-04 (#9384)
Motivation:

We should use latest jdk 1.8 release

Modifications:

Update to adopt@1.8.212-04

Result:

Use latest jdk 1.8 on ci
2019-07-17 09:42:14 +02:00
Norman Maurer
7f73eace6b Use latest OpenJDK13 EA release (#9378)
Motivation:

A new EA release for OpenJDK13 was released

Modifications:

Update EA version

Result:

Use latest OpenJDK 13 EA on ci
2019-07-17 09:30:15 +02:00
Dmitriy Dumanskiy
390fa95f32 prefer instanceOf instead of getClass() (#9366)
Motivation:

`instanceOf` doesn't perform null check like `getClass()` does. So `instanceOf` may be faster. However, it not true for all cases, as C2 could eliminate these null checks for `getClass()`.

Modification:

Replaced `string.getClass() == AsciiString.class` with `string instanceof AsciiString`.

Proof:

```
@BenchmarkMode(Mode.Throughput)
@Fork(value = 1)
@State(Scope.Thread)
@Warmup(iterations = 5, time = 1, batchSize = 1000)
@Measurement(iterations = 10, time = 1, batchSize = 1000)
public class GetClassInstanceOf {

    Object key;

    @Setup
    public void setup() {
        key = "123";
    }

    @Benchmark
    public boolean getClassEquals() {
        return key.getClass() == String.class;
    }

    @Benchmark
    public boolean instanceOf() {
        return key instanceof String;
    }

}
```

```
Benchmark                           Mode  Cnt       Score      Error  Units
GetClassInstanceOf.getClassEquals  thrpt   10  401863.130 ± 3092.568  ops/s
GetClassInstanceOf.instanceOf      thrpt   10  421386.176 ± 4317.328  ops/s
```
2019-07-16 21:25:59 +02:00
Emily Littleworth
0f8685ae9b Return null in HttpPostRequestEncoder (#9352)
Motivation:

If the encoded value of a form element happens to exactly hit
the chunk limit (8096 bytes), the post request encoder will
throw a NullPointerException.

Modifications:

Catch the null case and return.

Result:

No NPE.
2019-07-16 13:30:07 +02:00
Norman Maurer
7e818352e2 Move responsibility for creating upgrade stream to Http2FrameCodec (#9360)
Motivation:

The Http2FrameCodec should be responsible to create the upgrade stream.

Modifications:

Move code to create stream to Http2FrameCodec

Result:

More correct responsibility
2019-07-16 13:27:51 +02:00
Nick Hill
afbbc11c8f Fix epoll spliceTo file descriptor with offset (#9369)
Motivation

The AbstractEpollStreamChannel::spliceTo(FileDescriptor, ...) methods
take an offset parameter but this was effectively ignored due to what
looks like a typo in the corresponding JNI function impl. Instead it
would always use the file's own native offset.

Modification

- Fix typo in netty_epoll_native_splice0() and offset accounting in
AbstractEpollStreamChannel::SpliceFdTask.
- Modify unit test to include an invocation of the public spliceTo
method using non-zero offset.

Result

spliceTo FD methods work as expected when an offset is provided.
2019-07-16 13:23:36 +02:00
Dmitriy Dumanskiy
ef24732640 Cleanup in websockets, throw exception before allocating response if possible (#9361)
Motivation:

While fixing #9359 found few places that could be patched / improved separately.

Modification:

On handshake response generation - throw exception before allocating response objects if request is invalid.

Result:

No more leaks when exception is thrown.
2019-07-16 13:21:16 +02:00
Norman Maurer
b0b02d51d2 Add deprecation to Http2StreamChannelBootstrap.open0(...) as it was marked as public by mistake (#9372)
Motivation:

Mark Http2StreamChannelBootstrap.open0(...) as deprecated as the user should not use it. It was marked as public by mistake.

Modifications:

Add deprecation warning.

Result:

User will be aware the method should not be used directly.
2019-07-16 13:09:28 +02:00
Norman Maurer
a136a38778 Allow to disable automatically sending PING acks. (#9338)
Motivation:

There are situations where the user may want to be more flexible when to send the PING acks for various reasons or be able to attach a listener to the future that is used for the ping ack. To be able to do so we should allow to manage the acks manually.

Modifications:

- Add constructor to DefaultHttp2ConnectionDecoder that allows to disable the automatically sending of ping acks (default is to send automatically to not break users)
- Add methods to AbstractHttp2ConnectionHandlerBuilder (and sub-classes) to either enable ot disable auto acks for pings
- Make DefaultHttp2PingFrame constructor public that allows to write acks.
- Add unit test

Result:

More flexible way of handling acks.
2019-07-12 18:15:26 +02:00
Nick Hill
a6abff75a6 Epoll: Don't wake event loop when splicing (#9354)
Motivation

I noticed this while looking at something else.
AbstractEpollStreamChannel::spliceQueue is an MPSC queue but only
accessed from the event loop. So it could be just changed to e.g. an
ArrayDeque. This PR instead reverts to using is as an MPSC queue to
avoid dispatching a task to the EL, as appears was the original
intention.

Modification

Change AbstractEpollStreamChannel::spliceQueue to be volatile and lazily
initialized via double-checked locking. Add tasks directly to the queue
from the public methods rather than possibly waking the EL just to
enqueue.

An alternative is just to change PlatformDependent.newMpscQueue() to new
ArrayDeque() and be done with it :)

Result

Less disruptive channel/fd-splicing.
2019-07-12 18:14:06 +02:00
Andrey Mizurov
e9fec0a710 Fixed incorrect Sec-WebSocket-Origin header for v13, see #9134 (#9312)
Motivation:

Based on https://tools.ietf.org/html/rfc6455#section-1.3 - for non-browser
clients, Origin header field may be sent if it makes sense in the context of those clients.

Modification:

Replace Sec-WebSocket-Origin to Origin

Result:

Fixes #9134 .
2019-07-12 12:06:11 +02:00
Robin Gong
40228411d7 feat(example-mqtt): new MQTT heartBeat broker and client examples (#9336)
Motivation:

Recently I'm going to build MQTT broker and client based on Netty. I had MQTT encoder and decoder founded, while no basic examples. So I'm going to share my simple heartBeat MQTT broker and client as an example.

Modification:

New MQTT heartBeat example under io.netty.example/mqtt/heartBeat/.

Result:

Client would send CONNECT and PINGREQ(heartBeat message).
  - CONNECT: once channel active
  - PINGREQ: once IdleStateEvent triggered, which is 20 seconds in this example
Client would discard all messages it received.
MQTT broker could handle CONNECT, PINGREQ and DISCONNECT messages.
  - CONNECT: send CONNACK back
  - PINGREQ: send PINGRESP back
  - DISCONNECT: close the channel
Broker would close the channel if 2 heartBeat lost, which set to 45 seconds in this example.
2019-07-10 12:20:12 +02:00
Farid Zakaria
794bb6c7b6 Introduce SslMasterKeyHandler (#8653)
Motivation

Debugging SSL/TLS connections through wireshark is a pain -- if the cipher used involves Diffie-Hellman then it is essentially impossible unless you can have the client dump out the master key [1]

This is a work-in-progress change (tests & comments to come!) that introduces a new handler you can set on the SslContext to receive the master key & session id. I'm hoping to get feedback if a change in this vein would be welcomed.

An implementation that conforms to Wireshark's NSS key log[2] file is also included.

Depending on feedback on the PR going forward I am planning to "clean it up" by adding documentation, example server & tests. Implementation will need to be finished as well for retrieving the master key from the OpenSSL context.

[1] https://jimshaver.net/2015/02/11/decrypting-tls-browser-traffic-with-wireshark-the-easy-way/
[2] https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format

Modification

- Added SslMasterKeyHandler
- An implementation of the handler that conforms to Wireshark's key log format is included.

Result:

Be able to debug SSL / TLS connections more easily.

Signed-off-by: Farid Zakaria <farid.m.zakaria@gmail.com>
2019-07-10 12:16:56 +02:00
jingene
af614e4d6e Change the netty.io homepage scheme(http -> https) (#9344)
Motivation:

Netty homepage(netty.io) serves both "http" and "https".
It's recommended to use https than http.
Modification:

I changed from "http://netty.io" to "https://netty.io"
Result:

No effects.
2019-07-09 21:10:14 +02:00
Norman Maurer
cd7670dcaa HTTP2: Always apply the graceful shutdown timeout if configured (#9340)
Motivation:

Http2ConnectionHandler (and sub-classes) allow to configure a graceful shutdown timeout but only apply it if there is at least one active stream. We should always apply the timeout. This is also true when we try to send a GO_AWAY and close the connection because of an connection error.

Modifications:

- Always apply the timeout if one is configured
- Add unit test

Result:

Always respect gracefulShutdownTimeoutMillis
2019-07-09 21:06:22 +02:00
Norman Maurer
462e88af7e DecoratingHttp2ConnectionEncoder.consumeRemoteSettings must not throw if delegate is instance of Http2SettingsReceivedConsumer (#9343)
Motivation:

b3dba317d7 introduced the concept of Http2SettingsReceivedConsumer but did not correctly inplement DecoratingHttp2ConnectionEncoder.consumeRemoteSettings(...).

Modifications:

- Add missing `else` around the throws
- Add unit tests

Result:

Correctly implement DecoratingHttp2ConnectionEncoder.consumeRemoteSettings(...)
2019-07-09 14:41:29 +02:00
Nick Hill
760bd4ab90 Simplify HpackHuffmanDecoder table decode logic (#9335)
Motivation

The nice change made by @carl-mastrangelo in #9307 for lookup-table
based HPACK Huffman decoding can be simplified a little to remove the
separate flags field and eliminate some intermediate operations.

Modification

Simplify HpackHuffmanDecoder::decode logic including de-dup of the
per-nibble part.

Result

Less code, possibly better performance though not noticeable in a quick
benchmark.
2019-07-08 12:05:28 +02:00
Norman Maurer
f47cf0c4d3 Reduce object creation on Http2FrameCodec (#9333)
Motivation:

We don't need the extra ChannelPromise when writing headers anymore in Http2FrameCodec. This also means we cal re-use a ChannelFutureListener and so not need to create new instances all the time.

Modifications:

- Just pass the original ChannelPromise when writing headers
- Reuse the ChannelFutureListener

Result:

Two less objects created when writing headers for an not-yet created stream.
2019-07-06 09:14:16 +02:00
Norman Maurer
c5289fbe92 Fix test after bad cherry-pick of e96c37dea7 2019-07-05 10:24:27 +02:00
Norman Maurer
c5a602b272 Increase maxHeaderListSize for HpackDecoderBenchmark to be able to be… (#9321)
Motivation:

The previous used maxHeaderListSize was too low which resulted in exceptions during the benchmark run:

```
io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (8192)
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:103)
	at io.netty.handler.codec.http2.Http2Exception.headerListSizeError(Http2Exception.java:188)
	at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:231)
	at io.netty.handler.codec.http2.HpackDecoder$Http2HeadersSink.finish(HpackDecoder.java:545)
	at io.netty.handler.codec.http2.HpackDecoder.decode(HpackDecoder.java:132)
	at io.netty.handler.codec.http2.HpackDecoderBenchmark.decode(HpackDecoderBenchmark.java:85)
	at io.netty.handler.codec.http2.generated.HpackDecoderBenchmark_decode_jmhTest.decode_thrpt_jmhStub(HpackDecoderBenchmark_decode_jmhTest.java:120)
	at io.netty.handler.codec.http2.generated.HpackDecoderBenchmark_decode_jmhTest.decode_Throughput(HpackDecoderBenchmark_decode_jmhTest.java:83)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

```

Also we should ensure we only use ascii for header names.

Modifications:

Just use Integer.MAX_VALUE as limit

Result:

Be able to run benchmark without exceptions
2019-07-04 11:24:37 +02:00
jimin
d54678d645 Remove unnecessary code (#9303)
Motivation:

There are is some unnecessary code (like toString() calls) which can be cleaned up.

Modifications:

- Remove not needed toString() calls
- Simplify subString(...) calls
- Remove some explicit casts when not needed.

Result:

Cleaner code
2019-07-04 09:02:24 +02:00
Norman Maurer
d951140c56 Fix compile error introduced by bad cherry-pick of f7e8603d60 2019-07-04 09:02:23 +02:00
Norman Maurer
e177b74874 Use ByteProcessor in HpackHuffmanDecoder to reduce bound-checks and r… (#9317)
Motivation:

ff0045e3e1 changed HpackHuffmanDecoder to use a lookup-table which greatly improved performance. We can squeeze out another 3% win by using an ByteProcessor which will reduce the number of bound-checks / reference-count-checks needed by processing byte-by-byte.

Modifications:

Implement logic with ByteProcessor

Result:

Another ~3% perf improvement which shows up when using h2load to simulate load.

`h2load -c 100 -m 100 --duration 60 --warm-up-time 10 http://127.0.0.1:8080`

Before:

```
finished in 70.02s, 620051.67 req/s, 20.70MB/s
requests: 37203100 total, 37203100 started, 37203100 done, 37203100 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 37203100 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.21GB (1302108500) total, 41.84MB (43872600) headers (space savings 90.00%), 460.24MB (482598600) data
                     min         max         mean         sd        +/- sd
time for request:      404us     24.52ms     15.93ms      1.45ms    87.90%
time for connect:        0us         0us         0us         0us     0.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :    6186.64     6211.60     6199.00        5.18    65.00%
```

With this change:

```
finished in 70.02s, 642103.33 req/s, 21.43MB/s
requests: 38526200 total, 38526200 started, 38526200 done, 38526200 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 38526200 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.26GB (1348417000) total, 42.39MB (44444900) headers (space savings 90.00%), 466.25MB (488893900) data
                     min         max         mean         sd        +/- sd
time for request:      370us     24.89ms     15.52ms      1.35ms    88.02%
time for connect:        0us         0us         0us         0us     0.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :    6407.06     6435.19     6419.74        5.62    67.00%
```
2019-07-04 08:46:29 +02:00