Go to file

ZHANG Dapeng c2f893fcf3 Fix StreamBufferingEncoder GOAWAY bug (#11144 )

Motivation:

There is a bug in `StreamBufferingEncoder` such that when client receives GOWAY while there are pending streams due to MAX_CONCURRENT_STREAMS, we see the following error:
```
io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint.
        at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.checkNewStreamAllowed(DefaultHttp2Connection.java:896)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:748)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:668)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders0(DefaultHttp2ConnectionEncoder.java:201)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:167)
        at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:53)
        at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:153)
        at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:141)
        at io.grpc.netty.NettyClientHandler.createStreamTraced(NettyClientHandler.java:584) 
        at io.grpc.netty.NettyClientHandler.createStream(NettyClientHandler.java:567)
        at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:328)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
        at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)
        at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)
        at io.grpc.netty.WriteQueue$AbstractQueuedCommand.run(WriteQueue.java:213)
        at io.grpc.netty.WriteQueue.flush(WriteQueue.java:128)
        at io.grpc.netty.WriteQueue.drainNow(WriteQueue.java:114)
        at io.grpc.netty.NettyClientHandler.goingAway(NettyClientHandler.java:783)
        at io.grpc.netty.NettyClientHandler.access$300(NettyClientHandler.java:91)
        at io.grpc.netty.NettyClientHandler$3.onGoAwayReceived(NettyClientHandler.java:280)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.goAwayReceived(DefaultHttp2Connection.java:236)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.onGoAwayRead0(DefaultHttp2ConnectionDecoder.java:218)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onGoAwayRead(DefaultHttp2ConnectionDecoder.java:551)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onGoAwayRead(Http2InboundFrameLogger.java:119)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readGoAwayFrame(DefaultHttp2FrameReader.java:591)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:272)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)
```

The bug should come from the way that `StreamBufferingEncoder.writeHeaders()` handles the condition `connection().goAwayReceived()`. The current behavior is to delegate to `super.writeHeaders()` and let the stream fail, but this will end up with `Http2Exception` with the message "Maximum active streams violated for this endpoint" which is horrible. See e5951d46fc/codec-http2/src/main/java/io/netty/handler/codec/http2/StreamBufferingEncoder.java (L152-L155)

Modification:

Abort new stream immediately if goaway received *and* MAX_CONCURRENT_STREAM reached in `StreamBufferingEncoder` rather than delegating to the `writeHeaders()` method of its super class.

Result:

In the situation when GOAWAY received as well as MAX_CONCURRENT_STREAM exceeded, the client will fail the buffered streams with `Http2Error.NO_ERROR` and message "GOAWAY received" instead of "Maximum active streams violated for this endpoint".

Co-authored-by: Norman Maurer <norman_maurer@apple.com>

2021-04-19 16:06:13 +02:00

.github

Github Actions dawidd6/action-download-artifact set workflow_conclusion (#11096 )

2021-03-17 12:20:03 -07:00

.mvn

Use latest maven release (#9820 )

2019-11-27 14:45:48 +01:00

all

Update dependency declaration in all pom.xml (#10967 )

2021-01-27 10:25:36 +01:00

bom

Fix classifier usage in bom

2020-12-08 14:56:38 +01:00

buffer

Fix alignment handling for pooled direct buffers (#11106 )

2021-03-23 17:09:44 +01:00

codec

Add support for direct ByteBufs in JdkZlibEncoder and JdkZlibDecoder (#11057 )

2021-03-05 13:29:20 +01:00

codec-dns

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

codec-haproxy

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

codec-http

[Clean] Remove useless code (#11154 )

2021-04-19 10:10:07 +02:00

codec-http2

Fix StreamBufferingEncoder GOAWAY bug (#11144 )

2021-04-19 16:06:13 +02:00

codec-memcache

Introduce ByteBufConvertible interface (#11036 )

2021-02-26 15:03:58 +01:00

codec-mqtt

Simplity some code (#11000 )

2021-02-11 09:06:16 +01:00

codec-redis

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

codec-smtp

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

codec-socks

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

codec-stomp

Provide ability to extend StompSubframeEncoder and improve full stomp frame encoding (allocate one buffer for full frame considering the size of the headers) (#10778 )

2020-12-07 09:59:17 +01:00

codec-xml

Simplity some code (#11000 )

2021-02-11 09:06:16 +01:00

common

Log fewer stack traces from initialisation code (#11164 )

2021-04-19 09:17:32 +02:00

dev-tools

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

docker

Let's use gcc10 when cross-compiling for LSE support (#11112 )

2021-03-25 09:34:08 +01:00

example

Remove unnecessary this. (#11035 )

2021-02-26 12:13:59 +01:00

handler

Log fewer stack traces from initialisation code (#11164 )

2021-04-19 09:17:32 +02:00

handler-proxy

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

license

Enable nohttp check during the build (#10708 )

2020-10-23 15:26:25 +02:00

microbench

Minimize get byte multipart and fix buffer reuse (#11001 )

2021-02-26 14:26:24 +01:00

resolver

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

resolver-dns

Fix compile error introduced by 06739ed8906a2b71cea25a6932a01392db01981d

2021-03-08 12:17:34 +01:00

resolver-dns-native-macos

Use netty-jni-util 0.0.3.Final

2021-03-25 11:40:40 +01:00

scripts

Bring forward build automation changes (#11052 )

2021-03-02 17:44:03 +01:00

tarball

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite

SslHandler flushing with TCP Fast Open fix (#11077 )

2021-03-14 14:27:10 +01:00

testsuite-autobahn

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite-http2

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite-native

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite-native-image

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite-native-image-client

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite-native-image-client-runtime-init

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite-osgi

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

testsuite-shading

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

transport

Continue reading when the number of bytes is less then the configured… (#11089 )

2021-03-17 13:18:39 +01:00

transport-blockhound-tests

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

transport-native-epoll

Less blocking in ChunkedStream (#11150 )

2021-04-12 13:42:15 +02:00

transport-native-kqueue

AbstractKQueueChannel#writeFilter is invoked with the correct boolean depending on the ChannelOutboundBuffer state (#11128 )

2021-03-30 19:16:24 +02:00

transport-native-unix-common

Move SegmentedDatagramPacket to transport-native-unix-common (#11121 )

2021-03-29 14:10:38 +02:00

transport-native-unix-common-tests

Use Files.createTempFile(...) to ensure the file is created with proper permissions

2021-02-08 18:17:31 +01:00

transport-sctp

Use http in xmlns URIs to make maven release plugin happy again (#10788 )

2020-11-10 10:51:05 +01:00

transport-udt/lib/bin/lib/x86_64-MacOSX-gpp/jni

Replace reflection usage with MethodHandles when performance matters (#10097 )

2020-03-11 21:04:40 +01:00

.fbprefs

Updated Find Bugs configuration

2009-03-04 10:33:09 +00:00

.gitattributes

Include mvn wrapper to make setup of development env easier

2018-01-26 08:13:17 +01:00

.gitignore

Ignore .shelf/ folder generated by IntelliJ IDEA (#10445 )

2020-08-03 07:52:18 +02:00

.lgtm.yml

Enables lgtm.com to process this project and create a CodeQL database

2020-01-20 19:22:49 +01:00

CONTRIBUTING.md

Change the netty.io homepage scheme(http -> https) (#9344 )

2019-07-09 21:10:14 +02:00

LICENSE.txt

Enable nohttp check during the build (#10708 )

2020-10-23 15:26:25 +02:00

mvnw

Enable nohttp check during the build (#10708 )

2020-10-23 15:26:25 +02:00

mvnw.cmd

Enable nohttp check during the build (#10708 )

2020-10-23 15:26:25 +02:00

nohttp-checkstyle-suppressions.xml

Ensure Checkstyle suppression for dependency-reduced-pom.xml on Windows (#10899 )

2021-01-01 19:31:01 +01:00

nohttp-checkstyle.xml

Enable nohttp check during the build (#10708 )

2020-10-23 15:26:25 +02:00

NOTICE.txt

Fix License type of dnsinfo (#10773 )

2020-11-04 10:41:41 +01:00

pom.xml

Skip deployment of testsuite jars (#11127 )

2021-03-30 19:06:34 +02:00

README.md

Fix url in README.md (#10915 )

2021-01-11 07:50:43 +01:00

run-example.sh

Add DNS client examples for run-example.sh (#10283 )

2020-05-14 12:10:48 +02:00

SECURITY.md

Added a security policy (#10692 )

2020-10-15 20:40:05 +02:00

README.md

Netty Project

Netty is an asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients.

How to build

For the detailed information about building and developing Netty, please visit the developer guide. This page only gives very basic information.

You require the following to build Netty:

Latest stable OpenJDK 8
Latest stable Apache Maven
If you are on Linux, you need additional development packages installed on your system, because you'll build the native transport.

Note that this is build-time requirement. JDK 5 (for 3.x) or 6 (for 4.0+ / 4.1+) is enough to run your Netty-based application.

Branches to look

Development of all versions takes place in each branch whose name is identical to <majorVersion>.<minorVersion>. For example, the development of 3.9 and 4.1 resides in the branch '3.9' and the branch '4.1' respectively.

Usage with JDK 9+

Netty can be used in modular JDK9+ applications as a collection of automatic modules. The module names follow the reverse-DNS style, and are derived from subproject names rather than root packages due to historical reasons. They are listed below:

io.netty.all
io.netty.buffer
io.netty.codec
io.netty.codec.dns
io.netty.codec.haproxy
io.netty.codec.http
io.netty.codec.http2
io.netty.codec.memcache
io.netty.codec.mqtt
io.netty.codec.redis
io.netty.codec.smtp
io.netty.codec.socks
io.netty.codec.stomp
io.netty.codec.xml
io.netty.common
io.netty.handler
io.netty.handler.proxy
io.netty.resolver
io.netty.resolver.dns
io.netty.transport
io.netty.transport.epoll (native omitted - reserved keyword in Java)
io.netty.transport.kqueue (native omitted - reserved keyword in Java)
io.netty.transport.unix.common (native omitted - reserved keyword in Java)
io.netty.transport.rxtx
io.netty.transport.sctp
io.netty.transport.udt

Automatic modules do not provide any means to declare dependencies, so you need to list each used module separately in your module-info file.

README.md

Netty Project

Links

How to build

Branches to look

Usage with JDK 9+