Commit Graph

10247 Commits

Author SHA1 Message Date
Norman Maurer
02c460be14 Add support for UDP_GRO (#11120)
Motivation:

UDP_GRO can improve performance when reading UDP datagrams. This patch adds support for it.

See https://lwn.net/Articles/768995/

Modifications:

- Add recvmsg(...)
- Add support for UDP_GRO in recvmsg(...) and recvmmsg(...)
- Remove usage of recvfrom(...) and just always use recvmsg(...) or recvmmsg(...) to simplify things
- Refactor some code for sharing
- Add EpollChannelOption.UDP_GRO and the getter / setter in EpollDatagramConfig

Result:

UDP_GRO is supported when the underlying system supports it.
2021-03-29 08:53:46 +02:00
Norman Maurer
50a83ec8a8 DefaultThreadFactory must not use Thread.currentThread() when constructed without ThreadGroup (#11119)
Motivation:

We had a bug in out DefaulThreadFactory as it always retrieved the ThreadGroup to used during creation time when now explicit ThreadGroup was given. This is problematic as the Thread may die and so the ThreadGroup is destroyed even tho the DefaultThreadFactory is still used.

This could produce exceptions like:

java.lang.IllegalThreadStateException
        at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867)
        at java.lang.Thread.init(Thread.java:405)
        at java.lang.Thread.init(Thread.java:349)
        at java.lang.Thread.<init>(Thread.java:599)
        at io.netty.util.concurrent.FastThreadLocalThread.<init>(FastThreadLocalThread.java:60)
        at io.netty.util.concurrent.DefaultThreadFactory.newThread(DefaultThreadFactory.java:122)
        at io.netty.util.concurrent.DefaultThreadFactory.newThread(DefaultThreadFactory.java:106)
        at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)
        at io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)
        at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:978)
        at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:947)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:830)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:471)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
        at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323)
        at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:272)
        at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:239)
        at io.netty.incubator.codec.quic.QuicTestUtils.newServer(QuicTestUtils.java:138)
        at io.netty.incubator.codec.quic.QuicTestUtils.newServer(QuicTestUtils.java:143)
        at io.netty.incubator.codec.quic.QuicTestUtils.newServer(QuicTestUtils.java:147)
        at io.netty.incubator.codec.quic.QuicStreamFrameTest.testCloseHalfClosure(QuicStreamFrameTest.java:48)
        at io.netty.incubator.codec.quic.QuicStreamFrameTest.testCloseHalfClosureUnidirectional(QuicStreamFrameTest.java:35)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:748)

Modifications:

- If the user dont specify a ThreadGroup we will just pass null to the constructor of FastThreadLocalThread and so let it retrieve on creation time
- Adjust tests

Result:

Don't risk to see IllegalThreadStateExceptions.
2021-03-26 18:46:55 +01:00
Chris Vest
aa1b887536 Fix offset type for new PlatformDependent.put* methods
Motivation:
The offsets were accidentally typed as int, where they should have been typed as long.

Modification:
Change type of offset arguments to PlatformDependent.put*(Object,int,?) from int to long.

Result:
It is now possible to use these methods to store to memory at absolute memory addresses.
2021-03-25 16:49:43 +01:00
Norman Maurer
d39cc314c9 Fix build for aarch64 (#11116)
Motivation:

We had a typo and so the build for aarch64 failed before

Modifications:

Fix typo

Result:

Build for aarch64 works again
2021-03-25 12:07:15 +01:00
Norman Maurer
8d483b58b3 Use netty-jni-util 0.0.3.Final
netty-jni-util 0.0.2.Final is incompatible with static linking. Before
the netty-jni-util dependency was introduced netty-tcnative supported
static linking via NETTY_BUILD_STATIC. netty-jni-util 0.0.3.Final adds
static linking compatibility.

Modifications:

Bump netty-jni-util to version 0.0.3.Final and update to its new API
which requires the caller to manage packagePrefix.

Result:

Using latest version of netty-jni-util and restored static linking
compatibility.
2021-03-25 11:40:40 +01:00
Chris Vest
8ae59c1e7b Expose on/off heap agnostic Unsafe accessor methods
Motivation:
These are necessary for creating a buffer implementation that uses Unsafe, and works generically for both on-heap and off-heap memory.

Modification:
PlatformDependent previously forced clients to decide if they are working on on-heap memory, or off-heap memory, by giving accessors distinct APIs for each.
What is added here, are generic accessors that work the same in either case.

Result:
We can now make an Unsafe-based buffer implementation that is agnostic to whether the memory is on- or off-heap.
2021-03-25 11:38:18 +01:00
Norman Maurer
e3086d50e9 Let's use gcc10 when cross-compiling for LSE support (#11112)
Motivation:

LSE (https://mysqlonarm.github.io/ARM-LSE-and-MySQL/) can have a huge performance difference. Let's ensure we use a compiler that can support it.

Modifications:

Update to gc10 when cross-compiling as it supports LSE and enables it by default

Result:

More optimized builds for aarch64
2021-03-25 09:34:08 +01:00
Norman Maurer
0e16f0e819 Allow to have an empty path when convert a CONNECT request (#11108)
Motivation:

CONNECT requests have no path defined as stated in the HTTP/2 spec, at the moment we will throw an exception if we try to convert such a request to HTTP/1.1

Modifications:

- Don't throw an exception if we try to convert a HTTP/2 CONNECT request that has no path
- Add unit test

Result:

Related to https://github.com/netty/netty-incubator-codec-http3/pull/112.
2021-03-24 10:52:08 +01:00
Norman Maurer
9c39c0de35 Ensure we can correctly propagate exceptions to streams even if endStream flag is set (#11105)
Motivation:

We need to ensure we are still be able to correctly map errors to streams in all cases. The problem was that we sometimes called closeStreamRemote(...) in a finally block and so closed the underyling stream before the actual exception was propagated. This was only true in some cases and not in all. Generally speaking we should only call closeStreamRemote(...) if there was no error as in a case of error we should generate a RST frame.

Modifications:

- Only call closeStreamRemote(...) if no exeption was thrown and so let the Http2ConnectionHandler handle the exception correctly
- Add unit tests

Result:

Correctly handle errors even when endStream is set to true
2021-03-23 20:28:00 +01:00
Chris Vest
9ba653c851 Fix alignment handling for pooled direct buffers (#11106)
Motivation:
Alignment handling was broken, and basically turned into a fixed offset into each allocation address regardless of its initial value, instead of ensuring that the allocated address is either aligned or bumped to the nearest alignment offset.
The brokenness of the alignment handling extended so far, that overlapping ByteBuf instances could even be created, as was seen in #11101.

Modification:
Instead of fixing the per-allocation pointer bump, we now ensure that 1) the minimum page size is a whole multiple of the alignment, and 2) the reference memory for each chunk is bumped to the nearest aligned address, and finally 3) ensured that the reservations are whole multiples of the alignment, thus ensuring that the next allocation automatically occurs from an aligned address.

Incidentally, (3) above comes for free because the reservations are in whole pages, and in (1) we ensured that pages are sized in whole multiples of the alignment.

In order to ensure that the memory for a chunk is aligned, we introduce some new PlatformDependent infrastructure.
The PlatformDependent.alignDirectBuffer will produce a slice of the given buffer, and the slice will have an address that is aligned.
This method is plainly available on ByteBuffer in Java 9 onwards, but for pre-9 we have to use Unsafe, which means it can fail and might not be available on all platforms.
Attempts to create a PooledByteBufAllocator that uses alignment, when this is not supported, will throw an exception.
Luckily, I think use of aligned allocations are rare.

Result:
Aligned pooled byte bufs now work correctly, and never have any overlap.

Fixes #11101
2021-03-23 17:09:44 +01:00
Bennett Lynch
7753431e48 Do not send GOAWAY frame before connection preface (#11107)
Motivation

A GOAWAY frame (or any other HTTP/2 frame) should not be sent before the
connection preface. Clients that immediately close the channel may
currently attempt to send a GOAWAY frame before the connection preface,
resulting in servers receiving a seemingly-corrupt connection preface.

Modifications

* Ensure that the preface has been sent before attempting to
automatically send a GOAWAY frame as part of channel shutdown logic
* Add unit test that only passes with new behavior

Result

Fixes https://github.com/netty/netty/issues/11026

Co-authored-by: Bennett Lynch <Bennett-Lynch@users.noreply.github.com>
2021-03-23 09:05:49 +01:00
Stuart Douglas
56703b93ba
Fix exception if Response has content (#11093)
Motivation:

If compression is enabled and the HttpResponse also
implements HttpContent (but not LastHttpContent) then
the buffer will be freed to eagerly.

Modification:

I retain the buffer the same way that is done for the LastHttpContent case.

Note that there is another suspicious looking call a few lines above (if beginEncode returns null). I am not sure if this should also be retained.

Result:

Fixes #11092
2021-03-21 14:50:05 +01:00
Norman Maurer
a25103f5a8 Fix netty-build dependency
Motivation:

netty-build is now called netty-build-common

Modifications:

Rename netty-build to netty-build-common

Result:

Be able to compile branch again
2021-03-21 14:48:54 +01:00
Norman Maurer
22088cd4cd Update netty-build version (#11103)
Motivation:

We released a new version of netty-build

Modifications:

Update to latest version

Result:

Use latest netty-build version
2021-03-20 12:46:32 +01:00
Norman Maurer
98b9158f66 Don't bind to a specify port during SSLEngine tests as the port may also be used (#11102)
Motivation:

We should always bind to port 0 to ensure we not try to bind to a port that is already in use during our tests.
As we missed to do this in one test we did see test failures sometimes.

This showed up here:
https://pipelines.actions.githubusercontent.com/obCqqsCMwwGme5y2aRyYOiZvWeJK4O0EY5AYRUDMSELILdqEjV/_apis/pipelines/1/runs/1963/signedlogcontent/18?urlExpires=2021-03-19T12%3A41%3A21.4370902Z&urlSigningMethod=HMACV1&urlSignature=zL6O0msEkghT%2B0hOAL1lqLK66SR0Mp99QIjiau1yPe4%3D

Modifications:

- Use new InetSocketAddress(0)

Result:

Fixes possible test failures.
2021-03-20 12:46:08 +01:00
Elliotte Rusty Harold
5451b978c4
Fix grammar in javadoc (#11090)
Motivation:

There was some grammar / spelling error in the javadocs

Modification:

Fix error

Result:

Cleanup
2021-03-19 10:02:23 +01:00
Chris Vest
654a54bbad Make CompositeByteBuf throw IllegalStateException when components are missing (#11100)
Motivation:
Components in a composite buffer can "go missing" if the composite is a slice of another composite and the parent has changed its layout.

Modification:
Where we would previously have thrown a NullPointerException, we now have a null-check for the component, and we instead throw an IllegalStateException with a more descriptive message.

Result:
It's now a bit easier to understand what is going on in these situations.

Fixes #10908
2021-03-18 17:54:06 +01:00
Scott Mitchell
5deaf94bfe Github Actions dawidd6/action-download-artifact set workflow_conclusion (#11096)
Motivation:
Newer versions of dawidd6/action-download-artifact changed the default
workflow_conclusion from "completed" to "completed, success" which can
result in download failures if the associated job fails, which is
expected when tests fail.

Modifications:
- Explicitly set the workflow_conclusion to "completed"

Result:
Test failures which result in build failures will still download test
data and generate reports after updating
dawidd6/action-download-artifact.
2021-03-17 12:20:03 -07:00
Norman Maurer
163b2b659c Continue reading when the number of bytes is less then the configured… (#11089)
... number of bytes when using DatagramChannels

Motivation:

In our FixedRecvByteBufAllocator we dont continue to read if the number of bytes is less then what was configured. This is correct when using it for TCP but not when using it for UDP. When using UDP the number of bytes is the maximum of what we want to support but we often end up processing smaller datagrams in general. Because of this we should use contineReading(UncheckedBooleanSupplier) to determite if we should continue reading

Modifications:

- use contineReading(UncheckedBooleanSupplier) for DatagramChannels

Result:

Read more then once in the general case for DatagramChannels with the default config
2021-03-17 13:18:39 +01:00
Norman Maurer
817052d019 Allow to configure the maximum number of message to write per eventloop (#11086)
Motivation:

Allow to configure the maximum number of messages to write per eventloop run. This can be useful to ensure we read data in a timely manner and not let writes dominate the CPU time. This is especially useful in protocols like QUIC where you need to read "fast enough" as otherwise you may not read the ACKs fast enough.

Modifications:

- Add new ChannelOption / config that allows to limit the number of messages to write per eventloop run.
- Respect this setting for DatagramChannels

Result:

Reduce the risk of having WRITES block the processing of other events in a timely manner

Co-authored-by: terrarier2111 <58695553+terrarier2111@users.noreply.github.com>
2021-03-17 13:09:18 +01:00
Norman Maurer
a19d9c2ab6 Ensure we also build dependent modules for deployments (#10936)
Motivation:

We need to ensure we also build the modules we depend on as otherwise we may not be able to resolve the dependencies correctly

Modifications:

- Add -am when calling maven

Result:

Deployment works all the time
2021-03-15 11:07:52 +01:00
Norman Maurer
06ddb1f45e Fix EpollSocketTestPermutation after bad cherry-pick 2021-03-15 09:27:26 +01:00
Scott Mitchell
e5ff6216ff SslHandler flushing with TCP Fast Open fix (#11077)
Motivation:
SslHandler owns the responsibility to flush non-application data
(e.g. handshake, renegotiation, etc.) to the socket. However when
TCP Fast Open is supported but the client_hello cannot be written
in the SYN the client_hello may not always be flushed. SslHandler
may not wrap/flush previously written/flushed data in the event
it was not able to be wrapped due to NEED_UNWRAP state being
encountered in wrap (e.g. peer initiated renegotiation).

Modifications:
- SslHandler to flush in channelActive() if TFO is enabled and
  the client_hello cannot be written in the SYN.
- SslHandler to wrap application data after non-application data
  wrap and handshake status is FINISHED.
- SocketSslEchoTest only flushes when writes are done, and waits
  for the handshake to complete before writing.

Result:
SslHandler flushes handshake data for TFO, and previously flushed
application data after peer initiated renegotiation finishes.
2021-03-14 14:27:10 +01:00
Chris Vest
7a70fabb07 Make EpollSocketConnectTest correctly configure TFO (#11083)
Motivation:
The EpollSocketConnectTest was not correctly configuring TCP Fast Open on the server socket.
It's an option, not a child option.

Modification:
EpollSocketConnectTest now correctly enables TCP Fast Open on the server side, when available, for the test that needs it.

Result:
Test covers what it was intended to.
2021-03-14 14:17:18 +01:00
Eric Anderson
1914679b00 codec-http2: Exception as cause, not message format argument (#11084)
Motivation:

There are several overloads of streamError(), with one receiving the
Throwable to be made the cause of the new exception. However, the wrong
overload was being called and instead the IllegalArgumentException was
being passed as a message format argument which was summarily thrown
away as the message format didn't reference it.

Modifications:

Move IllegalArgumentException to proper argument position.

Result:

A useful exception, with the underlying cause available.
2021-03-14 14:15:26 +01:00
Norman Maurer
8da6ed3fc0 Merge pull request from GHSA-wm47-8v5p-wjpj
Motivation:

As stated by https://tools.ietf.org/html/rfc7540#section-8.1.2.6 we should report a stream error if the content-length does not match the sum of all data frames.

Modifications:

- Verify that the sum of data frames match if a content-length header was send.
- Handle multiple content-length headers and also handle negative values
- Add io.netty.http2.validateContentLength system property which allows to disable the more strict validation
- Add unit tests

Result:

Correctly handle the case when the content-length header was included but not match what is send and also when content-length header is invalid
2021-03-14 14:15:22 +01:00
Norman Maurer
5e11c007f7 Also support CompositeByteBuf with SegmentedDatagramPacket (#11081)
Motivation:

c22c6b845d introduced support for
UDP_SEGMENT but did restrict it to continous buffers. This is not needed
as it is also fine to use CompositeByteBuf

Modifications:

- Allow to use CompositeByteBuf as well
- Add unit test

Result:

More flexible usage of segmented datagrams possible
2021-03-12 15:38:35 +01:00
Bennett Lynch
00eb5618df Introduce HttpMessageDecoderResult to expose decoded header size (#11068)
Motivation

The HttpObjectDecoder accepts input parameters for maxInitialLineLength
and maxHeaderSize. These are important variables since both message
components must be buffered in memory. As such, many decoders (like
Netty and others) introduce constraints. Due to their importance, many
users may wish to add instrumentation on the values of successful
decoder results, or otherwise be able to access these values to enforce
their own supplemental constraints.

While users can perhaps estimate the sizes today, they will not be
exact, due to the decoder being responsible for consuming optional
whitespace and the like.

Modifications

* Add HttpMessageDecoderResult class. This class extends DecoderResult
and is intended for HttpMessage objects successfully decoded by the
HttpObjectDecoder. It exposes attributes for the decoded
initialLineLength and headerSize.
* Modify HttpObjectDecoder to produce HttpMessageDecoderResults upon
successfully decoding the last HTTP header.
* Add corresponding tests to HttpRequestDecoderTest &
HttpResponseDecoderTest.

Co-authored-by: Bennett Lynch <Bennett-Lynch@users.noreply.github.com>
2021-03-12 13:50:51 +01:00
Norman Maurer
5de2ddeac2 Add unit tests that validate that header validation exception is propagated to the child channels (#11076)
Motivation:

We should validate the exception is actually propagted

Modifications:

Add unit tests

Result:

Verify all works as expected
2021-03-12 11:40:05 +01:00
Norman Maurer
29a4dd6ea7 Test that we return correct result for Futures that are returned from UnorderedThreadPoolExecutor (#11074)
Motivation:

Add test to validate we are not affected by the regression introduced by fd8c1874b4

Modifications:

- Add unit test

Result:

Verify code works as expected
2021-03-11 08:34:00 +01:00
Norman Maurer
2e9557c51c Try to stabilize docker build by using https (#11070)
Motivation:

At the moment we use http to download rpms, let's switch to https

Modifications:

Use https for rpms

Result:

Hopefully more stable docker image builds
2021-03-09 09:10:36 +01:00
Bennett Lynch
172472c0c0 [HttpObjectDecoder] Include hex-value of illegal whitespace char (#11067)
Motivation:

HttpObjectDecoder may throw an IllegalArgumentException if it encounters
a character that Character.isWhitespace() returns true for, but is not
one of the two valid OWS (optional whitespace) values. Such values may
not have friendly or readable toString() values. We should include the
hex value so that the illegal character can always be determined.

Modifications:

Add hex value as well

Result:

Easier to debug 

Co-authored-by: Bennett Lynch <Bennett-Lynch@users.noreply.github.com>
2021-03-09 08:13:44 +01:00
Norman Maurer
0acbe074bf
Add missing docker compose config for staging (#11066) 2021-03-08 20:51:39 +01:00
Norman Maurer
928be58f47 Always include the OS in the tcnative library name (#11064)
Motivation:

These days we always include the OS in the library name. This means we also can simplify things

Modifications:

Adjust build configuration to address for libray name change

Result:

Simplify build
2021-03-08 19:27:06 +01:00
Norman Maurer
cf2d8460bc Fix compile error introduced by 06739ed890 2021-03-08 12:17:34 +01:00
Shoothzj
06739ed890 Allow to config dns bind address in DnsNameResolver (#11061)
Motivation:

The DnsResolver default start address listen to "0.0.0.0", which may be not what the user wants. 

Modification:

Add localAddress as a param of DnsNameResolver and its builder

Result:

The DnsNameResolver's bind address can be configured.
2021-03-08 12:09:09 +01:00
Norman Maurer
0da12a6fc2
Use correct docker file for deployments (#11062) 2021-03-08 11:16:07 +01:00
Norman Maurer
8a9320161c Support session cache for client and server when using native SSLEngine implementation (#10994)
Motivation:

At the moment we don't support session caching on the client side at all when using the native SSL implementation. We should at least allow to enable it.

Modification:

Allow to enable session cache for client side but disable ti by default due a JDK bug atm.

Result:

Be able to cache sessions on the client side when using native SSL implementation .
2021-03-07 19:35:31 +01:00
Norman Maurer
d209eb0e18 Move OpenSsl*X509Certificate to util package and rename it (#10955)
Motivation:

Creating certificates from a byte[] while lazy parse it is general useful and is also needed by https://github.com/netty/netty-incubator-codec-quic/pull/141

Modifications:

Move classes, rename these and make them public

Result:

Be able to reuse code
2021-03-07 19:30:17 +01:00
Norman Maurer
b1a8de0d7a Introduce SslContextOption which can be used for "optional" features … (#10981)
Motivation:

Some of the features we want to support can only be supported by some of the SslContext implementations. We should allow to configure these in a consistent way the same way as we do it with Channel / ChannelOption

Modifications:

- Add SslContextOption and add builder methods that take these
- Add OpenSslContextOption and define two options there which are specific to openssl

Result:

More flexible configuration and implementation of SslContext
2021-03-07 19:25:22 +01:00
Chris Vest
e1830ccf47
Add support for direct ByteBufs in JdkZlibEncoder and JdkZlibDecoder (#11057)
Motivation:
The JDK deflate implementation added support for operating on ByteBuffers in Java 11 or so.
This means that we don't need to restrict that implementation to ByteBufs that are heap based and can expose arrays.

Modification:
Add clauses to JdkZlibEncoder and JdkZlibDecoder for handling ByteBufs that don't have arrays, but do have one nioByteBuffer.
Expand the test coverage in JdkZlibTest to include all relevant combinations of buffer types and data types.

Result:
The JdkZlibEncoder and JdkZlibDecoder should now work on basically all non-composite ByteBufs, and likely also composite ByteBufs that have exactly one component.
2021-03-05 13:29:20 +01:00
Chris Vest
bfea65ef52
Remove support for JZlib (#11058)
Motivation:
This library is obsolete; hasn't been updated since 2013.

Modification:
Remove jzlib dependency, integration code and tests.

Result:
- No more jzlib support.
- Less code.
- The JdkZlib* code can now be simplified because it no longer share anything with jzlib.
2021-03-04 18:20:12 +01:00
Norman Maurer
2ce03e0a08 Fix NPE that can happen in the WriteTimeoutHandler when multiple executors are used (#11056)
Motivation:

In WriteTimeoutHandler we did make the assumption that the executor which is used to schedule the timeout is the same that is backing the write promise. This may not be true which will cause concurrency issues

Modifications:

Ensure we are on the right thread when try to modify the doubly-linked-list and if not schedule it on the right thread.

Result:

Fixes https://github.com/netty/netty/issues/11053
2021-03-04 15:27:02 +01:00
Chris Vest
75120472f5
Add tests verifying channel configure/initialisation order (#11051)
Motivation:
Channels need to have their configurations applied before we can call out to user-code via handlerAdded and initChannel.

Modification:
This adds tests for this behaviour, and fixes their failures.

Result:
Channel initialisers now have access to channel configuration and attributes.
2021-03-03 11:49:40 +01:00
Norman Maurer
bcb165bccd Ensure removal from queue happens before writeAndFlush(...) is called (#11049)
Motivation:

We need to ensure that we call queue.remove() before we cal writeAndFlush() as this operation may cause an event that also touches the queue and remove from it. If we miss to do so we may see NoSuchElementExceptions.

Modifications:

- Call queue.remove() before calling writeAndFlush(...)
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/11046
2021-03-02 22:02:27 +01:00
Chris Vest
d8ae9dfea3
Bring forward build automation changes (#11052)
This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0.

Details are as follows:

* Use Github workflows for CI (#10879)

Motivation:

We should just use GitHub Actions for the CI

Modifications:

- Adjust docker / docker compose files
- Add different workflows and jobs to deploy and build the project

Result:

Don't depend on external CI services

* Fix non leak build condition

* Only use build and deploy workflows for 4.1 for now

* Add deploy job for cross compiled aarch64 (#10883)

Motivation:

We should also deploy snapshots for our cross compiled native jars.

Modifications:

- Add job and docker files for deploying cross compiled native jars
- Ensure we map the maven cache into our docker containers

Result:

Deploy aarch64 jars and re-use cache

* Use correct docker-compose file to deploy cross compiled artifacts

* Use correct docker-compose task to deploy for cross compiled artifacts

* Split pr and normal build (#10884)

Motivation:

We should better use seperate workflows for PR and normal builds

Modifications:

- Split workflows
- Better cache reuse

Result:

Cleanup

* Only deploy snapshots for one arch

Motivation:

We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems.

See https://github.com/netty/netty/issues/10887

Modification:

Skip all other deploys then x86_64

Result:

Users are able to use SNAPSHOTS for x86_6

* Use maven cachen when running analyze job (#10888)

Motivation:

To prevent failures to problems while downloading dependencies we shoud cache these

Modifications:

Add maven cache

Result:

No more failures due problems while downloading dependencies

* Also include one PR job that uses boringssl (#10886)

Motivation:

When validating PRs we should also at least run one job that uses boringssl

Modifications:

- Add job that uses boringssl
- Cleanup docker compose files
- Fix buffer leak in test

Result:

Also run with boringssl when PRs are validated

* Use matrix for job configurations (#10889)

Motivation:

We can use the matrix feature to define our jobs. This reduces a lot of config

Modification:

Use job matrix

Result:

Easier to maintain

* Correctly deploy artifacts that are build on different archs (#10893)

Motivation:

We need to take special care when deploying snapshots as we need to generate the jars in multiple steps

Modifications:

- Use the nexus staging pluging to stage jars locally in multiple steps
- Add extra job that will merge these staged jars and deploy these

Result:

Fixes https://github.com/netty/netty/issues/10887

* Dont use cron for PRs

Motivation:

It doesnt make sense to use cron for PRs

Modifications:

Remove cron config

Result:

Cleanup

* We run all combinations when validate the PR, let's just use one type for normal push

Motivation:

Let us just only use one build config when building the 4.1 branch.

Modifications:

As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches

Result:

Faster build times

* Update action-docker-layer-caching (#10900)

Motivation:

We are three releases behind.

Modifications:

Update to latest version

Result:

Use up-to-date action-docker-layer-caching version

* Verify we can load native modules and add job that verifies on aarch64 as well (#10933)

Motivation:

As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions.

Modifications:

- Add new maven module which tests loading of native modules
- Add job that will also test loading on aarch64

Result:

Less likely to introduce regressions related to loading native code in the future

* Let script fail if one command fail (#10945)

Motivation:

We should use `set -e` to ensure we fail the script if one command fails.

Modifications:

Add set -e to script

Result:

Fail fast

* Use action to report unit test errors (#10966)

Motivation:

To make it easier to understand why the build fails lets use an action that will report which unit test failed

Modifications:

- Replace custom script with action-surefire-report

Result:

Easier to understand test failures

* Use custom script to check for build failures (#10968)

Motivation:

It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script.

Modifications:

- Replace action with custom script

Result:

Builds for PRs that are done via forks work again.

* Publish test results after PR run (#11002)

Motivation:

To make it easier to understand why a build failed let us publish the rest results

Modifications:

Use a new workflow to be able to publish the test reports

Result:

Easier to understand why a PR did fail

* Fix test reports name

* Add workflow to cut releases (#11019)

Motivation:

Doing releases manually is error-prone, it would be better if we could do it via a workflow

Modification:

- Add workflow to cut releases
- Add related scripts

Result:

Be able to easily cut a release via a workflow

* Update build for master branch

Motivation:
The build changes were brought forward from 4.1, and contain many things specific to 4.1.

Modification:
Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master".

Result:
Builds should now work for the master branch.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
Norman Maurer
0527ce16c1 Fix compile error introduced by bad cherry-pick 2021-02-28 09:54:00 +01:00
Norman Maurer
ea2a8799f2 Add support for UDP_SEGMENT (GSO) when using sendmmsg (#11038)
Motivation:

For protocols like QUIC using UDP_SEGMENT (GSO) can help to reduce the
overhead quite a bit. We should support it.

Modifications:

- Add a SegmentedDatagramPacket which can be used to use UDP_SEGMENT
- Add unit test

Result:

Be able to make use of UDP_SEGMENT
2021-02-27 21:58:21 +01:00
Chris Vest
ec18aa8731
Introduce ByteBufConvertible interface (#11036)
Motivation:
To make it possible to experiment with alternative buffer implementations, we need a way to abstract away the concrete buffers used throughout most of the Netty pipelines, while still having a common currency for doing IO in the end.

Modification:
- Introduce an ByteBufConvertible interface, that allow arbitrary objects to convert themselves into ByteBuf objects.
- Every place in the code, where we did an instanceof check for ByteBuf, we now do an instanceof check for ByteBufConvertible.
- ByteBuf itself implements ByteBufConvertible, and returns itself from the asByteBuf method.

Result:
It is now possible to use Netty with alternative buffer implementations, as long as they can be converted to ByteBuf.
This has been verified elsewhere, with an alternative buffer implementation.
2021-02-26 15:03:58 +01:00
Frédéric Brégier
d421ae10d7 Minimize get byte multipart and fix buffer reuse (#11001)
Motivation:
- Underlying buffer usages might be erroneous when releasing them internaly
in HttpPostMultipartRequestDecoder.

2 bugs occurs:
1) Final File upload seems not to be of the right size.
2) Memory, even in Disk mode, is increasing continuously, while it shouldn't.

- Method `getByte(position)` is too often called within the current implementation
of the HttpPostMultipartRequestDecoder.
This implies too much activities which is visible when PARANOID mode is active.
This is also true in standard mode.

Apply the same fix on buffer from HttpPostMultipartRequestDecoder to HttpPostStandardRequestDecoder
made previously.

Finally in order to ensure we do not rewrite already decoded HttpData when decoding
next ones within multipart, we must ensure the buffers are copied and not a retained slice.

Modification:
- Add some tests to check consistency for HttpPostMultipartRequestDecoder.
Add a package protected method for testing purpose only.

- Use the `bytesBefore(...)` method instead of `getByte(pos)` in order to limit the external
access to the underlying buffer by retrieving iteratively the beginning of a correct start
position.
It is used to find both LF/CRLF and delimiter.
2 methods in HttpPostBodyUtil were created for that.

The undecodedChunk is copied when adding a chunk to a DataMultipart is loaded.
The same buffer is also rewritten in order to release the copied memory part.

Result:

Just for note, for both Memory or Disk or Mixed mode factories, the release has to be done as:

      for (InterfaceHttpData httpData: decoder.getBodyHttpDatas()) {
          httpData.release();
          factory.removeHttpDataFromClean(request, httpData);
      }
      factory.cleanAllHttpData();
      decoder.destroy();

The memory used is minimal in Disk or Mixed mode. In Memory mode, a big file is still
in memory but not more in the undecodedChunk but its own buffer (copied).

In terms of benchmarking, the results are:

Original code Benchmark                                                             Mode  Cnt  Score    Error   Units
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigAdvancedLevel   thrpt    6  0,152 ±  0,100  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigDisabledLevel   thrpt    6  0,543 ±  0,218  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigParanoidLevel   thrpt    6  0,001 ±  0,001  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigSimpleLevel     thrpt    6  0,615 ±  0,070  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighAdvancedLevel  thrpt    6  0,114 ±  0,063  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighDisabledLevel  thrpt    6  0,664 ±  0,034  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighParanoidLevel  thrpt    6  0,001 ±  0,001  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighSimpleLevel    thrpt    6  0,620 ±  0,140  ops/ms

New code Benchmark                                                                  Mode  Cnt  Score   Error   Units
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigAdvancedLevel   thrpt    6  4,037 ± 0,358  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigDisabledLevel   thrpt    6  4,226 ± 0,471  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigParanoidLevel   thrpt    6  0,875 ± 0,029  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigSimpleLevel     thrpt    6  4,346 ± 0,275  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighAdvancedLevel  thrpt    6  2,044 ± 0,020  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighDisabledLevel  thrpt    6  2,278 ± 0,159  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighParanoidLevel  thrpt    6  0,174 ± 0,004  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighSimpleLevel    thrpt    6  2,370 ± 0,065  ops/ms

In short, using big file transfers, this is about 7 times faster with new code, while
using high number of HttpData, this is about 4 times faster with new code when using Simple Level.
When using Paranoid Level, using big file transfers, this is about 800 times faster with new code, while
using high number of HttpData, this is about 170 times faster with new code.
2021-02-26 14:26:24 +01:00