Commit Graph

9642 Commits

Author SHA1 Message Date
Norman Maurer
208a258d0e more WIP 2019-12-04 09:31:45 +01:00
Norman Maurer
7260f55922 WIP 2019-11-29 11:38:50 +01:00
Norman Maurer
0bc7b48deb WIP 2019-11-29 08:59:47 +01:00
Norman Maurer
ee593ace33 Revert "Protect ChannelHandler from reentrancee issues (#9358)"
This reverts commit 48634f1466.
2019-11-28 15:30:38 +01:00
Martin Furmanski
585ed4d08f Improve error handling in ByteToMessageDecoder when expand fails (#9822)
Motivation:

The buffer which the decoder allocates for the expansion can be
leaked if there is a subsequent issue writing to it.

Modifications:
The error handling has been improved so that the new buffer always
is released on failure in the expand.

Result:
The decoder will not leak in this scenario any more.

Fixes: https://github.com/netty/netty/issues/9812
2019-11-28 12:29:05 +01:00
Norman Maurer
0a252b48b1 Fix compile error in test caused by 2d48ec4e3f 2019-11-28 11:31:21 +01:00
Norman Maurer
8cfd71c354 Call ctx.flush() when onStreamClosed(...) produces a window update frame (#9818)
Motivation:

We use the onStreamClosed(...) callback to return unconsumed bytes back to the window of the connection when needed. When this happens we will write a window update frame but not automatically call ctx.flush(). As the user has no insight into this it could in the worst case result in a "deadlock" as the frame is never written out ot the socket.

Modifications:

- If onStreamClosed(...) produces a window update frame call ctx.flush()
- Add unit test

Result:

No stales possible due unflushed window update frames produced by onStreamClosed(...) when not all bytes were consumed before the stream was closed
2019-11-28 11:11:49 +01:00
Norman Maurer
16c88fd9ed Use EmbeddedChannel.finishAndReleaseAll() to remove boiler-plate code (#9824)
Motivation:

We can make use of EmbeddedChannel.finishAndReleaseAll() and so remove some code

Modifications:

Use finishAndReleaseAll()

Result:

Less code to maintain
2019-11-28 11:04:09 +01:00
Norman Maurer
371c431b6b Don't send window update frame for unconsumed bytes when stream is already closed (#9816)
Motivation:

At the moment we send a window update frame for the connection + stream when a stream is closed and there are unconsumed bytes left. While we need to do this for the connection it makes no sense to write a window update frame for the stream itself as it is already closed

Modifications:

- Don't write the window update frame for the stream when the stream is closed
- Add unit test

Result:

Don't write the window frame for closed streams
2019-11-28 09:06:44 +01:00
Norman Maurer
2d48ec4e3f Correctly set writerIndex when EpollChannelOption.MAX_DATAGRAM_PAYLOAD_SIZE is used in all cases (#9819)
Motivation:

Due a bug we did not correctly set the writerIndex of the ByteBuf when a
user specified EpollChannelOption.MAX_DATAGRAM_PAYLOAD_SIZE but we ended
up with a non scattering read.

Modifications:

- Set writerIndex to the correct value
- Add unit tests

Result:

Fixes https://github.com/netty/netty/issues/9788
2019-11-28 09:06:13 +01:00
Norman Maurer
aa3c800f12 Use latest maven release (#9820)
Motivation:

Apache Maven 3.6.3 was released

Modifications:

Update to latest version

Result:

Use latest version to build
2019-11-27 14:45:48 +01:00
Norman Maurer
b7ba807b30 Fix compile error introduced by 2c3d263e23 2019-11-27 09:08:20 +01:00
ursa
f48d9fa8d0 Bugfix #9673: Origin header is always sent from WebSocket client (#9692)
Those who need 'Origin' or 'Sec-WebSocket-Origin' headers should provide them explicitly, like it is stated in WebSocket specs.

E.g. through custom headers:

    HttpHeaders customHeaders = new DefaultHttpHeaders()
        .add(HttpHeaderNames.ORIGIN, "http://localhost:8080");
    new WebSocketClientProtocolHandler(
        new URI("ws://localhost:1234/test"), WebSocketVersion.V13, subprotocol,
        allowExtensions, customHeaders, maxFramePayloadLength, handshakeTimeoutMillis)

* Remove enforced origin headers.
* Update tests

Fixes #9673: Origin header is always sent from WebSocket client
2019-11-27 08:40:57 +01:00
Norman Maurer
f2596fd993 Simplify Deflate* implementations by using EmbeddedChannel.finishAndReleaseAll() (#9808)
Motivation:

We can simplify the code by just using finishAndReleaseAll()

Modifications:

Remove some code and simplify

Result:

Cleaner code
2019-11-27 06:54:52 +01:00
Norman Maurer
621af5ce27 Correctly guard against multiple RST frames for the same stream (#9811)
Motivation:

Http2ConnectionHandler tries to guard against sending multiple RST frames for the same stream. Unfortunally the code is not 100 % correct as it only updates the state after it calls write. This may lead to the situation of have an extra RST frame slip through if the second write for the RST frame is done from a listener that is attached to the promise.

Modifications:

- Update state before calling write
- Add unit test

Result:

Only ever send one RST frame per stream
2019-11-27 06:54:17 +01:00
Greg Lewis
154e40bcea Fix the transport-native-unix-common build on FreeBSD (#9814)
Motivation:

Modern versions of FreeBSD define IP_RECVORIGDSTADDR, but don't define
SOL_IP.  This causes the build to fail.

Modifications:

The equivalent to SOL_IP on FreeBSD is IPPROTO_IP.  Define SOL_IP as that
if SOL_IP is not defined and IPPROTO_IP is.

Result:

This allows a successful build on FreeBSD
2019-11-27 06:52:58 +01:00
ZhenLian
2c3d263e23 Support Passing KeyManager and TrustManager into SslContextBuilder (#9805) (#9786)
Motivation:

This is a PR to solve the problem described here: https://github.com/netty/netty/issues/9767
Basically this PR is to add two more APIs in SslContextBuilder, for users to directly specify
the KeyManager or TrustManager they want to use when building SslContext. This is very helpful
when users want to pass in some customized implementation of KeyManager or TrustManager.

Modification:

This PR takes the first approach in here:
https://github.com/netty/netty/issues/9767#issuecomment-551927994 (comment)
which is to immediately convert the managers into factories and let factories continue to pass
through Netty.

1. Add in SslContextBuilder the two APIs mentioned above
2. Create a KeyManagerFactoryWrapper and a TrustManagerFactoryWrapper, which take a KeyManager
and a TrustManager respectively. These are two simple wrappers that do the conversion from
XXXManager class to XXXManagerFactory class
3.Create a SimpleKeyManagerFactory class(and internally X509KeyManagerWrapper for compatibility),
which hides the unnecessary details such as KeyManagerFactorySpi. This serves the similar
functionalities with SimpleTrustManagerFactory, which was already inside Netty.

Result:

Easier usage.
2019-11-26 14:26:04 +01:00
Norman Maurer
3fcd37e07e Correctly only active not_x86_64 profile when using a non x86_64 platform (#9805)
Motivation:

21720e4a78 introduced a change which aimed to enable the not_x86_64 profile when building on a x86_64 platform. Unfortunaly it made an assemption which not holds true and so the profile was already enabled. This lead to the situation that native SSL tests were skipped if non boringssl impl was used.

Modifications:

Fix profile activation to work as expected

Result:

Correctly run aal native SSL tests
2019-11-25 14:56:30 +01:00
Bryce Anderson
3ebc946c35 Log expected STREAM_CLOSED exceptions for already closed streams at DEBUG level (#9798)
Motivation:

There is an intrinsic race between a local session resetting a stream
and the peer no longer sending any frames. This can result in the
session receiving frames for a stream that the local peer no longer
tracks. This results in a StreamException being thrown which triggers a
RST_STREAM frame, which is a good thing, but also logging at level WARN,
which is noisy for an expected and benign condition.

Modification:

Change the log level to DEBUG when logging stream errors with code
STREAM_CLOSED. All others are more interesting and will continue to be
logged at level WARN.

Additionally, it was found that DATA frames for streams that could not
have existed only resulted in a StreamException when the spec is clear
that such a situation should be fatal to the connection, resulting in a
GOAWAY(PROTOCOL_ERROR).

Fixes #8025.
2019-11-25 09:02:55 +01:00
Norman Maurer
3add35fe82 Remove dependency on GLIBC 2.12 by using syscalls directly (#9797)
Motivation:

394a1b3485 introduced a hard dependency on GLIBC 2.12 which was not the case before. This had the effect of not be able to use the native epoll transports on platforms which ship with earlier versions of GLIBC.
To make things a backward compatible as possible we should not introduce such changes in a bugfix release.

Special thanks to @weissi with all the help to fix this.

Modifications:

- Use syscalls directly to remove dependency on GLIBC 2.12
- Make code consistent that needs newer GLIBC versions
- Adjust scattering read test to only run if recvmmsg syscall is supported
- Cleanup pom.xml as some stuff is not needed anymore after using syscalls.

Result:

Fixes https://github.com/netty/netty/issues/9758.
2019-11-23 21:17:39 +01:00
Norman Maurer
b7b6156505
Remove atomic usage in DefaultChannelHandlerContext as all pipeline operations are on the EventLoop (#9794)
Motivation:

In netty 5.x we changed to have all pipeline operations be executed on the EventLoop so there is no need to have an atomic operation involved anymore to update the handler state.

Modifications:

Remove atomic usage to handle the handler state

Result:

Simpler code and less overhead
2019-11-22 10:12:21 +01:00
Norman Maurer
e8d72fda5f
Fix tests failures introduced by bad cherry-pick of 7ff8cde66f (#9795)
Motivation:

7ff8cde66f introduced some tests which needs some small adjustments for master.

Modifications:

Add explicit casts

Result:

master builds again without test failures
2019-11-22 08:54:25 +01:00
Nick Hill
d370d48d4a Update to latest JMH version (#9787)
Motivation

JMH 1.22 was released recently, we might as well use the latest when
running benchmarks.

Summary of changes:
https://mail.openjdk.java.net/pipermail/jmh-dev/2019-November/002879.html

Modifications

Update jmh dependencies in microbench module from version 1.21 to 1.22.

Result

Benchmarks run using latest JMH
2019-11-19 11:28:36 +01:00
ursa
7ff8cde66f Send close frame on channel close, when this frame was not send manually (#9745)
Motivation:
By default CloseWebSocketFrames are handled automatically.
However I need manually manage their sending both on client- and on server-sides.

Modification:
Send close frame on channel close automatically, when it was not send before explicitly.

Result:
No more messages like "Connection closed by remote peer" for normal close flows.
2019-11-18 20:32:45 +01:00
Norman Maurer
3bb75198cd Update to latest recommended maven version (#9785)
Motivation:

Latest recommended maven version is 3.6.2 so we should use it

Modifications:

Update from 3.5.2 to 3.6.2

Result:

Use latest recommended maven version
2019-11-18 11:04:02 +01:00
Norman Maurer
dde9211702 Add one new constructor with ThreadFactory / Executor only
Motivation:
In most cases, we want to use MultithreadEventLoopGroup without setting thread numbers but thread name only. We should simplify this for the user

Modifications:
Add a new constructor

Result:
User can only set ThreadFactory / Executor without setting the thread number to 0:
2019-11-18 10:01:19 +01:00
Codrut Stancu
ae1bf5fbdd Enable generation of test JARs (#9774)
Motivation:

Netty currently doesn't build and distribute the test JARs. Having easy access to the test JARs would enable downstream projects (such as GraalVM) to integrate the Netty unit tests in their CI pipeline to ensure continous compatibility with Netty features. The alternative would be to build Netty from source every time to obtain the test jars, however, depending on the CI setup, that may not always be possible.
 
Modifications:

Modify `pom.xml` to enable generation of test JARs and corresponding source JARs.
 
Result:

Running the Maven build will create the test JARs and corresponding source JARs. This change was tested locally via `mvn install` and the test JARs are correctly copied under the Maven cache. The expectation is that running `mvn deploy` will also copy the additional JARs to the maven repository.
2019-11-18 09:42:01 +01:00
stroller
2e449a6769 Improve java doc for MINIMAL_WAIT (#9779)
Motivation:

MINIMAL_WAIT is the key constant. Thus, When we see the constant, we must read more code logic to see if it is ms or ns. So improving java doc will be better.

Modifications:
Improve java doc by add "10ms" such as DEFAULT_CHECK_INTERVAL with "1s".

Result:

Easy to know it is ms and keep same java doc style with other constants such as DEFAULT_CHECK_INTERVAL.
2019-11-17 17:42:10 +01:00
Nick Hill
c6bdc2b7dc Reduce ByteBuffer duplication when resizing pooled direct ByteBufs (#9765)
Motivation:

Currently when use of Unsafe is disabled and an internal reallocation is
performed for a direct PooledByteBuf, a one-off temporary duplicate is
made of the source and destination backing nio buffers so that the
copy can be done in a threadsafe manner.

The need for this can be reduced by sharing the temporary duplicate
buffer that is already stored in the corresponding destination
PooledByteBuf instance.

Modifications:

Have PoolArena#memoryCopy(...) take the destination PooledByteBuf
instead of the underlying mem reference and offset, and use
internalNioBuffer() to obtain/initialize a reusable duplicate of the
backing nio buffer.

Result:

Fewer temporary allocations when resizing direct pooled ByteBufs in the
non-Unsafe case
2019-11-16 11:27:03 -08:00
Ameya Lokare
11255c6602 Add constants for fixed-header only MQTT messages (#9749)
Motivation:

Currently, the only way to create fixed-header only messages PINGREQ,
PINGRESP and DISCONNECT is to explicitly instantiate a `MqttFixedHeader` like:
```
MqttFixedHeader disconnectFixedHeader = new MqttFixedHeader(MqttMessageType.DISCONNECT,
    false, MqttQoS.AT_MOST_ONCE, false, 0);
MqttMessage disconnectMessage = new MqttMessage(disconnectFixedHeader);
```

According to the MQTT spec
(http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718077),
the fixed-header flags for these messages are reserved and  must be set to zero, otherwise
the receiver must close the connection. It's easy to mess this up when
you're creating the header explicitly, for e.g by setting the QoS bit to
`AT_LEAST_ONCE`.

As such, provide static constants for PINGREQ, PINGRESP and
DISCONNECT messages that will set the flags correctly for the developer.

Modification:

Add static constants to MqttMessage class to construct PINGREQ, PINGRESP and
DISCONNECT messages that will set the fixed-header flags correctly to 0.

Result:

Easier usage.
2019-11-08 10:17:16 +01:00
monkey-mas
fc60fa4426 Add test to check Connection-Specific headers are removed in HTTP/2 (by HttpConversionUtil.toHttp2Headers) (#9766)
Motivation:
To avoid regression regarding connection-specific headers[1], we should add a test.
[1] https://tools.ietf.org/html/rfc7540#section-8.1.2.2

Modification:
Add test that checks the following headers are removed.
- Connection
- Host
- Keep-Alive
- Proxy-Connection
- Transfer-Encoding
- Upgrade

Result:
There's no functional change.
2019-11-08 10:15:19 +01:00
Norman Maurer
cca886d26c Update to latest netty-tcnative release (#9763)
Motivation:

There is a new netty-tcnative release. We should use it.

Modifications:

Update to 2.0.27

Result:

Use latest netty-tcnative release
2019-11-07 20:31:26 +01:00
Norman Maurer
f8b05b1c84 Don't cache key material if sun.security.ssl.X509KeyManagerImpl is used (#9762)
Motivation:

sun.security.ssl.X509KeyManagerImpl will not use "stable" aliases and so aliases may be changed during invocations. This means caching is useless. Because of this we should disable the cache if its used.

Modifications:

- Disable caching if sun.security.ssl.X509KeyManagerImpl is used
- Add tests

Result:

More protection against https://github.com/netty/netty/issues/9747.
2019-11-07 15:28:00 +01:00
Norman Maurer
38dd3b6bd1 At the moment the cache provided by OpenSslCachingKeyMaterialProvider… (#9759)
Motivation:

At the moment te cache is not bound and so lead to huge memory consumpation. We should ensure its bound by default.

Modifications:

Ensure cache is bound

Result:

Fixes https://github.com/netty/netty/issues/9747.
2019-11-07 15:27:42 +01:00
Andrey Mizurov
af532d2b7e Fix encoding/decoding for UTF-8 stomp commands and headers (#9740)
Motivation:

According STOMP spec (https://stomp.github.io/stomp-specification-1.2.html#Value_Encoding) we have to encode and decode commands and headers to UTF-8

Modification:

Provide ability for StompSubframeDecoder and StompSubframeEncoder work with UTF-8
2019-11-06 12:07:58 +01:00
Nick Hill
625981a296 Introduce ByteBuf#isContiguous() method (#9735)
Motivation

There's currently no way to determine whether an arbitrary ByteBuf
behaves internally like a "singluar" buffer or a composite one, and this
can be important to know when making decisions about how to manipulate
it in an efficient way.

An example of this is the ByteBuf#discardReadBytes() method which
increases the writable bytes for a contiguous buffer (by readerIndex)
but does not for a composite one.

Unfortunately !(buf instanceof CompositeByteBuf) is not reliable, since
for example this will be true in the case of a sliced CompositeByteBuf
or some third-party composite implementation.

isContiguous was chosen over isComposite since we want to assume "not
contiguous" in the unknown/default case - the doc will it clear that
false does not imply composite.

Modifications

- Add ByteBuf#isContiguous() which returns true by default
- Override the "concrete" ByteBuf impls to return true and ensure
wrapped/derived impls delegate it appropriately
- Include some basic unit tests

Result

Better assumptions/decisions possible when manipulating arbitrary
ByteBufs, for example when combining/cumulating them.
2019-11-06 12:07:00 +01:00
时无两丶
d13ac5290a Clean up expired docs. (#9756)
Motivation:
Since the `extractFrame` has used `retainedSlice` to avoid memory copy, we should clean this doc that was expired.

Result:
Better doc.
2019-11-06 09:43:51 +01:00
zhangheng
86a533576f Remove padding when writing CONTINUATION frame (#9752)
Motivation:

Padding was removed from CONTINUATION frame in http2-spec, as showed in [PR](https://github.com/http2/http2-spec/pull/510). We should follow it.

Modifications:

- Remove padding when writing CONTINUATION frame in DefaultHttp2FrameWriter
- Add a unit test for writing large header with padding

Result:

More spec-compliant
2019-11-05 15:21:10 +01:00
Nick Hill
d3011d4f5b Use interval instead of mask comparison for Recycler ratio (#9748)
Motivation

The recycling ratio is currently implemented by comparing with a masked
count. The mask operation is not free and also not necessary.

Modification

Change the count(s) to just iterate over the corresponding interval,
which requires only a comparison and no mask.

Also make "first time recycle" behaviour consistent and revert change to
RecyclerTest made in #9727.

Result

Less recycling overhead
2019-11-05 15:21:09 +01:00
Norman Maurer
f0e1628426 Cleanup Recycler to better encapsulate stuff (#9739)
Motivation:

We can move some methods etc to make encapsulation better in Recycler

Modifications:

Move / rename methods to make usage more clear

Result:

Code cleanup
2019-11-04 17:44:12 +01:00
monkey-mas
83b35db2e5 Remove unnecessary line in Http2ClientUpgradeCodec (#9750)
Motivation:
To clean up code.

Modification:
Remove unnecessary line.

Result:
There's no functional change.
2019-11-04 11:20:41 +01:00
Nick Hill
7df012884f Rename SimpleChannelInboundHandler.channelRead0() to messageReceived() (#8819)
Motivation

Per javadoc in 4.1.x SimpleChannelInboundHandler:

"Please keep in mind that channelRead0(ChannelHandlerContext, I) will be
renamed to messageReceived(ChannelHandlerContext, I) in 5.0."

Modifications

Rename aforementioned method and all references/overrides.

Result

Method is renamed.
2019-11-01 07:23:07 +01:00
Norman Maurer
e77b9104d7 Enforce ratioMask also for WeakOrderQueue (#9727)
Motivation:

At the moment we only enfore ratioMask for the Stack which means that we only guard against recycle burts when recycled from the same Thread. We should also enforce the ratioMask in the WeakOrderQueue so we also guard against the bursts when recycle from other threads.

Modifications:

- Keep counter in WeakOrderQueue to enforce ratioMask as well
- Adjust unit test

Result:

Better guard against recycle bursts which could pollute the heap unnecessary.
2019-11-01 07:10:57 +01:00
Sergei Egorov
6591a6501c Test that NettyBlockHoundIntegration can be loaded with ServiceLoader (#9743)
Motivation:

If something is mis-configured, the "main" test will fail but it is unclear
whether it fails because the integration does not work or it wasn't applied
at all.
Also see:
https://github.com/netty/netty/issues/9738#issuecomment-548416693

Modifications:

This change adds a test that uses the same mechanism as BlockHound does
(`ServiceLoader`) and checks that `NettyBlockHoundIntegration` is present.

Result:

It is now clear whether the integration is not working or it wasn't loaded at all.
2019-11-01 07:07:59 +01:00
Sergei Egorov
17cff0ee77 Make BlockHound tests run on Java 13 (#9742)
Motivation:
Java 13 requires special flags to be set to make BlockHound work

Modifications:
- Added jdk13 profile to `transport-blockhound-tests`
- Enabled `-XX:+AllowRedefinitionToAddDeleteMethods` on jdk13

Result:
The tests work on Java 13
2019-11-01 07:05:09 +01:00
Norman Maurer
ed52dac8cc Upgrade various JDK flavors / version in our docker-compose files (#9737)
Motivation:

We should always test with the latest JDK versions on our CI.

Modifications:

Update versions

Result:

Use latest JDK versions on our CI
2019-10-31 12:16:50 +01:00
Nick Hill
7c543a48e5 Avoid synthetic methods in Recycler (#9736)
Motivation

Currently the visibility of the various Recycler inner classes and their
fields isn't optimal. Some private members are accessed by other classes
resulting in synthetic methods, and other non-private classes/members
are only accessed privately and so can be made private.

Modifications

- Increase/reduce visibility of various fields/methods/classes within
Recycler
- Have WeakOrderQueue extend WeakReference<Thread> to eliminate the
owner field
- Change local DefaultHandle var to DefaultHandle<?> to avoid raw type
compiler warning

Result

Tidier code, fewer implicit methods on hot paths (reducing inlining
depths)
2019-10-31 11:28:11 +01:00
Norman Maurer
3501f82ef2 Update to latest jdk8 release (#9717)
Motivation:

We should use latest jdk8 release to build on CI

Modifications:

Update to latest adoptjdk release

Result:

Use latest jdk8
2019-10-31 10:19:03 +01:00
Norman Maurer
e816018569 Remove usage of finalizer in Recycler (#9726)
Motivation:

We currently use a finalizer to ensure we correctly return the reserved back to the Stack but this is not really needed as we can ensure we return it when needed before dropping the WeakOrderQueue

Modifications:

Use explicit method call to ensure we return the reserved space back before dropping the object

Result:

Less finalizer usage and so less work for the GC
2019-10-31 09:58:43 +01:00
Norman Maurer
29761a91bd Correctly update size of the Stack before doing any validation in Recycler (#9731)
Motivation:

We null out the element in the array after we decrement the current size of the Stack but not directly write back the updated size to the stored field. This is problematic as we do some validation before we write it back and so may never do so if the validation fails. This then later can lead to have null objects returned where not expected

Modifications:

Update size directly after null out object

Result:

No more unexpected null value possible
2019-10-31 09:52:28 +01:00