Commit Graph

9184 Commits

Author SHA1 Message Date
Norman Maurer
83ab4ef5e3
Explict always call ctx.read() when AUTO_READ is false and HTTP/2 is used. (#8647)
Motivation:

We should always call ctx.read() even when AUTO_READ is false as flow-control is enforced by the HTTP/2 protocol.

See also https://tools.ietf.org/html/rfc7540#section-5.2.2.

We already did this before but not explicit and only did so because of some implementation details of ByteToMessageDecoder. It's better to be explicit here to not risk of breakage later on.

Modifications:

- Ensure we always call ctx.read() when AUTO_READ is false
- Add unit test.

Result:

No risk of staling the connection when HTTP/2 is used.
2018-12-13 19:02:20 +01:00
Feri73
d17bd5e160 Adding support for whitespace in resource path in tests (#8606)
Motivation:

In windows if the project is in a path that contains whitespace,
resources cannot be accessed and tests fail.

Modifications:

Adds ResourcesUtil.java in netty-common. Tests use ResourcesUtil.java to access a resource.

Result:

Being able to build netty in a path containing whitespace
2018-12-12 10:29:02 +01:00
Norman Maurer
1dacd37989
SSLSession.putValue / getValue / removeValue / getValueNames must be thread-safe. (#8648)
Motivation:

SSLSession.putValue / getValue / removeValue / getValueNames must be thread-safe as it may be called from multiple threads. This is also the case in the OpenJDK implementation.

Modifications:

Guard with synchronized (this) blocks to keep the memory overhead low as we do not expect to have these called frequently.

Result:

SSLSession implementation is thread-safe.
2018-12-12 07:41:26 +01:00
Paul Verest
25216be118 ReadTimeoutHandler - missing ) within JavaDoc example (#8645)
Motivation:

improve docs

Modification:

ReadTimeoutHandler - missing ) within JavaDoc example

No logic/unit tests affected
2018-12-10 20:50:38 +01:00
Norman Maurer
bdcad8ef47
Fix incorrect assert in Http2MultiplexCodec caused by 9f9aa1a. (#8639)
Motivation:

9f9aa1a did some changes related to fixing how we handle ctx.read() in child channel but did incorrectly change some assert.

Modifications:

Fix assert to be correct.

Result:

Code does not throw an AssertionError due incorrect assert check.
2018-12-07 21:00:47 +01:00
Norman Maurer
36c12a4c55
Fix typo in MessageToMessageDecoder api docs. (#8638)
Motivation:

We had some typo (most likely caused by copy-and-paste) in the api docs which should be fixed.

Modifications:

Replace encoder by decoder word.

Result:

Correct apidocs.
2018-12-07 20:45:26 +01:00
Norman Maurer
a564b70d51
More correct fix for using ChannelInitializer with custom EventExecutor. (#8633)
Motivation:

8331248671 did make some changes to fix a race in ChannelInitializer when using with a custom EventExecutor. Unfortunally these where a bit racy and so the testcase failed sometimes.

Modifications:

- More correct fix when using a custom EventExecutor
- Adjust the testcase to be more correct.

Result:

Proper fix for https://github.com/netty/netty/issues/8616.
2018-12-07 19:12:06 +01:00
多巴胺
22b2c4c3b8 Fix concurrency problem in UniqueIpFilter (#8635)
Motivation:

If two requests from the same IP are reached at the same time, `connected.contains(remoteIp)` may return false in both threads.

Modifications:

Check if there is already a connection with the same IP using return values.

Result:

Become thread safe.
2018-12-07 13:50:00 +01:00
tomer doron
2b651eb1a2 support publishing snapshots from docker based ci (#8634)
motivation: automate snapshot publishing from docker based ci

changes:
* add local settings.xml with env variables for publishing to sonatype-nexus-snapshots
* pipe UID/PWD env variable in docker compose
2018-12-07 05:43:06 +01:00
Norman Maurer
51a650979f
Skip test on windows as the semantics we expect are only true on Linux / Unix / BSD / MacOS (#8629)
Motivation:

In the test we assume some semantics on how RST is done that are not true for Windows so we should skip it.

Modifications:

Skip test when on windows.

Result:

Be able to run testsuite on windows. Fixes https://github.com/netty/netty/issues/8571.
2018-12-06 20:43:40 +01:00
Feri73
5df235c083 Correcting Maven Dependencies (#8622)
Motivation:

Most of the maven modules do not explicitly declare their
dependencies and rely on transitivity, which is not always correct.

Modifications:

For all maven modules, add all of their dependencies to pom.xml

Result:

All of the (essentially non-transitive) depepdencies of the modules are explicitly declared in pom.xml
2018-12-06 09:01:14 +01:00
Norman Maurer
8331248671
ChannelInitializer may be invoked multiple times when used with custom EventExecutor. (#8620)
Motivation:

The ChannelInitializer may be invoked multipled times when used with a custom EventExecutor as removal operation may be done asynchronously. We need to guard against this.

Modifications:

- Change Map to Set which is more correct in terms of how we use it.
- Ensure we only modify the internal Set when the handler was removed yet
- Add unit test.

Result:

Fixes https://github.com/netty/netty/issues/8616.
2018-12-05 19:30:17 +01:00
Norman Maurer
6739755d39
NioEventLoop.register(...) should offload to the EventLoop if not alr… (#8612)
Motivation:

java.nio.channels.spi.AbstractSelectableChannel.register(...) need to obtain multiple locks during execution which may produce a long wait time if we currently select. This lead to multiple CI failures in the past.

Modifications:

Ensure the register call takes place on the EventLoop.

Result:

No more flacky CI test timeouts.
2018-12-05 15:31:21 +01:00
Norman Maurer
9f9aa1ae01
Respect ctx.read() calls while processing reads for the child channels when using the Http2MultiplexCodec. (#8617)
Motivation:

We did not correct respect ctx.read() calls while processing a read for a child Channel. This could lead to read stales when auto read is disabled and no other read was requested.

Modifications:

- Keep track of extra read() calls while processing reads
- Add unit tests that verify that read() is respected when triggered either in channelRead(...) or channelReadComplete(...)

Result:

Fixes https://github.com/netty/netty/issues/8209.
2018-12-05 15:29:33 +01:00
Nick Travers
d0d30f1fbe Loosen bounds check on CompositeByteBuf's maxNumComponents (#8621)
Motivation:

In versions of Netty prior to 4.1.31.Final, a CompositeByteBuf could be
created with any size (including potentially nonsensical negative
values). This behavior changed in e7737b993, which introduced a bounds
check to only allow for a component size greater than one. This broke
some existing use cases that attempted to create a byte buf with a
single component.

Modifications:

Lower the bounds check on numComponents to include the single component
case, but still throw an exception for anything less than one.

Add unit tests for the case of numComponents being less than, equal to,
and greater than this lower bound.

Result:

Return to the behavior of 4.1.30.Final, allowing one component, but
still include an explicit check against a lower bound.

Note that while creating a CompositeByteBuf with a single component is
in some ways a contradiction of the term "composite", this patch caters
for existing uses while excluding the clearly nonsensical case of asking
for a CompositeByteBuf with zero or fewer components.

Fixes #8613.
2018-12-05 08:42:23 +01:00
Francesco Nigro
b8a3394f9b Adding an execute burst cost benchmark for Netty executors (#8594)
Motivation:

Netty executors doesn't have yet any means to compare with each others
nor to compare with the j.u.c. executors

Modifications:

A new benchmark measuring execute burst cost is being added

Result:

It's now possible to compare some of Netty executors with each others
and with the j.u.c. executors
2018-12-04 15:46:25 +01:00
Norman Maurer
2680357423
Provide a way to cache the internal nioBuffer of the PooledByteBuffer… (#8603)
Motivation:

Often a temporary ByteBuffer is used which can be cached to reduce the GC pressure.

Modifications:

Cache the ByteBuffer in the PoolThreadCache as well.

Result:

Less GC.
2018-12-04 15:26:05 +01:00
Norman Maurer
dcbd7c492b
Update to OpenJDK 12 ea22 (#8618)
Motivation:

We should use the latest OpenJDK 12 release when running tests against Java12.

Modifications:

- Update to OpenJDK 12 ea22.
- Update pax exam version
- skip OSGI testsuite on Java12 as it does not work ea22 yet.

Result:

Use latest OpenJDK 12 version when running on the CI.
2018-12-04 11:59:10 +01:00
Julien Hoarau
d05666ae2d Set-Cookie headers should not be combined (#8611)
Motivation:

According to the HTTP spec set-cookie headers should not be combined
because they are not using the list syntax.

Modifications:

Do not combine set-cookie headers.

Result:

Set-Cookie headers won't be combined anymore
2018-12-01 10:47:18 +01:00
Nick Hill
a0c3081d82 Reduce http2 buffer slicing (#8598)
Motivation

DefaultHttp2FrameReader currently does a fair amount of "intermediate"
slicing which can be avoided.

Modifications

Avoid slicing the input buffer in DefaultHttp2FrameReader until
necessary. In one instance this also means retainedSlice can be used
instead (which may also avoid allocating).

Results

Less allocations when using http2.
2018-11-29 19:45:52 +01:00
root
8eb313072e [maven-release-plugin] prepare for next development iteration 2018-11-29 11:15:09 +00:00
root
afcb4a37d3 [maven-release-plugin] prepare release netty-4.1.32.Final 2018-11-29 11:14:20 +00:00
Nick Hill
fedf3ccecb Harden ref-counting concurrency semantics (#8583)
Motivation

#8563 highlighted race conditions introduced by the prior optimistic
update optimization in 83a19d5650. These
were known at the time but considered acceptable given the perf
benefit in high contention scenarios.

This PR proposes a modified approach which provides roughly half the
gains but stronger concurrency semantics. Race conditions still exist
but their scope is narrowed to much less likely cases (releases
coinciding with retain overflow), and even in those
cases certain guarantees are still assured. Once release() returns true,
all subsequent release/retains are guaranteed to throw, and in
particular deallocate will be called at most once.

Modifications

- Use even numbers internally (including -ve) for live refcounts
- "Final" release changes to odd number (equivalent to refcount 0)
- Retain still uses faster getAndAdd, release uses CAS loop
- First CAS attempt uses non-volatile read
- Thread.yield() after a failed CAS provides a net gain

Result

More (though not completely) robust concurrency semantics for ref
counting; increased latency under high contention, but still roughly
twice as fast as the original logic. Bench results to follow
2018-11-29 08:32:32 +01:00
Norman Maurer
057c19f92a
Move less common code-path to extra method to allow inlining of writeUtf8. (#8600)
Motivation:

ByteBuf is used everywhere so we should try hard to be able to make things inlinable. During benchmarks it showed that writeCharSequence(...) fails to inline writeUtf8 because it is too big even if its hots.

Modifications:

Move less common code-path to extra method to allow inlining.

Result:

Be able to inline writeUtf8 in most cases.
2018-11-27 21:03:35 +01:00
Norman Maurer
15e4fe05a8 Revert "Provide a way to cache the internal nioBuffer of the PooledByteBuffer to reduce GC. (#8593)"
This reverts commit 8cd005ba43 as it seems to produce some failures in some cases. This needs more research.
2018-11-27 20:02:34 +01:00
Norman Maurer
8cd005ba43
Provide a way to cache the internal nioBuffer of the PooledByteBuffer to reduce GC. (#8593)
Motivation:

Often a temporary ByteBuffer is used which can be cached to reduce the GC pressure.

Modifications:

Add a Deque per PoolChunk which will be used for caching.

Result:

Less GC.
2018-11-27 13:55:13 +01:00
Rolandz
89639ce322 Fix offset calculation in PooledByteBufAllocator when used
Motivation:

When we create new chunk with memory aligned, the offset of direct memory should be
'alignment - address & (alignment - 1)', not just 'address & (alignment - 1)'.

Modification:

Change offset calculating formula to offset = alignment - address & (alignment - 1) in PoolArena.DirectArena#offsetCacheLine and add a unit test to assert that.

Result:

Correctly calculate offset.
2018-11-27 11:47:34 +01:00
Norman Maurer
f4e4147df8
LocationAwareSlf4jLogger does not correctly format log message. (#8595)
Motivation:

We did miss to use MessageFormatter inside LocationAwareSlf4jLogger and so {} was not correctly replaced in log messages when using slf4j.
This regression was introduced by afe0767e9c.

Modifications:

- Make use of MessageFormatter
- Add unit test.

Result:

Fixes https://github.com/netty/netty/issues/8483.
2018-11-27 11:44:27 +01:00
Norman Maurer
2278991db7
Use addAndGet(...) as a replacement for compareAndSet(...) when tracking the direct memory usage. (#8596)
Motivation:

We can change from using compareAndSet to addAndGet, which emits a different CPU instruction on x86 (CMPXCHG to XADD) when count direct memory usage. This instruction is cheaper in general and so produce less overhead on the "happy path". If we detect too much memory usage we just rollback the change before throwing the Error.

Modifications:

Replace compareAndSet(...) with addAndGet(...)

Result:

Less overhead when tracking direct memory.
2018-11-27 08:33:28 +01:00
Norman Maurer
af63626777
Factor out less common code-path into own method to allow inlining. (#8590)
Motivation:

During benchmarks two methods showed up as "hot method too big". We can easily make these smaller by factor out some less common code-path to an extra method and so allow inlining.

Modifications:

Factor out less common code path to an extra method.

Result:

Hot methods can be inlined.
2018-11-25 21:46:14 +01:00
Norman Maurer
af34287fd1
HeadContext is inbound and outbound (#8592)
Motivation:

Our HeadContext in DefaultChannelPipeline does handle inbound and outbound but we only marked it as outbound. While this does not have any effect in the current code-base it can lead to problems when we change our internals (this is also how I found the bug).

Modifications:

Construct HeadContext so it is also marked as handling inbound.

Result:

More correct code.
2018-11-24 10:47:56 +01:00
Norman Maurer
2a2bc21067
Remove @Deprecated from package-info.java file (#8591)
Motivation:

31fd66b617 added @Deprecated to some classes but also to the package-info.java files. IntelliJ does not like to have these annotations on package-info.java

Modifications:

Remove annotation from package-info.java

Result:

Be able to compile against via IntelliJ
2018-11-23 17:03:29 +01:00
Norman Maurer
31fd66b617
Mark OIO based transports as deprecated as preparation for removal in Netty 5. (#8579)
Motivation:

We plan to remove the OIO based transports in Netty 5 so we should mark these as deprecated already.

Modifications:

Mark all OIO based transports as deprecated.

Result:

Give the user a heads-up for removal.
2018-11-21 15:15:01 +01:00
Norman Maurer
d728a72e74
Combine flushes in DnsNameResolver to allow usage of sendmmsg to reduce syscall costs (#8470)
Motivation:

Some of transports support gathering writes when using datagrams. For example this is the case for EpollDatagramChannel. We should minimize the calls to flush() to allow making efficient usage of sendmmsg in this case.

Modifications:

- minimize flush() operations when we query for multiple address types.
- reduce GC by always directly schedule doResolveAll0(...) on the EventLoop.

Result:

Be able to use sendmmsg internally in the DnsNameResolver.
2018-11-21 06:42:40 +01:00
Norman Maurer
3d2fdc459c
Remove transitive dependency on slf4j in example (#8582)
Motivation:

We currently depend on slf4j in an transitive way in one of our classes in the examples. We should not do this.

Modifications:

Remove logging in example.

Result:

Remove not needed dependency.
2018-11-21 06:39:28 +01:00
Norman Maurer
cd689ee775
Fix javadoc to correctly explain how ChannelDuplexHandler.deregister(...) works. (#8577)
Motivation:

We had an error in the javadoc which was most likely caused by copy and paste.

Modifications:

Fix javadoc.

Result:

Correct javadoc.
2018-11-20 16:45:15 +01:00
Norman Maurer
38524ec3e2
Fix test that assumed detection of peer supported algs is not supported in BoringSSL. (#8573)
Motivation:

0d2e38d5d6 added supported for detection of peer supported algorithms but we missed to fix the testcase.

Modifications:

Fix test-case.

Result:

No more failing tests with BoringSSL.
2018-11-19 12:13:05 +01:00
Norman Maurer
278b49b2a7
Recover from Selector IOException (#8569)
Motivation:

When the Selector throws an IOException during our EventLoop processing we should rebuild it and transfer the registered Channels. At the moment we will continue trying to use it which will never work.

Modifications:

- Rebuild Selector when an IOException is thrown during any select*(...) methods.
- Add unit test.

Result:

Fixes https://github.com/netty/netty/issues/8566.
2018-11-19 07:41:43 +01:00
Jake Luciani
63dc1f5aaa Allow adjusting of lead detection sampling interval. (#8568)
Motivation:

We should allow adjustment of the leak detecting sampling interval when in SAMPLE mode.

Modifications:

Added new int property io.netty.leakDetection.samplingInterval

Result:

Be able to consume changes made by the user.
2018-11-16 17:22:03 +01:00
Andremoniy
abc8a08c96 Rethrow Error during retrieving remoteAddress / localAddress
Motivation:

Besides an error caused by closing socket in Windows a bunch of other errors may happen at this place which won't be somehow logged. For instance any VirtualMachineError as OutOfMemoryError will be simply ignored. The library should at least log the problem.

Modification:

Added logging of the throwable object.

Result:

Fixes #8499.
2018-11-16 17:20:28 +01:00
Norman Maurer
cb0d23923f
Refresh DNS configuration each 5 minutes. (#8468)
Motivation:

We should refresh the DNS configuration each 5 minutes to be able to detect changes done by the user. This is inline with what OpenJDK is doing

Modifications:

Refresh config every 5 minutes.

Result:

Be able to consume changes made by the user.
2018-11-16 10:37:29 +01:00
JStroom
ce02d5a184 Update SslHandler.java (#8564)
Swallow SSL Exception "closing inbound before receiving peer's close_notify" when running on Java 11 (#8463)

Motivation:

When closing a inbound SSL connection before the remote peer has send a close notify, the Java JDK is trigger happy to throw an exception. This exception can be ignored since the connection is about to be closed.
The exception wasn't printed in Java 8, based on filtering on the exception message. In Java 11 the exception message has been changed.

Modifications:
Update the if statement to also filter/swallow the message on Java 11.

Result:
On Java 11 the exception isn't printed with log levels set to debug. The old behaviour is maintained.
2018-11-16 10:32:34 +01:00
Norman Maurer
8d4d76d216
ReferenceCountedOpenSslEngine SSLSession.getLocalCertificates() / getLocalPrincipial() did not work when KeyManagerFactory was used. (#8560)
Motivation:

The SSLSession.getLocalCertificates() / getLocalPrincipial() methods did not correctly return the local configured certificate / principal if a KeyManagerFactory was used when configure the SslContext.

Modifications:

- Correctly update the local certificates / principial when the key material is selected.
- Add test case that verifies the SSLSession after the handshake to ensure we correctly return all values.

Result:

SSLSession returns correct values also when KeyManagerFactory is used with the OpenSSL provider.
2018-11-16 07:38:32 +01:00
Norman Maurer
20d4fda55e
Return the correct pointer from ReferenceCountedOpenSslContext.context() and sslCtxPointer() (#8562)
Motivation:

We did not return the pointer to SSL_CTX put to the internal datastructure of tcnative.

Modifications:

Return the correct pointer.

Result:

Methods work as documented in the javadocs.
2018-11-16 07:37:57 +01:00
Norman Maurer
7667361924
Update to netty-tcnative 2.0.20.Final (#8561)
Motivation:

Update to netty-tcnative 2.0.20.Final which fixed a bug related to retrieving the remote signature algorithms when using BoringSSL.

Modifications:

Update netty-tcnative

Result:

Be able to correctly detect the remote signature algorithms when using BoringSSL.
2018-11-15 18:05:15 +01:00
Norman Maurer
845a65b31c
Nio|Epoll|KqueueEventLoop task execution might throw UnsupportedOperationException on shutdown. (#8476)
Motivation:

There is a racy UnsupportedOperationException instead because the task removal is delegated to MpscChunkedArrayQueue that does not support removal. This happens with SingleThreadEventExecutor that overrides the newTaskQueue to return an MPSC queue instead of the LinkedBlockingQueue returned by the base class such as NioEventLoop, EpollEventLoop and KQueueEventLoop.

Modifications:

- Catch the UnsupportedOperationException
- Add unit test.

Result:

Fix #8475
2018-11-15 07:19:28 +01:00
Norman Maurer
0d2e38d5d6
Correctly convert supported signature algorithms when using BoringSSL (#8481)
* Correctly convert supported signature algorithms when using BoringSSL

Motivation:

BoringSSL uses different naming schemes for the signature algorithms so we need to adjust the regex to also handle these.

Modifications:

- Adjust SignatureAlgorithmConverter to handle BoringSSL naming scheme
- Ensure we do not include duplicates
- Add unit tests.

Result:

Correctly convert boringssl signature algorithm names.
2018-11-14 19:23:11 +01:00
Norman Maurer
d1654484c1
Correctly convert between openssl / boringssl and java cipher names when using TLSv1.3 (#8485)
Motivation:

We did not correctly convert between openssl / boringssl and java ciphers when using TLV1.3 which had different effects when either using openssl or boringssl.
 - When using openssl and TLSv1.3 we always returned SSL_NULL_WITH_NULL_NULL as cipher name
 - When using boringssl with TLSv1.3 we always returned an incorrect constructed cipher name which does not match what is defined by Java.

Modifications:

 - Add correct mappings in CipherSuiteConverter for both openssl and boringssl
 - Add unit tests for CipherSuiteConvert
 - Add unit in SSLEngine which checks that we do not return SSL_NULL_WITH_NULL_NULL ever and that server and client returns the same cipher name.

Result:

Fixes https://github.com/netty/netty/issues/8477.
2018-11-14 08:49:13 +01:00
Tim Brooks
11ec7d892e Cleanup SslHandler handshake/renegotiation (#8555)
Motivation:

The code for initiating a TLS handshake or renegotiation process is
currently difficult to reason about.

Modifications:

This commit introduces to clear paths for starting a handshake. The
first path is a normal handshake. The handshake is started and a timeout
is scheduled.

The second path is renegotiation. If the first handshake is incomplete,
the renegotiation promise is added as a listener to the handshake
promise. Otherwise, the renegotiation promise replaces the original
promsie. At that point the handshake is started again and a timeout is
scheduled.

Result:

Cleaner and easier to understand code.
2018-11-14 08:19:06 +01:00
Bryce Anderson
044515f369 Defer HTTP/2 stream transition state on initial write until headers are written (#8471)
Motivation:
When the DefaultHttp2ConnectionEncoder writes the initial headers for a new
locally created stream we create the stream in the half-closed state if the
end-stream flag is set which signals to the life cycle manager that the headers
have been sent. However, if we synchronously fail to write the headers the
life cycle manager then sends a RST_STREAM on our behalf which is a connection
level PROTOCOL_ERROR because the peer sees the stream in an IDLE state.

Modification:
Don't open the stream in the half-closed state if the end-stream flag is
set and let the life cycle manager take care of it.

Result:
Cleaner state management in the DefaultHttp2ConnectionEncoder.

Fixes #8434.
2018-11-14 08:17:43 +01:00