Commit Graph

9073 Commits

Author SHA1 Message Date
Norman Maurer
f4bafd4fe0
Correctly check if cipher is supported for each SslProvider before trying to run test and fix buffer leaks in test. (#8247)
Motivation:

5aaa16b24c introduced a testcase for specific ciphersuites and checked if these are supported by our native implementation before running it. Unfortunally this is not good enough as even on the JDK it may not be supported on various JDK versions (like Java7). Beside this the test leaked buffers.

Modifications:

- Correctly check if ciphersuite is supported on each SslProvider before trying to run test.
- Fix buffer leaks.

Result:

Testsuite pass again on Java7 and others when -Pleak is used.
2018-09-01 08:11:13 +02:00
Norman Maurer
c1a335446d
Correctly implement SelectedSelectionKeySet iterator(), contains(...) and remove(...) (#8244)
Motivation:

Our SelectedSelectionKeySet does not correctly implement various methods which can be done without any performance overhead.

Modifications:

Implement iterator(), contains(...) and remove(...)

Result:

Related to https://github.com/netty/netty/issues/8242.
2018-09-01 08:10:02 +02:00
Norman Maurer
9d8846cfce
Cleanup Log4J2Logger (#8245)
Motivation:

Log4J2Logger had some code-duplication with AbstractInternalLogger

Modifications:

Reuse AbstractInternaLogger.EXCEPTION_MESSAGE in Log4J2Logger and so remove some code-duplication

Result:

Less duplicated code.
2018-08-31 17:08:38 +02:00
Norman Maurer
e26666a7ea
Update to latest netty-tcnative (#8246)
Motivation:

We should use the latest netty-tcnative release which contains a fix to correctly support DH based ciphers when using openssl 1.1.x

Modifications:

Update to latest netty-tcnative which has the fix.

Result:

Correctly support DH ciphers in all cases. Fixes https://github.com/netty/netty/issues/8165.
2018-08-31 17:08:14 +02:00
Chi-Joung So
a9863f8128 Add headers to MqttMessage returned by MqttDecoder in case of DecoderException (#8219)
Motivation:
When the MqttDecoder decodes a message larger than the 'maxBytesInMessage' a DecoderException is thrown and a MqttMessage with just the failure cause is returned. Even if I can't handle the message, I might want to send an ACK so that I won't have to worry about it again.

Modification:
The DecoderException is thrown after the variableHeader is decoded. The fixed and variable headers are then added to the MqttMessage along with the failure cause.

Result:
The invalid MqttMessage will have headers if available.
2018-08-31 15:06:09 +02:00
Norman Maurer
a644563625
Add more debug informations when log SSL errors. (#8241)
Motivation:

ea626ef8c3 added more debug logging but we can even include a bit more.

Modifications:

Always log the error number as well.

Result:

More informations for debugging SSL errors.
2018-08-30 20:44:47 +02:00
Norman Maurer
38eee409c8
We should be able to use the ByteBuffer cleaner on java8 (and earlier… (#8234)
* We should be able to use the ByteBuffer cleaner on java8 (and earlier versions) even if sun.misc.Unsafe is not present.

Motivation:

At the moment we have a hard dependency on sun.misc.Unsafe to use the Cleaner on Java8 and earlier. This is not really needed as we can still use pure reflection if sun.misc.Unsafe is not present.

Modifications:

Refactor Cleaner6 to fallback to pure reflection if sun.misc.Unsafe is not present on system.

Result:

More timely releasing of direct memory on Java8 and earlier when sun.misc.Unsafe is not present.
2018-08-30 07:43:10 +02:00
Norman Maurer
4a5b61fc13
Fix log message about using non-direct buffers by default (#8235)
Motivation:

f77891cc17 changed slightly how we detect if we should prefer direct buffers or not but did miss to also take this into account when logging.

Modifications:

Fix branch for log message to reflect changes in f77891cc17.

Result:

Correct logging.
2018-08-30 06:57:12 +02:00
Norman Maurer
b73f785631
We should call the UnLoad methods when we detect an error during calling OnLoad (#8237)
Motivation:

We should ensure we call *UnLoad when we detect an error during calling *OnLoad and previous *OnLoad calls were succesfull.

Modifications:

Correctly call *UnLoad when needed.

Result:

More correct code and no leaks when an error happens during loading the native lib.
2018-08-30 06:56:42 +02:00
Norman Maurer
ea626ef8c3
Log more details when shutdown SSL because of an error. (#8236)
Motivation:

We should log a bit more details about why we shutdown the SSL.

Modifications:

Add the return value of SSL_get_error(...) as well in debug mode.

Result:

More logging to make it easier to understand why an SSL error happened.
2018-08-29 21:52:26 +02:00
Terence Yim
79706357c7 Fix race condition in the NonStickyEventExecutorGroup (#8232)
Motivation:

There was a race condition between the task submitter and task executor threads such that the last Runnable submitted may not get executed. 

Modifications:

The bug was fixed by checking the task queue and state in the task executor thread after it saw the task queue was empty.

Result:

Fixes #8230
2018-08-29 19:42:01 +02:00
Norman Maurer
54f565ac67
Allow to use native transports when sun.misc.Unsafe is not present on… (#8231)
* Allow to use native transports when sun.misc.Unsafe is not present on the system

Motivation:

We should be able to use the native transports (epoll / kqueue) even when sun.misc.Unsafe is not present on the system. This is especially important as Java11 will be released soon and does not allow access to it by default.

Modifications:

- Correctly disable usage of sun.misc.Unsafe when -PnoUnsafe is used while running the build
- Correctly increment metric when UnpooledDirectByteBuf is allocated. This was uncovered once -PnoUnsafe usage was fixed.
- Implement fallbacks in all our native transport code for when sun.misc.Unsafe is not present.

Result:

Fixes https://github.com/netty/netty/issues/8229.
2018-08-29 19:36:33 +02:00
Roger
5aaa16b24c An unit test to ensure that cipher suites don't break/disappear between releases. (#8225)
Motivation

Ensure classes of cipher suites continue working between releases. Adding just a DHE check for now as it caused #8165 but this test can be expaned to other suites.

Modifications

Adding an unit test that checks for the presence of a cipher suite.

Result

Prevent #8165 from happening in the future.
2018-08-29 14:14:26 +02:00
Norman Maurer
f77891cc17
We should prefer direct buffers if we can access the cleaner even if sun.misc.Unsafe is not present. (#8233)
Motivation:

We should prefer direct buffers whenever we can use the cleaner even if sun.misc.Unsafe is not present.

Modifications:

Correctly prefer direct buffers in all cases.

Result:

More correct code.
2018-08-29 08:21:07 +02:00
Norman Maurer
8679c5ef43
CleanerJava9 should be able to do its job even with a SecurityManager installed. (#8204)
Motivation:

CleanerJava9 currently fails whever a SecurityManager is installed. We should make use of AccessController.doPrivileged(...) so the user can give it the correct rights.

Modifications:

Use doPrivileged(...) when needed.

Result:

Fixes https://github.com/netty/netty/issues/8201.
2018-08-28 16:32:29 +02:00
zhaojigang
338ef96931 Recycler will produce npe error when multiple recycled at different thread
Motivation:

Recycler may produce a NPE when the same object is recycled multiple times from different threads.

Modifications:

- Check if the id has changed or if the Stack became null and if so throw an IllegalStateException
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/8220.
2018-08-27 08:58:40 +02:00
Norman Maurer
1bd9e662dd
Clarify deprecation docs a bit. (#8226)
Motivation:

It seems to sometimes confuse people what to do to replace setMaxMessagePerRead(...).

Modifications:

Add some more details to the javadocs about the correct replacement.

Result:

Related to https://github.com/netty/netty/issues/8214.
2018-08-25 08:14:38 +02:00
Norman Maurer
37a4f99f25
Use Java11+ea28 during build. (#8113)
Motivation:

We should ensure we use the latest Java11 EA during build to catch any regressions etc.

Modifications:

Update from ea19 to ea28.

Result:

Use latest Java11 release.
2018-08-25 07:26:56 +02:00
Norman Maurer
6888af6ba5
Allow to parse hosts file which is stored in a different encoding then the default system encoding. (#8211)
Motivation:

We should support to parse and read a hosts file which is stored in a different encoding then the system default. Beside this when we are on windows we should just try to parse it with multiple different charset before giving up as there is no real standard what charset to use.

Modifications:

- Add more method overloads to HostsFileParser that take a Charset.
- Try to parse with multiple Charsets in DefaultHostsFileEntriesResolver when windows is used.
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/8208.
2018-08-24 19:48:27 +02:00
root
a580dc7585 [maven-release-plugin] prepare for next development iteration 2018-08-24 06:36:33 +00:00
root
3fc789e83f [maven-release-plugin] prepare release netty-4.1.29.Final 2018-08-24 06:36:06 +00:00
Norman Maurer
a0a4d87eab
Update to netty-tcnative 2.0.14 which does correctly handle shading (#8218) 2018-08-24 06:42:21 +02:00
Norman Maurer
2bb9f64e16
Try to monkey-patch library id when shading is used and we are on Mac… (#8210)
* Try to monkey-patch library id when shading is used and we are on MacOS / OSX.

Motivation:

ea4c315b45 did ensure we support using multiple versions of the same shaded native library but the user still needed to run install_name_tool -id on MacOS to ensure the ID is unique.
This is kind of error prone and also means that the shading itself would need to be done on MacOS / OSX.

This is related to https://github.com/netty/netty/issues/7272.

Modifications:

- Monkey patch the shaded native lib on MacOS to ensure the id is unique while unpacking it to the tempory location.

Result:

Easier way of using shaded native libs in netty.
2018-08-23 11:07:09 +02:00
Norman Maurer
bbb6e126b1
Correctly handle DNS redirects for NS servers that have no ADDITIONAL record (#8177)
Motiviation:

We incorrectly did ignore NS servers during redirect which had no ADDITIONAL record. This could at worse have the affect that we failed the query completely as none of the NS servers had a ADDITIONAL record. Beside this using a DnsCache to cache authoritative nameservers does not work in practise as we we need different features and semantics when cache these servers (for example we also want to cache unresolved nameservers and resolve these on the fly when needed).

Modifications:

- Correctly take NS records into account that have no matching ADDITIONAL record
- Correctly handle multiple ADDITIONAL records for the same NS record
- Introduce AuthoritativeDnsServerCache as a replacement of the DnsCache when caching authoritative nameservers + adding default implementation
- Add an adapter layer to reduce API breakage as much as possible
- Replace DnsNameResolver.uncachedRedirectDnsServerStream(...) with newRedirectDnsServerStream(...)
- Add unit tests

Result:

Our DnsResolver now correctly handle redirects in all cases.
2018-08-22 17:49:22 +02:00
Norman Maurer
ea4c315b45
Ensure multiple shaded version of the same netty artifact can be loaded as long as the shaded prefix is different (#8207)
Motivation:

We should support to load multiple shaded versions of the same netty artifact as netty is often used in multiple dependencies.

This is related to https://github.com/netty/netty/issues/7272.

Modifications:

- Use -fvisibility=hidden when compiling and use JNIEXPORT for things we really want to have exported
- Ensure fields are declared as static so these are not exported
- Adjust testsuite-shading to use install_name_tool on MacOS to change the id of the lib. Otherwise the wrong may be used.

Result:

Be able to use multiple shaded versions of the same netty artifact.
2018-08-21 07:53:45 +02:00
Norman Maurer
182ffdaf6d
Only use manual safepoint polling in PlatformDependent0.copyMemory(...) when using java <= 8 (#8124)
Motivation:

Java9 and later does the safepoint polling by itself so there is not need for us to do it.

Modifications:

Check for java version before doing manual safepoint polling.

Result:

Less custom code and less overhead when using java9 and later. Fixes https://github.com/netty/netty/issues/8122.
2018-08-18 21:09:18 +02:00
Ziyan Mo
785473788f (Nio|Epoll)EventLoop.pendingTasks does not need to dispatch to the EventLoop (#8197)
Motivation:

EventLoop.pendingTasks should be (reasonably) cheap to invoke so it can be used within observability. 

Modifications:

Remove code that dispatch access to the internal taskqueue to the EventLoop when invoked as this is not needed anymore with the current MPSC queues we are using. 

See https://github.com/netty/netty/issues/8196#issuecomment-413653286.

Result:

Fixes https://github.com/netty/netty/issues/8196
2018-08-18 07:28:31 +02:00
Norman Maurer
df00539fa2
Allow to load PrivateKey via OpenSSL Engine (#8200)
Motivation:

OpenSSL itself has an abstraction which allows you to customize some things. For example it is possible to load the PrivateKey from the engine. We should support this.

Modifications:

Add two new static methods to OpenSslX509KeyManagerFactory which allow to create an OpenSslX509KeyManagerFactory that loads its PrivateKey via the OpenSSL Engine directly.

Result:

More flexible usage of OpenSSL possible
2018-08-18 07:20:44 +02:00
Norman Maurer
bbe2e4d224
We should try to load netty-tcnative before using it in OpenSslCertificateException. (#8202)
Motivation:

In OpenSslCertificateException we should ensure we try to load netty-tcnative before trying to use any class from it as otherwise it may throw an error due missing linking of the native libs.

Modifications:

- Ensure we call OpenSsl.isAvailable() before we try to use netty-tcnative for validation
- Add testcase.

Result:

No more errors causing by not loading native libs before trying to use these.
2018-08-18 06:26:45 +02:00
Norman Maurer
8255f85f24
Rename SslHandler.close(...) to closeOutbound(...) as it is still useful and delegate to the methods. (#8193)
* Rename SslHandler.close(...) to closeOutbound(...) as it is still useful and delegate to the methods.

Motivation:

Sometimes the user may want to send a close_notify without closing the underlying Channel. For this we offered the SslHandler.close(...) methods which were marked as deeprecated. We should offer an way to still do this without the user calling deprecated methods.

See https://stackoverflow.com/questions/51710231/using-nettys-sslhandlerclosechannelhandlercontext-channelpromise/51753742#comment90555949_51753742 .

Modifications:

- Remove deprecation of the SslHandler.close(...) method that exactly allows this and rename these to closeOutbound(...) as this is more clear.
- Add close(...) methods that delegate to these and mark these as deprecated.

Result:

Be able to send close_notify without closing the Channel.
2018-08-15 20:07:56 +02:00
Norman Maurer
2fa7a0aa57
Keep the amount of scheduled tasks for DefaultDnsCache at a minimum (#8187)
Motivation:

We are currently always remove all entries from the cache for a hostname if the lowest TTL was reached but schedule one for each of the cached entries. This is wasteful.

Modifications:

- Reimplement logic to schedule TTL to only schedule a new removal task if the requested TTL was actual lower then the one for the already scheduled task.
- Ensure we only remove from the internal map if we did not replace the Entries in the meantime.

Result:

Less overhead in terms of scheduled tasks for the DefaultDnsCache
2018-08-15 09:07:13 +02:00
Norman Maurer
bd25fd03e3
Add testcase for ChannelInitializer.initChannel(...) when throwing an Exception (#8188)
Motivation:

We had a report that the exception may not be correctly propagated. This test shows it is.

Modifications:

Add testcase.

Result:

Test for https://github.com/netty/netty/issues/8158
2018-08-10 08:54:21 +02:00
Norman Maurer
f22781f176
Correctly handle hostnames with and without trailing dot in the DefaultDnsCache and use it for searchdomains. (#8181)
Motivation:

We should ensure we return the same cached entries for the hostname and hostname ending with dot. Beside this we also should use it for the searchdomains as well.

Modifications:

- Internally always use hostname with a dot as a key and so ensure we correctly handle it in the cache.
- Also query the cache for each searchdomain
- Add unit tests

Result:

Use the same cached entries for hostname with and without trailing dot. Query the cache for each searchdomain query as well
2018-08-10 08:53:59 +02:00
Norman Maurer
56eb1e92cc
Add tests to verify caches are cleared when the resolver is closed. (#8186)
Motivation:

55fec94592 fixed a bug where we did not correctly clear all caches when the resolver was closed but did not add a testcase.

Modifications:

Add testcase.

Result:

More tests.
2018-08-10 08:46:15 +02:00
Lai Jiang
dbc9ec1ab2 Set SNI servernames in OpenSSL engine when created in client mode (#8178)
Motivation:

When using the JDK SSL provider in client mode, the SNI host names (called serverNames in SslEngineImpl) is set to the peerHost (if available) that is used to initialize the SSL Engine:

http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/security/ssl/SSLEngineImpl.java#l377

This allows one to call SslEngine.getSSLParameters() and inspect what is the SNI name to be sent. The same should be done in the OpenSSL provider as well. Currently even though the the SNI name is sent by the OpenSSL provider during handshake when the peerHost is specified, it is missing from the parameters.

Modification:

Set the sniHostNames field when SNI is to be used. Also verifies the peer is actually a hostname before setting it as the SNI name, which is consistent with JDK SSL provider's behavior.

Result:

SslEngine using the OpenSSL provider created in client mode with peerHost will initialize sniHostNames with the peerHost.

Calling SslEngine.getSSLParameters().getServerNames() will return a list that contains that name.
2018-08-09 21:30:02 +02:00
Norman Maurer
534de73d28
Workaround JDK bug that will cause an AssertionError when calling ServerSocketChannel.config().getOptions(). (#8183)
Motivation:

There is a JDK bug which will return IP_TOS as supported option for ServerSocketChannel even if its not supported afterwards and cause an AssertionError.
See http://mail.openjdk.java.net/pipermail/nio-dev/2018-August/005365.html.

Modifications:

Add a workaround for the JDK bug.

Result:

ServerSocketChannel.config().getOptions() will not throw anymore and work as expected.
2018-08-09 13:11:08 +02:00
Scott Mitchell
b3b04d0de2 DnsNameResolver hangs if search domain results in invalid hostname (#8180)
Motivation:
DnsNameResolver manages search domains and will retry the request with the different search domains provided to it. However if the query results in an invalid hostname, the Future corresponding to the resolve request will never be completed.

Modifications:
- If a resolve attempt results in an invalid hostname and the query isn't issued we should fail the associated promise

Result:
No more hang from DnsNameResolver if search domain results in invalid hostname.
2018-08-08 08:14:18 +02:00
vincent-grosbois
0bea8ecf5d CompositeByteBuf nioBuffer doesn't always alloc (#8176)
In nioBuffer(int,int) in CompositeByteBuf , we create a sub-array of nioBuffers for the components that are in range, then concatenate all the components in range into a single bigger buffer.
However, if the call to nioBuffers() returned only one sub-buffer, then we are copying it to a newly-allocated buffer "merged" for no reason.

Motivation:

Profiler for Spark shows a lot of time spent in put() method inside nioBuffer(), while usually no copy of data is required.

Modification:
This change skips this last step and just returns a duplicate of the single buffer returned by the call to nioBuffers(), which will in most implementation not copy the data

Result:
No copy when the source is only 1 buffer
2018-08-07 11:31:24 +02:00
Norman Maurer
55fec94592
Also clear the authoritativeDnsServerCache when closing the Channel. (#8174)
Motivation:

At the moment we only clear the resolveCache when the Channel is closed. We should also do the same for the authoritativeDnsServerCache.

Modifications:

Add authoritativeDnsServerCache.clear() to the Channel closeFuture.

Result:

Correctly clear all caches.
2018-08-06 08:31:17 +02:00
Norman Maurer
2c13f71c73
Ensure NIO transport can be used on Java6 again. (#8168)
Motivation:

952eeb8e1e introduced the possibility to use any JDK SocketOption when using the NIO transport but broke the possibility to use netty with java6.

Modifications:

Do not use java7 types in method signatures of the static methods in NioChannelOption to prevent class-loader issues on java6.

Result:

Fixes https://github.com/netty/netty/issues/8166.
2018-08-03 07:07:09 +02:00
Norman Maurer
44d3753c48
Fix NPE exception when using invalid cipher during building SslContext. (#8171)
Motivation:

We missed to do a null check before trying to destroy the OpenSslSessionContext, which could lead to a NPE.

Modifications:

Add null check and tests.

Result:

Fix https://github.com/netty/netty/issues/8170.
2018-08-02 21:42:21 +02:00
Norman Maurer
3ab7cac620 Disable test as it sometimes fails on the CI
Motivation:

Temporary disable test that wwas introduced as part of f60d08fd32 as it sometimes fail on the CI. We need to figure out why it fails there (can not reproduce so far even on the CI after ssh into it).

Modifications:

Ignore test.

Result:

More stable builds until we figure out the flackyness.
2018-08-01 08:31:31 +02:00
Norman Maurer
fe14bad69c
Adjust SSL related tests to be more correct and so pass in the next EA release of java11. (#8162)
Motivation:

In some of our tests we not correctly init the SSLEngine before trying to perform a handshake which can cause an IllegalStateException. While this not happened in previous java releases it does now on Java11 (which is "ok" as its even mentioned in the api docs). Beside this how we selected the ciphersuite to test renegotation was not 100 % safe.

Modifications:

- Correctly init SSLEngine before using it
- Correctly select ciphersuite before testing for renegotation.

Result:

More correct tests and also pass on next java11 EA release.
2018-08-01 06:37:53 +02:00
Nick Hill
630c82717d Lazy initialize NativeDatagramPacketArray and IovArray in EpollEventLoop (#8160)
Motivation:

Avoid unnecessary native memory allocation if UDP / TCP isn't being
used.

Modifications:

Create the reused NativeDatagramPacketArray and IovArray upon first use
instead of EpollEventLoop construction.

Also correct related comment in NativeDatagramPacketArray.

Result:

Reduced native memory use when using epoll in many cases
2018-07-29 18:22:27 +08:00
Norman Maurer
f60d08fd32
Add test for shading netty-tcnative. (#8157)
Motivation:

d67d639f5f added a test for shading the native transport of netty. We should also test that shading netty-tcnative is possible.

Modifications:

Add test for shading netty-tcnative

Result:

More testing.
2018-07-28 02:53:43 +08:00
root
fcb19cb589 [maven-release-plugin] prepare for next development iteration 2018-07-27 04:59:28 +00:00
root
ff785fbe39 [maven-release-plugin] prepare release netty-4.1.28.Final 2018-07-27 04:59:06 +00:00
Scott Mitchell
53b2dea3f4
HTTP/2 child channel read cycle doesn't respect RecvByteBufAllocator and (#8147)
Motivation:
Http2MultiplexCodec queues data internally if data is delivered from the
parent channel but the child channel did not request data. If the parent
channel notifies of a stream closure it is possible data in the queue
will be discarded before closing the channel.
Http2MultiplexCodec interacts with RecvByteBufAllocator to control the
child channel's demand for read. However it currently only ever reads a
maximum of one time per loop. This can thrash the read loop and bloat
the call stack if auto read is on, because channelReadComplete will
re-enter the read loop synchronously, and also neglect to deliver data
during the parent's read loop (if it is active). This also meant the
readPendingQueue was not utilized as originally intended (to extend the
child channel's read loop during the parent channel's read loop if
demand for data still existed).

Modifications:
- Modify the child channel's read loop to respect the
RecvByteBufAllocator, and append to the parents readPendingQueue if
appropriate.
- Stream closure notification behaves like EPOLL and KQUEUE transports
and reads all queued data, because the data is already queued in memory
and it is known there will be no more data. This will also replenish the
connection flow control window which may otherwise be constrained by a
closed stream.

Result:
More correct read loop and less risk of dropping data.
2018-07-26 19:44:21 -04:00
Norman Maurer
620dad0c26
Allow to validate sni hostname with underscore (#8150)
Motivation:

We should allow to also validate sni hostname which contains for example underscore when using our native SSL impl. The JDK implementation does this as well.

Modifications:

- Construct the SNIHostName via byte[] and not String.
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/8144.
2018-07-27 01:56:32 +08:00
Norman Maurer
9b08dbca00
Leak detection combined with composite buffers results in incorrectly handled writerIndex when calling ByteBufUtil.writeAscii/writeUtf8 (#8153)
Motivation:

We need to add special handling for WrappedCompositeByteBuf as these also extend AbstractByteBuf, otherwise we will not correctly adjust / read the writerIndex during processing.

Modifications:

- Add instanceof checks for WrappedCompositeByteBuf as well.
- Add testcases

Result:

Fixes https://github.com/netty/netty/issues/8152.
2018-07-27 01:56:09 +08:00