Commit Graph

9152 Commits

Author SHA1 Message Date
Norman Maurer
4d1458604a
Fix leak in SniClientTest. (#8324)
Motivation:

We need to release the ReferenceCountedSslContext to eliminate resource leaks.

Reported in https://garage.netty.io/teamcity/viewLog.html?buildId=33353&buildTypeId=netty_build_oraclejdk8&tab=buildLog#_focus=157264.

Modifications:

Call release on the SslContext instances.

Result:

No more leaks in tests.
2018-09-26 19:59:00 +02:00
Norman Maurer
60a7ece4c3
Update to final Java11 release (#8320)
Motivation:

We should use final Java11 release during builds.

Modifications:

Update to final Java11 release

Result:

Use latest release.
2018-09-26 12:16:19 +02:00
Norman Maurer
9a3be347af
Ensure we always encode all data in JdkZlibEncoder. (#8305)
Motivation:

In theory our estimation of the needed buffer could be off and so we need to ensure we grow it if there is no space left.

Modifications:

Ensure we grow the buffer if there is no space left in there but we still have data to deflate.

Result:

Correctly deflate data in all cases.
2018-09-22 13:34:12 -07:00
Norman Maurer
a80c49828f
Cleanup SSL test. (#8301)
Motivation:

I noticed that we had some errors showing up in a test (which did not fail it tho) because we tried to full-fill the promise multiples times.

Modifications:

Use trySuccess(...) as we may produce multiple exceptions.

Result:

Less errors during test-run.
2018-09-21 08:04:24 -07:00
Norman Maurer
01db30a163
Correctly implement ExtendedSSLSession.getStatusResponses() for ReferenceCountedOpenSslEngine (#8297)
Motivation:

Java9 added getStatusResponses() to ExtendedSSLSession which we should correctly support when possible.

Modifications:

Implement the method correctly.

Result:

More complete and correct implementation.
2018-09-19 17:13:44 -07:00
Björn Kautler
1b6e47ab2b Fix incorrectly encoded empty SOCKS5 address (#8292)
Motivation:

If you encode a SOCKS5 message like new DefaultSocks5CommandResponse(FAILURE, DOMAIN, "", 0)
you correctly get a result of 05010003000000.
But if the bndAddr is null, for example like new DefaultSocks5CommandResponse(FAILURE, DOMAIN)
the encoded result is 0501000301000000 which means the domain name has a length of one and consists of a 0-byte.

Modification:

With this commit it is also correctly encoded as a string of 0 length.

Result:

Correctly encode empty SOCKS5 address
2018-09-18 14:20:28 -07:00
Norman Maurer
687275361f
Update to Conscrypt 1.3.0 (#8296)
Motivation:

Conscrypt 1.3.0 was just released and adds support for TLSv1.3

Modifications:

Update to 1.3.0

Result:

Use latest conscrypt during build / test.
2018-09-18 14:19:06 -07:00
Andrey Mizurov
2ab3e13f08 Fix get charset from content-type header with multiple parameters (#8286)
Motivation:

Get charset from Content-Type header even it contains multiple parameters.

Modification:

Extract charset value from the charset parameter if it is not last.

Result:

Fixes #8273
2018-09-14 21:39:01 +02:00
Norman Maurer
34d52fcbfe
Implemented ExtendedOpenSslSession.getStatusResponses() so it not throws an UnsupportedOperationException. (#8290)
Motivation:

6ed7c6c75d added support for ExtendedOpenSslSession but we did not override getStatusResponses(). This lead to test failures on java9.

Modifications:

Implement ExtendedOpenSslSession.getStatusResponses() so it just returns an empty list.

Result:

Test pass again on Java9.
2018-09-14 20:33:09 +02:00
Norman Maurer
2b1514ec5a
Only use KeyManagerFactory in SniClientTest when supported by OpenSSL version. (#8289)
Motivation:

6ed7c6c75d added a test which blindly assumed we can use a KeyManagerFactory all the time. This is only true if have OpenSSL 1.0.2 or later, which may not be the case.

Modifications:

Only use KeyManagerFactory in test if the OpenSSL version does support it.

Result:

More robust tests.
2018-09-14 19:01:55 +02:00
Norman Maurer
6ed7c6c75d
Return an ExtendSSLSession whenever possible to allow more strict checking when using OpenSSL (#8281)
Motivation:

When an ExtendedSSLSession is used its possible to do more strict checking of the keys during handshake. We should do this whenever possible.

Modification:

- Return an ExtendedSSLSession when using client-mode and Java7+
- Add unit test
- Simplify unit tests

Result:

More consistent behaviour.
2018-09-14 14:33:11 +02:00
Norman Maurer
9eb124bb62
Don't cause ClassCastException if registration fails during constructing DnsNameResolver. (#8280)
Motivation:

We should not try to cast the Channel to a DatagramChannel as this will cause a ClassCastException.

Modifications:

- Do not cast
- rethrow from constructor if we detect the registration failed.
- Add unit test.

Result:

Propagate correct exception.
2018-09-11 20:34:37 +02:00
Carl Mastrangelo
1dff107de1 Don't re-arm timerfd each epoll_wait (#7816)
Motivation:
The Epoll transport checks to see if there are any scheduled tasks
before entering epoll_wait, and resets the timerfd just before.
This causes an extra syscall to timerfd_settime before doing any
actual work.   When scheduled tasks aren't added frequently, or
tasks are added with later deadlines, this is unnecessary.

Modification:
Check the *deadline* of the peeked task in EpollEventLoop, rather
than the *delay*.  If it hasn't changed since last time, don't
re-arm the timer

Result:
About 2us faster on gRPC RTT 50pct latency benchmarks.

Before (2 runs for 5 minutes, 1 minute of warmup):

```
50.0%ile Latency (in nanos):		64267
90.0%ile Latency (in nanos):		72851
95.0%ile Latency (in nanos):		78903
99.0%ile Latency (in nanos):		92327
99.9%ile Latency (in nanos):		119691
100.0%ile Latency (in nanos):		13347327
QPS:                           14933

50.0%ile Latency (in nanos):		63907
90.0%ile Latency (in nanos):		73055
95.0%ile Latency (in nanos):		79443
99.0%ile Latency (in nanos):		93739
99.9%ile Latency (in nanos):		123583
100.0%ile Latency (in nanos):		14028287
QPS:                           14936
```

After:
```
50.0%ile Latency (in nanos):		62123
90.0%ile Latency (in nanos):		70795
95.0%ile Latency (in nanos):		76895
99.0%ile Latency (in nanos):		90887
99.9%ile Latency (in nanos):		117819
100.0%ile Latency (in nanos):		14126591
QPS:                           15387

50.0%ile Latency (in nanos):		61021
90.0%ile Latency (in nanos):		70311
95.0%ile Latency (in nanos):		76687
99.0%ile Latency (in nanos):		90887
99.9%ile Latency (in nanos):		119527
100.0%ile Latency (in nanos):		6351615
QPS:                           15571
```
2018-09-11 13:38:38 +02:00
Matteo Merli
2a1596a4e9 Allow to configure socket option SO_BUSY_POLL (#8268)
Motivation:

When using Epoll based transport, allow applications to configure SO_BUSY_POLL socket option:

       SO_BUSY_POLL (since Linux 3.11)
              Sets the approximate time in microseconds to busy poll on a
              blocking receive when there is no data.  Increasing this value
              requires CAP_NET_ADMIN.  The default for this option is con‐
              trolled by the /proc/sys/net/core/busy_read file.

              The value in the /proc/sys/net/core/busy_poll file determines
              how long select(2) and poll(2) will busy poll when they oper‐
              ate on sockets with SO_BUSY_POLL set and no events to report
              are found.

              In both cases, busy polling will only be done when the socket
              last received data from a network device that supports this
              option.

              While busy polling may improve latency of some applications,
              care must be taken when using it since this will increase both
              CPU utilization and power usage.

Modification:

Added SO_BUSY_POLL socket option
Result:

Able to configure SO_BUSY_POLL from Netty
2018-09-07 20:50:51 +02:00
Norman Maurer
c14efd952d
Directly init refCnt to 1 (#8274)
Motivation:

We should just directly init the refCnt to 1 and not use the AtomicIntegerFieldUpdater.

Modifications:

Just assing directly to 1.

Result:

Cleaner code and possible a bit faster as the JVM / JIT may be able to optimize the first store easily.
2018-09-07 19:04:19 +02:00
Norman Maurer
e542a2cf26
Use a non-volatile read for ensureAccessible() whenever possible to reduce overhead and allow better inlining. (#8266)
Motiviation:

At the moment whenever ensureAccessible() is called in our ByteBuf implementations (which is basically on each operation) we will do a volatile read. That per-se is not such a bad thing but the problem here is that it will also reduce the the optimizations that the compiler / jit can do. For example as these are volatile it can not eliminate multiple loads of it when inline the methods of ByteBuf which happens quite frequently because most of them a quite small and very hot. That is especially true for all the methods that act on primitives.

It gets even worse as people often call a lot of these after each other in the same method or even use method chaining here.

The idea of the change is basically just ue a non-volatile read for the ensureAccessible() check as its a best-effort implementation to detect acting on already released buffers anyway as even with a volatile read it could happen that the user will release it in another thread before we actual access the buffer after the reference check.

Modifications:

- Try to do a non-volatile read using sun.misc.Unsafe if we can use it.
- Add a benchmark

Result:

Big performance win when multiple ByteBuf methods are called from a method.

With the change:
UnsafeByteBufBenchmark.setGetLongUnsafeByteBuf  thrpt   20  281395842,128 ± 5050792,296  ops/s

Before the change:
UnsafeByteBufBenchmark.setGetLongUnsafeByteBuf  thrpt   20  217419832,801 ± 5080579,030  ops/s
2018-09-07 07:47:02 +02:00
Norman Maurer
afe0767e9c
Log the correct line-number when using SLF4j with netty if possible. (#8258)
* Log the correct line-number when using SLF4j with netty if possible.

Motivation:

At the moment we do not log the correct line number in many cases as it will log the line number of the logger wrapper itself. Slf4j does have an extra interface that can be used to filter out these nad make it more usable with logging wrappers.

Modifications:

Detect if the returned logger implements LocationAwareLogger and if so make use of its extra methods to be able to log the correct origin of the log request.

Result:

Better logging when using slf4j.
2018-09-07 07:34:22 +02:00
Norman Maurer
052c2fbefe
Update to jmh 1.2.1 (#8270)
Motivation:

We should use the latest jmh version which also supports -prof dtraceasm on MacOS.

Modifications:

Update to latest jmh version.

Result:

Better benchmark / profiling support on MacOS.
2018-09-06 22:31:52 +02:00
Norman Maurer
5ff6b57940
PemPrivateKey.toPem(...) should throw IllegalArgumentException when P… (#8253)
* PemPrivateKey.toPem(...) should throw IllegalArgumentException when PrivateKey which does not support encoding is used.

Motivation:

At the moment when a PrivateKey is used that does not support encoding we throw a NPE when trying to convert the key. We should better throw an IllegalArgumentException with the details about what key we tried to encode.

Modifications:

- Check if PrivateKey.getEncoded() returns null and if so throw an IllegalArgumentException
- Add unit test.

Result:

Better handling of non-supported PrivateKey implementations.
2018-09-05 20:33:40 +02:00
Norman Maurer
02d559e6a4
Remove flags when running benchmarks. (#8262)
Motivation:

Some of the flags we used are not supported anymore on more recent JDK versions. We should just remove all of them and only keep what we really need. This may also reflect better what people use in production.

Modifications:

Remove some flags when running the benchmarks.

Result:

Benchmarks also run with JDK11.
2018-09-05 19:05:02 +02:00
Norman Maurer
8635d88d4d
Allow to generate a jmh uber jar to run benchmarks easily from cmdline with different arguments. (#8264)
Motivation:

It is sometimes useful to be able to run benchmarks easily from the commandline and passs different arguments / options here. We should support this.

Modifications:

Add the benchmark-jar profile which allows to generate such an "uber-jar" that can be used directly to run benchmarks as documented at http://openjdk.java.net/projects/code-tools/jmh/.

Result:

More flexible way to run benchmarks.
2018-09-05 18:28:35 +02:00
Norman Maurer
3c2dbdb5db
NioEventLoop should also use our special SelectionKeySet on Java9 and later. (#8260)
Motivation:

In Java8 and earlier we used reflection to replace the used key set if not otherwise told. This does not work on Java9 and later without special flags as its not possible to call setAccessible(true) on the Field anymore.

Modifications:

- Use Unsafe to instrument the Selector with out special set when sun.misc.Unsafe is present and we are using Java9+.

Result:

NIO transport produce less GC on Java9 and later as well.
2018-09-05 07:23:03 +02:00
Norman Maurer
ade60c11e1
PlatformDependent0 should be able to better detect if unaligned access is supported on java9 and later. (#8255)
Motivation:

In Java8 and earlier we used reflection to detect if unaligned access is supported. This fails in Java9 and later as we would need to change the accessible level of the method.
Lucky enough we can use Unsafe directly to read the content of the static field here.

Modifications:

Add special handling for detecting if unaligned access is supported on Java9 and later which does not fail due jigsaw.

Result:

Better and more correct detection on Java9 and later.
2018-09-05 07:22:16 +02:00
Norman Maurer
dc1b511fcf
Correctly reset offset when fail lazy because of too long frame. (#8257)
Motivation:

We need to reset the offset to 0 when we fail lazy because of a too long frame.

Modifications:

- Reset offset
- Add testcase

Result:

Fixes https://github.com/netty/netty/issues/8256.
2018-09-04 19:13:56 +02:00
Carl Mastrangelo
379a56ca49 Add an Epoll benchmark
Motivation:
Optimizing the Epoll channel needs an objective measure of how fast
it is.

Modification:
Add a simple, closed loop,  ping-pong benchmark.

Result:
Benchmark can be used to measure #7816

Initial numbers:

```
Result "io.netty.microbench.channel.epoll.EpollSocketChannelBenchmark.pingPong":
  22614.403 ±(99.9%) 797.263 ops/s [Average]
  (min, avg, max) = (21093.160, 22614.403, 24977.387), stdev = 918.130
  CI (99.9%): [21817.140, 23411.666] (assumes normal distribution)

Benchmark                              Mode  Cnt      Score     Error  Units
EpollSocketChannelBenchmark.pingPong  thrpt   20  22614.403 ± 797.263  ops/s
```
2018-09-04 10:15:15 +02:00
Francesco Nigro
c78be33443 Added configurable ByteBuf bounds checking (#7521)
Motivation:

The JVM isn't always able to hoist out/reduce bounds checking (due to ref counting operations etc etc) hence making it configurable could improve performances for most CPU intensive use cases.

Modifications:

Each AbstractByteBuf bounds check has been tested against a new static final configuration property similar to checkAccessible ie io.netty.buffer.bytebuf.checkBounds.

Result:

Any user could disable ByteBuf bounds checking in order to get extra performances.
2018-09-03 20:33:47 +02:00
Norman Maurer
3eec66a974
Do not fail on runtime when an older version of Log4J2 is on the classpath. (#8240)
Motivation:

At the moment we will just assume the correct version of log4j2 is used when we find it on the classpath. This may lead to an AbstractMethodError at runtime. We should not use log4j2 if the version is not correct.

Modifications:

Check on class loading if we can use Log4J2 or not.

Result:

Fixes #8217.
2018-09-03 18:07:53 +02:00
Norman Maurer
c74b3f3a3b Correctly implement SelectedSelectionKeySet.Iterator remove()
Motivation:

We need to implement remove() by ourselves to make it work on Java7 as otherwise it will throw an AbstractMethodError. This is a followup of c1a335446d.

Modifications:

Just implemented remove()

Result:

Works on Java7 as well.
2018-09-01 08:59:08 +02:00
Norman Maurer
187b1b8a55 Correctly implement SelectedSelectionKeySet.remove(...) / contains(...) again so it works with the NIO Selector.
Motivation:

c1a335446d reimplemented remove(...) and contains(...) in a way which made it not work anymore when used by the Selector.

Modifications:

Partly revert changes in c1a335446d.

Result:

Works again as expected
2018-09-01 08:43:50 +02:00
Norman Maurer
f4bafd4fe0
Correctly check if cipher is supported for each SslProvider before trying to run test and fix buffer leaks in test. (#8247)
Motivation:

5aaa16b24c introduced a testcase for specific ciphersuites and checked if these are supported by our native implementation before running it. Unfortunally this is not good enough as even on the JDK it may not be supported on various JDK versions (like Java7). Beside this the test leaked buffers.

Modifications:

- Correctly check if ciphersuite is supported on each SslProvider before trying to run test.
- Fix buffer leaks.

Result:

Testsuite pass again on Java7 and others when -Pleak is used.
2018-09-01 08:11:13 +02:00
Norman Maurer
c1a335446d
Correctly implement SelectedSelectionKeySet iterator(), contains(...) and remove(...) (#8244)
Motivation:

Our SelectedSelectionKeySet does not correctly implement various methods which can be done without any performance overhead.

Modifications:

Implement iterator(), contains(...) and remove(...)

Result:

Related to https://github.com/netty/netty/issues/8242.
2018-09-01 08:10:02 +02:00
Norman Maurer
9d8846cfce
Cleanup Log4J2Logger (#8245)
Motivation:

Log4J2Logger had some code-duplication with AbstractInternalLogger

Modifications:

Reuse AbstractInternaLogger.EXCEPTION_MESSAGE in Log4J2Logger and so remove some code-duplication

Result:

Less duplicated code.
2018-08-31 17:08:38 +02:00
Norman Maurer
e26666a7ea
Update to latest netty-tcnative (#8246)
Motivation:

We should use the latest netty-tcnative release which contains a fix to correctly support DH based ciphers when using openssl 1.1.x

Modifications:

Update to latest netty-tcnative which has the fix.

Result:

Correctly support DH ciphers in all cases. Fixes https://github.com/netty/netty/issues/8165.
2018-08-31 17:08:14 +02:00
Chi-Joung So
a9863f8128 Add headers to MqttMessage returned by MqttDecoder in case of DecoderException (#8219)
Motivation:
When the MqttDecoder decodes a message larger than the 'maxBytesInMessage' a DecoderException is thrown and a MqttMessage with just the failure cause is returned. Even if I can't handle the message, I might want to send an ACK so that I won't have to worry about it again.

Modification:
The DecoderException is thrown after the variableHeader is decoded. The fixed and variable headers are then added to the MqttMessage along with the failure cause.

Result:
The invalid MqttMessage will have headers if available.
2018-08-31 15:06:09 +02:00
Norman Maurer
a644563625
Add more debug informations when log SSL errors. (#8241)
Motivation:

ea626ef8c3 added more debug logging but we can even include a bit more.

Modifications:

Always log the error number as well.

Result:

More informations for debugging SSL errors.
2018-08-30 20:44:47 +02:00
Norman Maurer
38eee409c8
We should be able to use the ByteBuffer cleaner on java8 (and earlier… (#8234)
* We should be able to use the ByteBuffer cleaner on java8 (and earlier versions) even if sun.misc.Unsafe is not present.

Motivation:

At the moment we have a hard dependency on sun.misc.Unsafe to use the Cleaner on Java8 and earlier. This is not really needed as we can still use pure reflection if sun.misc.Unsafe is not present.

Modifications:

Refactor Cleaner6 to fallback to pure reflection if sun.misc.Unsafe is not present on system.

Result:

More timely releasing of direct memory on Java8 and earlier when sun.misc.Unsafe is not present.
2018-08-30 07:43:10 +02:00
Norman Maurer
4a5b61fc13
Fix log message about using non-direct buffers by default (#8235)
Motivation:

f77891cc17 changed slightly how we detect if we should prefer direct buffers or not but did miss to also take this into account when logging.

Modifications:

Fix branch for log message to reflect changes in f77891cc17.

Result:

Correct logging.
2018-08-30 06:57:12 +02:00
Norman Maurer
b73f785631
We should call the UnLoad methods when we detect an error during calling OnLoad (#8237)
Motivation:

We should ensure we call *UnLoad when we detect an error during calling *OnLoad and previous *OnLoad calls were succesfull.

Modifications:

Correctly call *UnLoad when needed.

Result:

More correct code and no leaks when an error happens during loading the native lib.
2018-08-30 06:56:42 +02:00
Norman Maurer
ea626ef8c3
Log more details when shutdown SSL because of an error. (#8236)
Motivation:

We should log a bit more details about why we shutdown the SSL.

Modifications:

Add the return value of SSL_get_error(...) as well in debug mode.

Result:

More logging to make it easier to understand why an SSL error happened.
2018-08-29 21:52:26 +02:00
Terence Yim
79706357c7 Fix race condition in the NonStickyEventExecutorGroup (#8232)
Motivation:

There was a race condition between the task submitter and task executor threads such that the last Runnable submitted may not get executed. 

Modifications:

The bug was fixed by checking the task queue and state in the task executor thread after it saw the task queue was empty.

Result:

Fixes #8230
2018-08-29 19:42:01 +02:00
Norman Maurer
54f565ac67
Allow to use native transports when sun.misc.Unsafe is not present on… (#8231)
* Allow to use native transports when sun.misc.Unsafe is not present on the system

Motivation:

We should be able to use the native transports (epoll / kqueue) even when sun.misc.Unsafe is not present on the system. This is especially important as Java11 will be released soon and does not allow access to it by default.

Modifications:

- Correctly disable usage of sun.misc.Unsafe when -PnoUnsafe is used while running the build
- Correctly increment metric when UnpooledDirectByteBuf is allocated. This was uncovered once -PnoUnsafe usage was fixed.
- Implement fallbacks in all our native transport code for when sun.misc.Unsafe is not present.

Result:

Fixes https://github.com/netty/netty/issues/8229.
2018-08-29 19:36:33 +02:00
Roger
5aaa16b24c An unit test to ensure that cipher suites don't break/disappear between releases. (#8225)
Motivation

Ensure classes of cipher suites continue working between releases. Adding just a DHE check for now as it caused #8165 but this test can be expaned to other suites.

Modifications

Adding an unit test that checks for the presence of a cipher suite.

Result

Prevent #8165 from happening in the future.
2018-08-29 14:14:26 +02:00
Norman Maurer
f77891cc17
We should prefer direct buffers if we can access the cleaner even if sun.misc.Unsafe is not present. (#8233)
Motivation:

We should prefer direct buffers whenever we can use the cleaner even if sun.misc.Unsafe is not present.

Modifications:

Correctly prefer direct buffers in all cases.

Result:

More correct code.
2018-08-29 08:21:07 +02:00
Norman Maurer
8679c5ef43
CleanerJava9 should be able to do its job even with a SecurityManager installed. (#8204)
Motivation:

CleanerJava9 currently fails whever a SecurityManager is installed. We should make use of AccessController.doPrivileged(...) so the user can give it the correct rights.

Modifications:

Use doPrivileged(...) when needed.

Result:

Fixes https://github.com/netty/netty/issues/8201.
2018-08-28 16:32:29 +02:00
zhaojigang
338ef96931 Recycler will produce npe error when multiple recycled at different thread
Motivation:

Recycler may produce a NPE when the same object is recycled multiple times from different threads.

Modifications:

- Check if the id has changed or if the Stack became null and if so throw an IllegalStateException
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/8220.
2018-08-27 08:58:40 +02:00
Norman Maurer
1bd9e662dd
Clarify deprecation docs a bit. (#8226)
Motivation:

It seems to sometimes confuse people what to do to replace setMaxMessagePerRead(...).

Modifications:

Add some more details to the javadocs about the correct replacement.

Result:

Related to https://github.com/netty/netty/issues/8214.
2018-08-25 08:14:38 +02:00
Norman Maurer
37a4f99f25
Use Java11+ea28 during build. (#8113)
Motivation:

We should ensure we use the latest Java11 EA during build to catch any regressions etc.

Modifications:

Update from ea19 to ea28.

Result:

Use latest Java11 release.
2018-08-25 07:26:56 +02:00
Norman Maurer
6888af6ba5
Allow to parse hosts file which is stored in a different encoding then the default system encoding. (#8211)
Motivation:

We should support to parse and read a hosts file which is stored in a different encoding then the system default. Beside this when we are on windows we should just try to parse it with multiple different charset before giving up as there is no real standard what charset to use.

Modifications:

- Add more method overloads to HostsFileParser that take a Charset.
- Try to parse with multiple Charsets in DefaultHostsFileEntriesResolver when windows is used.
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/8208.
2018-08-24 19:48:27 +02:00
root
a580dc7585 [maven-release-plugin] prepare for next development iteration 2018-08-24 06:36:33 +00:00
root
3fc789e83f [maven-release-plugin] prepare release netty-4.1.29.Final 2018-08-24 06:36:06 +00:00