Commit Graph

7688 Commits

Author SHA1 Message Date
Scott Mitchell
a7496ed83d FlowControlHandlerTest synchronization issues
Motivation:
2b65258568 only partially addressed the synchronization issues that are present in FlowControlHandlerTest. A few tests are attempting to validate state changes made across an EventLoop thread and the JUnit thread but are not properly synchronized.

Modifications:
- Ensure that conditions which verify expectations set in another thread have synchronization gates to ensure the event has actually occurred.
- Remove the message counter verification in favor of using individual CountDownLatch objects

Result:
FLowControlHanderTest has less race conditions which may lead to test failures.
2016-06-13 14:13:40 +02:00
Sina Tadayon
eb1d932466 Support WebSocket data chunked transfer
Motivation:
Support fetches data chunk by chunk for use with WebSocket chunked transfers.

Modifications:
Create a WebSocketChunkedInput.java that add to io.netty.handler.codec.http.websocketx package

Result:
The WebSocket transfers/fetches data chunk by chunk.
2016-06-13 08:55:07 +02:00
Dmitry Spikhalskiy
428c61673b Logs in invokeExceptionCaught have been made consistent and full
Motivation:

In case of exception in invokeExceptionCaught() only original exception passed to invokeExceptionCaught() will be logged on any log level.
+ AbstractChannelHandlerContext and CombinedChannelDuplexHandler log different exceptions.

Modifications:

Fix inconsistent logging code and add ability to see both stacktraces on DEBUG level.

Result:

Both handlers log now both original exception and thrown from invokeExceptionCaught. To see full stacktrace of exception thrown from invokeExceptionCaught DEBUG log level must be enabled.
2016-06-11 20:11:11 +02:00
Guido Medina
b921f80057 Make JCtools available (provided scope) for tests and optional OSGI, issue #5383 2016-06-11 07:15:48 +02:00
Roger Kapsi
cc580e3ba1 Let OpenSslContext take pre-encoded pkcs#8 private key/cert bytes
Motivation

OpenSslContext is expecting Java's PrivateKey and X509Certificate objects as input
(for JdkSslContext API compatibility reasons) but doesn't really use them beyond
turning them into PEM/PKCS#8 strings.

This conversion can be entirely skipped if the user can pass in private keys and
certificates in a format that Netty's OpenSSL code can digest.

Modifications

Two new classes have been added that act as a wrapper around the pre-encoded byte[]
and also retain API compatibility to JdkSslContext.

Result

It's possible to pass PEM encoded bytes straight into OpenSSL without having to
parse them (e.g. File to Java's PrivateKey) and then encode them (i.e. PrivateKey
into PEM/PKCS#8).

File pemPrivateKeyFile;
byte[] pemBytes = readBytes(pemPrivateKeyFile);
PemPrivateKey pemPrivateKey = PemPrivateKey.valueOf(pemBytes);

SslContextBuilder.forServer(pemPrivateKey)
    .sslProvider(SslProvider.OPENSSL)
2016-06-10 18:07:40 +02:00
Guido Medina
c3abb9146e Use shaded dependency on JCTools instead of copy and paste
Motivation:
JCTools supports both non-unsafe, unsafe versions of queues and JDK6 which allows us to shade the library in netty-common allowing it to stay "zero dependency".

Modifications:
- Remove copy paste JCTools code and shade the library (dependencies that are shaded should be removed from the <dependencies> section of the generated POM).
- Remove usage of OneTimeTask and remove it all together.

Result:
Less code to maintain and easier to update JCTools and less GC pressure as the queue implementation nt creates so much garbage
2016-06-10 13:19:45 +02:00
Norman Maurer
398efb1f71 Ensure valid message sequence if channel is closed before receive headers.
Motivation:

When the channel is closed while we still decode the headers we currently not preserve correct message sequence. In this case we should generate an invalid message with a current cause.

Modifications:

Create an invalid message with a PrematureChannelClosureException as cause when the channel is closed while we decode the headers.

Result:

Correct message sequence preserved and correct DecoderResult if the channel is closed while decode headers.
2016-06-09 22:42:46 +02:00
Norman Maurer
88dbd96376 [#5372] Ensure OpenSslClientContext / OpenSslServerContext can be garbage collected
Motivation:

OpenSslClientContext / OpenSslServerContext can never be garbage collected as both are part of a reference to a callback that is stored as global reference in jni code.

Modifications:

Ensure the callbacks are static and so not hold the reference.

Result:

No more leak due not collectable OpenSslClientContext / OpenSslServerContext
2016-06-09 22:37:13 +02:00
Scott Mitchell
52bbfd3310 epoll_wait timeout reset when called multiple times
Motivation:
epoll_wait accepts a timeout argument which will specify the maximum amount of time the epoll_wait will wait for an event to occur. If the epoll_wait method returns for any reason that is not fatal (e.g. EINTR) the original timeout value is re-used. This does not honor the timeout interface contract and can lead to unbounded time in epoll_wait.

Modifications:
- The time taken by epoll_wait should be decremented before calling epoll_wait again, and if the remaining time is exhausted we should return 0 according to the epoll_wait interface docs http://man7.org/linux/man-pages/man2/epoll_wait.2.html
- link librt which is needed for some platforms to use clock_gettime

Result:
epoll_wait will wait for at most timeout ms according to the epoll_wait interface contract.
2016-06-09 11:14:40 -07:00
Scott Mitchell
56a2f64665 Clarify Future.removeListener[s] javaDocs
Motivation:
The javaDocs for Future.removeListener do not clarify that only the first occurrence of the listener is guaranteed to be removed.

Modifications:
- Clarify the javaDocs for Future.removeListener[s] so it is known that the only the first occurrence of the listener will be removed.

Result:
Fixes https://github.com/netty/netty/issues/5351
2016-06-08 15:48:26 -07:00
Scott Mitchell
783567420f OpenSslEngine encrypt more data per wrap call
Motivation:
OpenSslEngine.wrap will only encrypt at most 1 buffer per call. We may be able to encrypt multiple buffers per call.

Modifications:
- OpensslEngine.wrap should continue encrypting data until there is an error, no more data, or until the destination buffer would be overflowed.

Result:
More encryption is done per OpenSslEngine.wrap call
2016-06-08 12:23:19 -07:00
Jon Chambers
829be86223 Use a default resolver with bootstrap.resolver(null).
Motivation:

`Bootstrap` has a notion of a default resolver group, but it's hidden from the public. To allow callers to reset a `Bootstrap` instance's resolver group, we could either make `DEFAULT_RESOLVER` public, or we could allow callers to pass `null` as an argument to `Bootstrap#resolver(AddressResolverGroup<?>)`. This pull request does the latter.

Modifications:

- Allow `Bootstrap#resolver(AddressResolverGroup<?>)` to accept `null` as an argument

Result:

Callers may pass `null` to `Bootstrap#resolver(AddressResolverGroup<?>)` to cause the `Bootstrap` instance to use its default resolver group.
2016-06-08 06:57:55 +02:00
Norman Maurer
4dec7f11b7 [maven-release-plugin] prepare for next development iteration 2016-06-07 18:52:34 +02:00
Norman Maurer
cf670fab75 [maven-release-plugin] prepare release netty-4.1.1.Final 2016-06-07 18:52:22 +02:00
Scott Mitchell
9e2c400f89 OpenSslEngine writePlaintextData WANT_READ with no data in BIO buffer
Motivation:
CVE-2016-4970

OpenSslEngine.wrap calls SSL_write which may return SSL_ERROR_WANT_READ, and if in this condition there is nothing to read from the BIO the OpenSslEngine and SslHandler will enter an infinite loop.

Modifications:
- Use the error code provided by OpenSSL and go back to the EventLoop selector to detect if the socket is closed

Result:
OpenSslEngine correctly handles the return codes from OpenSSL and does not enter an infinite loop.
2016-06-07 08:59:13 -07:00
Yuri Schimke
a14eda7db0 typo: Skelton 2016-06-06 16:28:21 -07:00
Scott Mitchell
79f2e3604e HTTP/2 close only send GO_AWAY if one has not already been sent
Motivation:
Http2ConnectionHandler will always send a GO_AWAY when the channel is closed. This may cause problems if the user is attempting to control when GO_AWAY is sent and the content of the GO_AWAY.

Modifications:
- When the channel is closed Http2ConnectionHandler should only send a GO_AWAY if one has not already been sent

Result:
The user has more control over when GO_AWAY is sent
Fixes https://github.com/netty/netty/issues/5307
2016-06-06 11:18:30 -07:00
Norman Maurer
b461c9d54c Allow to specify a custom EventExecutorChooserFactory. Related to [#1230]
Motivation:

Sometimes it may be benefitially for an user to specify a custom algorithm when choose the next EventExecutor/EventLoop.

Modifications:

Allow to specify a custom EventExecutorChooseFactory that allows to customize algorithm.

Result:

More flexible api.
2016-06-06 11:04:56 +02:00
Norman Maurer
7137d22994 Ensure the correct wrapped buffer is released in AbstractPooledDerivedByteBuf.deallocate()
Motivation:

We need to first store a reference to the wrapped buffer before recycle the AbstractPooledDerivedByteBuf instance. This is needed as otherwise it is possible that the same AbstractPooledDerivedByteBuf is again obtained and init(...) is called before we actually have a chance to call release(). This leads to call release() on the wrong buffer.

Modifications:

Store a reference to the wrapped buffer before call recycle and call release on the previous stored reference.

Result:

Always release the correct wrapped buffer when deallocate the AbstractPooledDerivedByteBuf.
2016-06-06 09:49:42 +02:00
Norman Maurer
ee5969edfd Use higher maxQueriesPerResolve and make exception message more clear.
Motivation:

We use a default of 3 for maxQueriesPerResolve when using the DnsNameResolverBuilder, which is too low if you want to resolve a hostname that uses a lot of CNAME records.

Modifications:

- Use higher default (16)
- Make exception message more clear why it failed.

Result:

Be able to resolve more domains by default and be able to better trouble shoot why a resolver failed.
2016-06-06 09:22:11 +02:00
Norman Maurer
584fbac6ed [#3419] Only use SecureRandom to generate initialSeed if requested
Motivation:

We tried to always use SecureRandom to generate the initialSeed for our ThreadLocalRandom, this can sometimes give warnings under normal usage. We should better not use SecureRandom as default (just as the implementation in jsr166y does) and only try if the user specified -Djava.util.secureRandomSeed=true .

Modifications:

Only try to use SecureRandom when -Djava.util.secureRandomSeed=true is used.

Result:

Less likely to see entropy warnings.
2016-06-06 09:09:29 +02:00
Norman Maurer
3a7dcde320 [#5224] Allow to use Unsafe.reallocateMemory(...) in UnpooledUnsafeNoCleanerDirectByteBuf.
Motivation:

If the user uses unsafe direct buffers with no cleaner we can use Unsafe.reallocateMemory(...) as optimization when we need to expand the buffer.

Modifications:

Use Unsafe.relocateMemory(...) in UnpooledUnsafeNoCleanerDirectByteBuf.

Result:

Less expensive expanding of buffers.
2016-06-04 19:21:41 +02:00
Norman Maurer
9bd94ea021 Make DnsAddressResolverGroup easier to extend
Motivation:

DnsAddressResolverGroup allows to override the newResolver(...) method to change the settings used by the user. We should better let the user override another method and always apply the InflightNameResolver.

Modifications:

- Mark newResolver(...) method as deprecated, we will make it private soon.
- Add newNameResolver(...) method that user can override.

Result:

Easier to extend DnsAddressResolverGroup
2016-06-04 17:40:45 +02:00
Norman Maurer
dc816e9807 Ensure we null out cumulation buffer before fire through the pipeline in handlerRemoved(...)
Motivation:

We should ensure we null out the cumulation buffer before we fire it through the pipleine in handlerRemoved(...) as in theory it could be possible that another method is triggered as result of the fireChannelRead(...) or fireChannelReadComplete() that will try to access the cumulation.

Modifications:

Null out cumulation buffer early in handlerRemoved(...)

Result:

No possible to access the cumulation buffer that was already handed over.
2016-06-04 09:18:07 +02:00
Norman Maurer
e847ac0443 Fix possible deadlock in DefaultChannelPipeline.destroyDown(...)
Motivation:

We need to ensure we not hold a lock while executor callHandlerRemoved(...) as this may lead to a deadlock if handlerRemoved(...) will call another method in DEfaultChannelPipeline from another thread that will need to obtain the lock as well and wait for the result.

Modifications:

Release the lock before call handlerRemoved0(...).

Result:

No more deadlock possible
2016-06-04 09:13:54 +02:00
Norman Maurer
f8b306f61c [#5313] Correctly catch errors during bootstrap.
Motivation:

We not correctly catched errors during resolving in bootstrap and so may not have notified the future correctly.

Modifications:

Move code into try / catch block and try to fail the promise.

Result:

Promise is always notified
2016-06-04 09:12:54 +02:00
Norman Maurer
0b8a647f4e Fix harmless typo in native code
Motivation:

We used transfered in native code which is not correct spelling. It should be transferred.

Modifications:

Fix typo.

Result:

Less typos in source code.
2016-06-04 09:10:40 +02:00
Scott Mitchell
f7cf00cb5f OpenSslEngine remove unecessary rejectRemoteInitiatedRenegation call
Motivation:
OpenSslEngine calls rejectRemoteInitiatedRenegation in a scenario where the number of handshakes has not been observed to change. The number of handshakes has only been observed to change after readPlaintextData is called.

Modifications:
- Remove the call to rejectRemoteInitiatedRenegation before calls to readPlaintextData

Result:
Less code.
2016-06-03 13:02:01 -07:00
Norman Maurer
6fb5b14ef5 [#5308] Ensure InetSocketAddressResolver.close() will close the wrapped NameResolver.
Motivation:

InetSocketAddressResolver.close() must call close() on the wrapped NameResolver.

Modifications:

Correctly call close() on wrapped NameResolver and added test.

Result:

close() is correctly propergated to the wrapped resolver.
2016-06-03 21:41:22 +02:00
Norman Maurer
3d29bcfc8d Allow to create Unsafe ByteBuf implementations that not use a Cleaner to clean the native memory.
Motivation:

Using the Cleaner to release the native memory has a few drawbacks:

- Cleaner.clean() uses static synchronized internally which means it can be a performance bottleneck
- It put more load on the GC

Modifications:

Add new buffer implementations that can be enabled with a system flag as optimizations. In this case no Cleaner is used at all and the user must ensure everything is always released.

Result:

Less performance impact by direct buffers when need to be allocated and released.
2016-06-03 21:20:10 +02:00
Scott Mitchell
f6ad9df8ac DefaultPromise StackOverflowError protection updates
Modifications:
DefaultPromise provides a ThreadLocal queue to protect against StackOverflowError because of executors which may immediately execute runnables instead of queue them (i.e. ImmediateEventExecutor). However this may be better addressed by fixing these executors to protect against StackOverflowError instead of just fixing for a single use case. Also the most commonly used executors already provide the desired behavior and don't need the additional overhead of a ThreadLocal queue in DefaultPromise.

Modifications:
- Remove ThreadLocal queue from DefaultPromise
- Change ImmediateEventExecutor so it maintains a queue of runnables if reentrant condition occurs

Result:
DefaultPromise StackOverflowError code is simpler, and ImmediateEventExecutor protects against StackOverflowError.
2016-06-02 09:22:47 -07:00
buchgr
af7b0a04a0 Fix DefaultHeaders.toString() for keys with multiple values.
Motivation:

For example,

DefaultHttp2Headers headers = new DefaultHttp2Headers();
headers.add("key1", "value1");
headers.add("key1", "value2");
headers.add("key1", "value3");
headers.add("key2", "value4");

produces:

DefaultHttp2Headers[key1: value1key1: value2key1: value3, key2: value4]

while correctly it should be

DefaultHttp2Headers[key1: value1, key1: value2, key1: value3, key2: value4]

Modifications:

Change the toString() method to produce the beforementioned output.

Result:

toString() format is correct also for keys with multiple values.
2016-06-02 17:24:51 +02:00
Scott Mitchell
e200e4811b SocketRstTest fails due to exception message check
Motivation:
For lack of a better way the SocketRstTest inspects the content of the exception message to check if a RST occurred. However on windows the exception message is different than on other Unix based platforms and the assertion statement fails.

Modifications:
- Hack another string check in the unit test

Result:
SocketRstTest passes on windows
Fixes https://github.com/netty/netty/issues/5335
2016-06-02 08:15:43 +02:00
Alex Petrov
bbed330468 Fix the possible reference leak in Recycler
Motivation:

Under very unlikely (however possible) circumstances, Recycler may leak
references. This happens _only_ when the object was already recycled
at least once (which means it's got written to the stack) and then
taken out again, and never returned.

The "never returned" part may be the fault of the user (forgotten
`finally` clause) or the situation when Recycler drops the possibly
youngest item itself.

Modifications:

Nullify the item taken from the stack.

Result:

Reference is cleaned up. If the object is lost, it will be a subject for
GC. The rest of Stack / Recycler functionality remains unaffected.
2016-06-01 06:47:43 +02:00
Norman Maurer
1dfcfc17fa Allow to change link capacity via system property
Motivation:

Sometimes people may want to trade GC with memory overhead. For this it can be useful to allow to change the capacity of the array that is hold in the Link that is used by the Recycler internally.

Modifications:

Introduce a new system property , io.netty.recycler.linkCapacity which allows to change the capcity.

Result:

More flexible configuration of netty.
2016-05-31 14:05:23 +02:00
Norman Maurer
c43b424f56 Add missing null check that was missed in 844976a0a2 2016-05-31 14:05:03 +02:00
Norman Maurer
844976a0a2 Ensure the same ByteBufAllocator is used in the EmbeddedChannel when compress / decompress. Related to [#5294]
Motivation:

The user may specify to use a different allocator then the default. In this case we need to ensure it is shared when creating the EmbeddedChannel inside of a ChannelHandler

Modifications:

Use the config of the "original" Channel in the EmbeddedChannel and so share the same allocator etc.

Result:

Same type of buffers are used.
2016-05-31 09:08:33 +02:00
Milos Fabian
4c186c4c41 Make netty-transport-native-epoll-*-linux-x86_64.jar working in OSGi
Motivation:

Currenlty, netty-transport-native-epoll-*-linux-x86_64.jar is not packed as OSGi bundle
and thus not working in OSGi environment.

Modifications:

In netty-transport-native-epoll's pom.xml added configuration
to attach manifest to the jar with a native library.
In netty-common's pom.xml added configuration instruction (DynamicImport-Package)
to maven bnd plugin to make sure the native code is loaded from
netty-transport-native-epoll bundle.

Result:

The netty-transport-native-epoll-*-linux-x86_64.jar is a bundle (MANIFEST.MF attached)
and the inluced native library can be successfuly loaded in OSGi environment.
Fixing #5119
2016-05-30 22:04:01 +02:00
Norman Maurer
db6b72da19 Add optimized version of setZero(...) / writeZero(...) for Unsafe*ByteBuf implementations
Motivation:

Unsafe offers a method to set memory to a specific value. This can be used to implement an optimized version of setZero(...) and writeZero(...)

Modifications:

Add implementation for all Unsafe*ByteBuf implementations.

Result:

Faster setZero(...) and writeZero(...)
2016-05-30 15:10:15 +02:00
Norman Maurer
339b512e70 Fix small race in DefaultChannelPipeline introduced by a729e0fcd9
Motivation:

There is a small race while adding handlers to the pipeline because callHandlerAddedForAllHandlers() may not be run when the user calls add* but the Channel is already registered.

Modifications:

Ensure we always delay handlerAdded(..) / handlerRemoved(...) until callHandlerAddedForAllHandlers() was called.

Result:

No more race on pipeline modifications possible.
2016-05-30 15:05:48 +02:00
Norman Maurer
dcd93e3be0 Remove volatile where not needed.
Motivation:

We can remove the volatile keyword from the cached Runnables as at worse these will just be re-created.

Modifications:

Remove volatile.

Result:

Less overhead.
2016-05-30 07:32:45 +02:00
floragunn
528b83e277 Set the session id context properly to make client authentication work with open ssl provider.
Motivation:

When netty is used with open ssl provider and client authentication the following errors can occur:
error:140D9115:SSL routines:ssl_get_prev_session:session id context uninitialized
error:140A1175:SSL routines:ssl_bytes_to_cipher_list:inappropriate fallback
error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Modifications:

Set the session id context in OpenSslServerContext so that sessions which use client authentication
which are cached have the same context id value.

Result:

Client authentication now works with open ssl provider.
2016-05-28 21:29:40 +02:00
Norman Maurer
86f53083e7 [#5297] Ensure calling NioEventLoop.pendingTasks() and EpollEventLoop.pendingTasks() will not produce livelock
Motivation:

SingleThreadEventExecutor.pendingTasks() will call taskQueue.size() to get the number of pending tasks in the queue. This is not safe when using MpscLinkedQueue as size() is only allowed to be called by a single consumer.

Modifications:

Ensure size() is only called from the EventLoop.

Result:

No more livelock possible when call pendingTasks, no matter from which thread it is done.
2016-05-28 20:04:07 +02:00
Xiaoyan Lin
3b72ee0fd0 Add includeScope to not include test dependencies for maven-dependency-plugin
Motivation:

maven-dependency-plugin will include dependencies of all scopes by default. We should limit the scope to avoid pulling test dependencies. See #5317.

Modifications:

Add includeScope to avoid pulling test dependencies.

Result:

netty-all doesn't include codes from netty-build
2016-05-27 23:00:09 -07:00
Norman Maurer
6ca49d1336 [maven-release-plugin] prepare for next development iteration 2016-05-25 19:16:44 +02:00
Norman Maurer
446b38db52 [maven-release-plugin] prepare release netty-4.1.0.Final 2016-05-25 19:14:15 +02:00
nmittler
79a06eaede Fix NPE in Http2ConnectionHandler.onHttpServerUpgrade
Motivation:

Performing a server upgrade with a new initial flow control window will cause an NPE in the DefaultHttp2RemoteFlowController. This is due to the fact that the monitor does not check whether or not the channel is writable.

Modifications:

Added a check for channel writability before calling `writePendingBytes`. Also fixed a unit test that was supposed to be testing this :).

Result:

Fixes #5301
2016-05-25 09:13:06 -07:00
Scott Mitchell
b3c56d5f69 DefaultPromise StackOverflowError protection
Motivation:
f2ed3e6ce8 removed the previous mechanism for StackOverflowError because it didn't work in all cases (i.e. ImmediateExecutor). However if a chain of listeners which complete other promises is formed there is still a possibility of a StackOverflowError.

Modifications:
- Use a ThreadLocal to save any DefaultPromises which could not be notified due to the stack being too large. After the first DefaultPromise on the stack completes notification this ThreadLocal should be used to notify any DefaultPromises which have not yet been notified.

Result:
DefaultPromise has StackOverflowError protection that works with all EventExecutor types.
2016-05-24 14:59:27 -07:00
Scott Mitchell
f5d58e2e1a DefaultPromise concurrency bug
Motivation:
If the executor changes while listeners are added and notification of listeners is being done then listeners can be notified out of order and concurrently. We should ensure that only one executor is used at any given time to notify listeners and ensure the listeners are notified in FIFO order.

Modifications:
- Move the notifyingListeners member variable from DefaultPromise into the synchronized block to prevent concurrent notification of listeners and preserve FIFO notification order

Result:
If the executor is changed for a DefaultPromise the listener notification order should be FIFO.
2016-05-24 11:46:43 -07:00
Norman Maurer
2618c2a649 [#5239] Allow adding handlers to pipeline with null name.
Motivation:

While doing 8fe3c83e4c I made a change which disallowed using null as name for handlers in the pipeline (this generated a new name before).

Modifications:

Revert to old behaviour and adding test case.

Result:

Allow null name again
2016-05-24 06:22:44 +02:00