Commit Graph

354 Commits

Author SHA1 Message Date
Norman Maurer
c9512480f4 [#3785] Correctly handle connection refused with native transport
Motivation:

Due a bug we not correctly handled connection refused errors and so failed the connect promise with the wrong exception.
Beside this we some times even triggered fireChannelActive() which is not correct.

Modifications:

- Add testcase
- correctly detect connect errors

Result:

Correct and consistent handling.
2015-05-27 13:38:38 +02:00
Norman Maurer
d1c46ca987 [maven-release-plugin] prepare for next development iteration 2015-05-07 11:33:47 -04:00
Norman Maurer
005d4a42fc [maven-release-plugin] prepare release netty-4.0.28.Final 2015-05-07 11:33:09 -04:00
Norman Maurer
0f4d3a981e Revert "[maven-release-plugin] prepare for next development iteration"
This reverts commit 3c10ffab5e.
2015-05-07 11:02:03 -04:00
Norman Maurer
3c10ffab5e [maven-release-plugin] prepare for next development iteration 2015-05-07 09:09:23 -04:00
Norman Maurer
4581c441c9 Allow rejection of remote initiated renegotiation
Motivation:

To prevent from DOS attacks it can be useful to disable remote initiated renegotiation.

Modifications:

Add new flag to OpenSslContext that can be used to disable it
Adding a testcase

Result:

Remote initiated renegotion requests can be disabled now.
2015-05-07 14:41:00 +02:00
Norman Maurer
93f08aa28b Fix race-condition in SocketSslEchoTest
Motivation:

Because we tried to grab the SSL renegotation future to early we could see test-failures.

Modifications:

Access the future at the correct time.

Result:

No more test-failures.
2015-05-06 06:25:15 +02:00
Trustin Lee
a8af0debcb Fix the flaky SocketSslEchoTest
Motivation:

The SSL peer who did not initiate renegotiation sometimes does not get
the notification for renegotition due to an unknown reason.

Modification:

Until the exact cause is understood, relax the assertions of the flaky
tests.

Result:

Build stability
2015-05-04 11:52:05 +09:00
Trustin Lee
3f5ce7c1d2 Reduce the memory requirement of heap dump compression
Motivation:

The current heap dump compression preset (9) requires way too much
memory (768 MiB at maximum for dictionary), resulting in OOME in many
cases.

Modifications:

- Use the default preset (6) which uses 8 MiB dictionary.
- Do not fail abruptly even when OOME has been raised.

Result:

More stable heap dump acquisition
2015-05-01 11:57:28 +09:00
Ben Evans
44d64ce039 Add support for SCTP 'unordered' flag.
Motivation:

Some SCTP applications require the SCTP unordered flag.
This flag was not exposed by Netty so applications were unable
to use it.

Modifications:

- Add unordered flag to SctpMessage.
- {Nio,Oio}SctpChannel pass unordered flag to MessageInfo on write.
- SctpOutboundByteStreamHandler may optionally request unordered
  delivery for all outbound messages.
- Added test case to SctpEchoTest using unordered flag.

Result:

Fixes #3698. New constructors and methods added to SctpMessage and
SctpOutboundByteStreamHandler, but changes are backward compatible.
2015-04-30 08:06:53 +02:00
Norman Maurer
3b1bf6348a Correctly test for non-auto-read correctness in testsuite
Motiviation:

Our tests for non-auto-read did actually not test this correctly as auto-read was never disabled on the Bootstrap and ServerBootstrap.

Modifications:

- Correctly disable auto-read on Bootstrap and ServerBootstrap
- Fix tests to call ChannelHandlerContext.read() once a Channel becomes active.

Result:

Correctly test that non-auto-read works.
2015-04-20 09:02:40 +02:00
Norman Maurer
f2fedbcdef [maven-release-plugin] prepare for next development iteration 2015-03-31 22:06:30 -04:00
Norman Maurer
054e7c5d17 [maven-release-plugin] prepare release netty-4.0.27.Final 2015-03-31 22:05:43 -04:00
Norman Maurer
37264bb72b [maven-release-plugin] prepare for next development iteration 2015-03-02 01:31:30 -05:00
Norman Maurer
0dbc96cffd [maven-release-plugin] prepare release netty-4.0.26.Final 2015-03-02 01:30:58 -05:00
Norman Maurer
e99d89c04d [maven-release-plugin] rollback the release of netty-4.0.26.Final 2015-02-28 21:28:06 +01:00
Norman Maurer
b86e2e6ac0 [maven-release-plugin] prepare release netty-4.0.26.Final 2015-02-28 13:55:01 -05:00
Norman Maurer
261a30d8af Allow to use WebSocketClientHandshaker and WebSocketServerHandshaker with HttpResponse / HttpRequest
Motivation:

To use WebSocketClientHandshaker / WebSocketServerHandshaker it's currently a requirement of having a HttpObjectAggregator in the ChannelPipeline. This is not a big deal when a user only wants to server WebSockets but is a limitation if the server serves WebSockets and normal HTTP traffic.

Modifications:

Allow to use WebSocketClientHandshaker and WebSocketServerHandshaker without HttpObjectAggregator in the ChannelPipeline.

Result:

More flexibility
2015-02-06 10:42:53 +01:00
Norman Maurer
b898bdda84 Add support for Unix Domain Sockets when using native epoll transport
Motivation:

Using Unix Domain Sockets can be very useful when communication should take place on the same host and has less overhead then using loopback. We should support this with the native epoll transport.

Modifications:

- Add support for Unix Domain Sockets.
- Adjust testsuite to be able to reuse tests.

Result:

Unix Domain Sockets are now support when using native epoll transport.
2015-02-04 15:34:13 +01:00
Trustin Lee
0e61aeb849 [maven-release-plugin] prepare for next development iteration 2014-12-31 20:58:44 +09:00
Trustin Lee
087db82e78 [maven-release-plugin] prepare release netty-4.0.25.Final 2014-12-31 20:58:33 +09:00
Trustin Lee
ab607ed0a6 Enable client-side OpenSSL in SocketSslEchoTest 2014-12-30 23:55:47 +09:00
Frederic Bregier
b886c056bc Fix big transfer and Write traffic shaping issues
Motivation:

Several issues were shown by various ticket (#2900 #2956).
Also use the improvement on writability user management from #3036.
And finally add a mixte handler, both for Global and Channels, with
the advantages of being uniquely created and using less memory and
less shaping.

Issue #2900

When a huge amount of data are written, the current behavior of the
TrafficShaping handler is to limit the delay to 15s, whatever the delay
the previous write has. This is wrong, and when a huge amount of writes
are done in a short time, the traffic is not correctly shapened.

Moreover, there is a high risk of OOM if one is not using in his/her own
handler for instance ChannelFuture.addListener() to handle the write
bufferisation in the TrafficShapingHandler.

This fix use the "user-defined writability flags" from #3036 to
allow the TrafficShapingHandlers to "user-defined" managed writability
directly, as for reading, thus using the default isWritable() and
channelWritabilityChanged().
This allows for instance HttpChunkedInput to be fully compatible.

The "bandwidth" compute on write is only on "acquired" write orders, not
on "real" write orders, which is wrong from statistic point of view.

Issue #2956

When using GlobalTrafficShaping, every write (and read) are
synchronized, thus leading to a drop of performance.
ChannelTrafficShaping is not touched by this issue since synchronized is
then correct (handler is per channel, so the synchronized).

Modifications:
The current write delay computation takes into account the previous
write delay and time to check is the 15s delay (maxTime) is really
exceeded or not (using last scheduled write time). The algorithm is
simplified and in the same time more accurate.

This proposal uses the #3036 improvement on user-defined writability
flags.

When the real write occurs, the statistics are update accordingly on a
new attribute (getRealWriteThroughput()).

To limit the synchronisations, all synchronized on
GlobalTrafficShapingHandler on submitWrite were removed. They are
replaced with a lock per channel (since synchronization is still needed
to prevent unordered write per channel), as in the sendAllValid method
for the very same reason.
Also all synchronized on TrafficCounter on read/writeTimeToWait() are
removed as they are unnecessary since already locked before by the
caller.
Still the creation and remove operations on lock per channel (PerChannel
object) are synchronized to prevent concurrency issue on this critical
part, but then limited.

Additionnal changes:
1) Use System.nanoTime() instead of System.currentTimeMillis() and
minimize calls
2) Remove / 10 ° 10 since no more sleep usage
3) Use nanoTime instead of currentTime such that time spend is computed,
not real time clock. Therefore the "now" relative time (nanoTime based)
is passed on all sub methods.
4) Take care of removal of the handler to force write all pending writes
and release read too
8) Review Javadoc to explicit:

- recommandations to take into account isWritable

- recommandations to provide reasonable message size according to
traffic shaping limit

- explicit "best effort" traffic shaping behavior when changing
configuration dynamically

Add a MixteGlobalChannelTrafficShapingHandler which allows to use only one
handler for mixing Global and Channel TSH. I enables to save more memory and
tries to optimize the traffic among various channels.

Result:
The traffic shaping is more stable, even with a huge number of writes in
short time by taking into consideration last scheduled write time.

The current implementation of TrafficShapingHandler using user-defined
writability flags and default isWritable() and
fireChannelWritabilityChanged works as expected.

The statistics are more valuable (asked write vs real write).

The Global TrafficShapingHandler should now have less "global"
synchronization, hoping to the minimum, but still per Channel as needed.

The GlobalChannel TrafficShapingHandler allows to have only one handler for all channels while still offering per channel in addition to global traffic shaping.

And finally maintain backward compatibility.
2014-12-29 15:36:02 +09:00
Norman Maurer
60f24ac2a1 Add OpenSslClientContext to allow creating SslEngine for client side
Motivation:

We only support openssl for server side at the moment but it would be also useful for client side.

Modification:

* Upgrade to new netty-tcnative snapshot to support client side openssl support
* Add OpenSslClientContext which can be used to create SslEngine for client side usage
* Factor out common logic between OpenSslClientContext and OpenSslServerContent into new abstract base class called OpenSslContext
* Correctly detect handshake failures as soon as possible
* Guard against segfault caused by multiple calls to destroyPools(). This can happen if OpenSslContext throws an exception in the constructor and the finalize() method is called later during GC

Result:

openssl can be used for client and servers now.
2014-12-22 19:37:48 +01:00
Trustin Lee
28553c9ee0 Do not use SLF4J/Logback API directly
Motivation:

TrafficShapingHandlerTest uses Logback API directly, which is
discouraged.  Also, it overrides the global default log level, which
silences the DEBUG messages from other tests.

Modifications:

Remove the direct use of Logback API

Result:

The tests executed after TrafficShapingHandlerTest logs their DEBUG
messages correctly.
2014-12-17 10:19:48 +09:00
Trustin Lee
99b9cda01d Fix checkstyle 2014-12-16 20:34:25 +09:00
Trustin Lee
9c4438e383 Log detailed information about renegotiation and traffic
Motivation:

We need more information to understand why SocketSslEchoTest fails
sporadically in the CI machine.

Modifications:

- Refactor SocketSslEchoTest so that it is easier to retrieve the
  information about renegotiation and the current progress

Result:

We will get more information when the test fails.
2014-12-16 17:46:29 +09:00
Trustin Lee
0baa5d5119 Compress the heap dump after tests are finished
Motivation:

Tests sometimes time out because it took too long to compress the
generated heap dump.

Modifications:

- Move the compression logic to a new method 'compressHeapDumps()'
- Call TestUtils.compressHeapDumps() at the end of the tests, so that
  the tests do not fail because of timeout

Result:

JUnit reports the real cause of the test failure instead of timeout
exception.
2014-12-16 16:16:29 +09:00
Trustin Lee
f86356f083 Generate non-test JAR for netty-testsuite
Motivation:

So far, we generated and deployed test JARs to Maven repositories. The
deployed JAR had the classifier 'test-jar'.  The test JAR is consumed by
transport-native-epoll as a test dependency.

The problem is, when netty-transport-native-epoll pulls the test JAR as
a dependency, that Maven resolves its transitive dependencies at
'compile' and 'runtime' scope only, which is incorrect.

I was bitten by this problem recently while trying to add a new
dependency to netty-testsuite.  Because I added a new dependency at the
'test' scope, the new dependency was not pulled transitively by
transport-native-epoll and caused an unexpected build failure.

- d6160208c3
- bf77bb4c3a

Modifications:

- Move all classes in netty-testsuite from src/test to src/main
- Update the 'compile' scope dependencies of netty-testsuite
- Override the test directory configuration properties of the surefire
  plugin
- Do not generate the test JAR anymore
- Update the dependency of netty-transport-native-epoll

Result:

It is less error-prone to add a new dependency to netty-testsuite.
2014-12-15 09:18:20 +09:00
Trustin Lee
80f788cea2 Fix build errors due to missing dependency 2014-12-14 21:30:57 +09:00
Trustin Lee
c718657cf2 Compress the heap dump generated by TestUtils.dump()
Motivation:

It takes too long to download the heap dump from the CI server.

Modifications:

Compress the heap dump as much as possible.

Result:

When heap dump is generated by certain test failure, the generated heap
dump file is about 3 times smaller than before, although the compression
time will increase the build time when the test fails.
2014-12-14 12:03:11 +09:00
Trustin Lee
d0e91e2f15 Add more assertions related with TLS renegotiation 2014-12-12 18:00:50 +09:00
Trustin Lee
9c4ba81b0b Test TLS renegotiation with explicit cipher suite change
Motivation:

So far, our TLS renegotiation test did not test changing cipher suite
during renegotiation explicitly.

Modifications:

- Switch the cipher suite during renegotiation

Result:

We are now sure the cipher suite change works.
2014-12-12 17:50:48 +09:00
Trustin Lee
5cb0ef6551 Add log messages when dump starts
.. to make it easier to find the right dump file for a test when there
are multiple dump files.
2014-12-12 11:54:14 +09:00
Trustin Lee
689d59b04b Make sure to notify handshake success even if SSLEngine is closed
Related:

e9685ea45a

Motivation:

SslHandler.unwrap() does not evaluate the handshake status of
SSLEngine.unwrap() when the status of SSLEngine.unwrap() is CLOSED.

It is not correct because the status does not reflect the state of the
handshake currently in progress, accoding to the API documentation of
SSLEngineResult.Status.

Also, sslCloseFuture can be notified earlier than handshake notification
because we call sslCloseFuture.trySuccess() before evaluating handshake
status.

Modifications:

- Notify sslCloseFuture after the unwrap loop is finished
- Add more assertions to SocketSslEchoTest

Result:

Potentially fix the regression caused by:
- e9685ea45a
2014-12-12 11:47:52 +09:00
Trustin Lee
7124ef88cb Generate heap and thread dump when some tests fail
Motivation:

We have a few sporadic test failures which are only easily reproduceable
in our CI machine.  To get more information about the failure, we need
heap and full thread dump at the moment of failure.

Modifications:

- Add TestUtils.dump() method to dump heap and threads
- Modify SocketGatheringWriteTest and SocketSslEchoTest to call
  TestUtils.dump() on failure

Result:

We get more information about the test failure.
2014-12-12 10:34:38 +09:00
Trustin Lee
e9685ea45a Add SslHandler.renegotiate()
Related: #3125

Motivation:

We did not expose a way to initiate TLS renegotiation and to get
notified when the renegotiation is done.

Modifications:

- Add SslHandler.renegotiate() so that a user can initiate TLS
  renegotiation and get the future that's notified on completion
- Make SslHandler.handshakeFuture() return the future for the most
  recent handshake so that a user can get the future of the last
  renegotiation
- Add the test for renegotiation to SocketSslEchoTest

Result:

Both client-initiated and server-initiated renegotiations are now
supported properly.
2014-12-11 18:06:16 +09:00
Norman Maurer
1914b77c71 [maven-release-plugin] prepare for next development iteration 2014-10-29 11:48:40 +01:00
Norman Maurer
c170e7df3f [maven-release-plugin] prepare release netty-4.0.24.Final 2014-10-29 11:47:19 +01:00
Trustin Lee
c13f72c5b6 Fix an infinite loop when writing a zero-length FileRegion
Related: #2964

Motivation:

Writing a zero-length FileRegion to an NIO channel will lead to an
infinite loop.

Modification:

- Do not write a zero-length FileRegion by protecting with proper 'if'.
- Update the testsuite

Result:

Another bug fixed
2014-10-17 16:06:57 +09:00
Trustin Lee
8416a47106 Make TestUtils.getFreePort() check both TCP and UDP
Motivation:

We see occational failures in the datagram tests saying 'address already
in use' when we attempt to bind on a port returned by
TestUtils.getFreePort().

It turns out that TestUtils.getFreePort() only checks if TCP port is
available.

Modifications:

Also check if UDP port is available, so that the datagram tests do not
fail because of the 'address already in use' error during a bind
attempt.

Result:

Less chance of datagram test failures
2014-10-17 15:05:01 +09:00
Norman Maurer
fcb80e34ca Fix buffer leak in DatagramUnicastTest caused by incorrect usage of CompositeByteBuf
Motivation:

Due incorrect usage of CompositeByteBuf a buffer leak was introduced.

Modifications:

Correctly handle tests with CompositeByteBuf.

Result:

No more buffer leaks
2014-09-10 11:40:53 +02:00
Norman Maurer
7867f986fa Add support for sendmmsg(...) and so allow to write multiple DatagramPackets with one syscall. Related to [#2719]
Motivation:

On linux with glibc >= 2.14 it is possible to send multiple DatagramPackets with one syscall. This can be a huge performance win and so we should support it in our native transport.

Modification:

- Add support for sendmmsg by reuse IovArray
- Factor out ThreadLocal support of IovArray to IovArrayThreadLocal for better separation as we use IovArray also without ThreadLocal in NativeDatagramPacketArray now
- Introduce NativeDatagramPacketArray which is used for sendmmsg(...)
- Implement sendmmsg(...) via jni
- Expand DatagramUnicastTest to test also sendmmsg(...)

Result:

Netty now automatically use sendmmsg(...) if it is supported and we have more then 1 DatagramPacket in the ChannelOutboundBuffer and flush() is called.
2014-09-09 09:43:07 +02:00
Norman Maurer
07876ebf68 Allow to write CompositeByteBuf directly via EpollDatagramChannel. Related to [#2719]
Motivation:

On linux it is possible to use the sendMsg(...) system call to write multiple buffers with one system call when using datagram/udp.

Modifications:

- Implement the needed changes and make use of sendMsg(...) if possible for max performance
- Add tests that test sending datagram packets with all kind of different ByteBuf implementations.

Result:

Performance improvement when using CompoisteByteBuf and EpollDatagramChannel.
2014-09-09 09:43:07 +02:00
Frédéric Brégier
b32ab860a9 Better fix for TrafficShapingHandlerTest
Motivation:
The test procedure is unstable when testing quick time (factor less or equal to 1). Changing to default 10ms in this case will force time to be correct and time to be checked only when factor is >= 2.

Modifications:
When factor is <= 1, minimalWaitBetween is 10ms

Result:
Hoping this version is finally stable.
2014-08-16 18:11:36 +02:00
Frédéric Brégier
b830959ff7 Better fix for TrafficShapingHandlerTest
Motivation:
It seems that in certain conditions, the write back from the server is so quick that the handler has no time to compute traffic shaping. So 10ms of wait before acknowledging is added in server side.

Modifications:
Add 10ms waiting before server ackonwledge the client.

Result:
The timing is now suppsed to be stable.
2014-08-16 10:55:38 +02:00
Trustin Lee
7710e7da44 [maven-release-plugin] prepare for next development iteration 2014-08-16 03:02:02 +09:00
Trustin Lee
208198c0cb [maven-release-plugin] prepare release netty-4.0.23.Final 2014-08-16 03:01:57 +09:00
fbregier
7327d681bb Fix for issue #2765 relative to unstable trafficshaping test procedure
Motivation:

The test procedure is unstable due to not enough precise timestamping
during the check.

Modifications:

Reducing the test cases and cibling "stable" test ("timestamp-able")
bring more stability to the tests.

Result:

Tests for TrafficShapingHandler seem more stable (whatever using JVM 6,
7 or 8).
2014-08-15 10:09:45 -07:00
Norman Maurer
f31c630c8c [#2769] Fix regression when writing different message types
Motivation:

Due a regression NioSocketChannel.doWrite(...) will throw a ClassCastException if you do something like:

channel.write(bytebuf);
channel.write(fileregion);
channel.flush();

Modifications:

Correctly handle writing of different message types by using the correct message count while loop over them.

Result:

No more ClassCastException
2014-08-15 11:55:48 +02:00