Commit Graph

316 Commits

Author SHA1 Message Date
Trustin Lee
d70d13091c Enable client-side OpenSSL in SocketSslEchoTest 2014-12-30 23:56:53 +09:00
Frederic Bregier
0d6563a845 Fix big transfer and Write traffic shaping issues
Motivation:

Several issues were shown by various ticket (#2900 #2956).
Also use the improvement on writability user management from #3036.
And finally add a mixte handler, both for Global and Channels, with
the advantages of being uniquely created and using less memory and
less shaping.

Issue #2900

When a huge amount of data are written, the current behavior of the
TrafficShaping handler is to limit the delay to 15s, whatever the delay
the previous write has. This is wrong, and when a huge amount of writes
are done in a short time, the traffic is not correctly shapened.

Moreover, there is a high risk of OOM if one is not using in his/her own
handler for instance ChannelFuture.addListener() to handle the write
bufferisation in the TrafficShapingHandler.

This fix use the "user-defined writability flags" from #3036 to
allow the TrafficShapingHandlers to "user-defined" managed writability
directly, as for reading, thus using the default isWritable() and
channelWritabilityChanged().
This allows for instance HttpChunkedInput to be fully compatible.

The "bandwidth" compute on write is only on "acquired" write orders, not
on "real" write orders, which is wrong from statistic point of view.

Issue #2956

When using GlobalTrafficShaping, every write (and read) are
synchronized, thus leading to a drop of performance.
ChannelTrafficShaping is not touched by this issue since synchronized is
then correct (handler is per channel, so the synchronized).

Modifications:
The current write delay computation takes into account the previous
write delay and time to check is the 15s delay (maxTime) is really
exceeded or not (using last scheduled write time). The algorithm is
simplified and in the same time more accurate.

This proposal uses the #3036 improvement on user-defined writability
flags.

When the real write occurs, the statistics are update accordingly on a
new attribute (getRealWriteThroughput()).

To limit the synchronisations, all synchronized on
GlobalTrafficShapingHandler on submitWrite were removed. They are
replaced with a lock per channel (since synchronization is still needed
to prevent unordered write per channel), as in the sendAllValid method
for the very same reason.
Also all synchronized on TrafficCounter on read/writeTimeToWait() are
removed as they are unnecessary since already locked before by the
caller.
Still the creation and remove operations on lock per channel (PerChannel
object) are synchronized to prevent concurrency issue on this critical
part, but then limited.

Additionnal changes:
1) Use System.nanoTime() instead of System.currentTimeMillis() and
minimize calls
2) Remove / 10 ° 10 since no more sleep usage
3) Use nanoTime instead of currentTime such that time spend is computed,
not real time clock. Therefore the "now" relative time (nanoTime based)
is passed on all sub methods.
4) Take care of removal of the handler to force write all pending writes
and release read too
8) Review Javadoc to explicit:

- recommandations to take into account isWritable

- recommandations to provide reasonable message size according to
traffic shaping limit

- explicit "best effort" traffic shaping behavior when changing
configuration dynamically

Add a MixteGlobalChannelTrafficShapingHandler which allows to use only one
handler for mixing Global and Channel TSH. I enables to save more memory and
tries to optimize the traffic among various channels.

Result:
The traffic shaping is more stable, even with a huge number of writes in
short time by taking into consideration last scheduled write time.

The current implementation of TrafficShapingHandler using user-defined
writability flags and default isWritable() and
fireChannelWritabilityChanged works as expected.

The statistics are more valuable (asked write vs real write).

The Global TrafficShapingHandler should now have less "global"
synchronization, hoping to the minimum, but still per Channel as needed.

The GlobalChannel TrafficShapingHandler allows to have only one handler for all channels while still offering per channel in addition to global traffic shaping.

And finally maintain backward compatibility.
2014-12-29 15:51:20 +09:00
Norman Maurer
6600ea3fac Add OpenSslClientContext to allow creating SslEngine for client side
Motivation:

We only support openssl for server side at the moment but it would be also useful for client side.

Modification:

* Upgrade to new netty-tcnative snapshot to support client side openssl support
* Add OpenSslClientContext which can be used to create SslEngine for client side usage
* Factor out common logic between OpenSslClientContext and OpenSslServerContent into new abstract base class called OpenSslContext
* Correctly detect handshake failures as soon as possible
* Guard against segfault caused by multiple calls to destroyPools(). This can happen if OpenSslContext throws an exception in the constructor and the finalize() method is called later during GC

Result:

openssl can be used for client and servers now.
2014-12-22 20:10:24 +01:00
Trustin Lee
7bb7b8742f Do not use SLF4J/Logback API directly
Motivation:

TrafficShapingHandlerTest uses Logback API directly, which is
discouraged.  Also, it overrides the global default log level, which
silences the DEBUG messages from other tests.

Modifications:

Remove the direct use of Logback API

Result:

The tests executed after TrafficShapingHandlerTest logs their DEBUG
messages correctly.
2014-12-17 10:20:13 +09:00
Trustin Lee
770126f707 Fix checkstyle 2014-12-16 20:34:01 +09:00
Trustin Lee
1ecb3fe88c Log detailed information about renegotiation and traffic
Motivation:

We need more information to understand why SocketSslEchoTest fails
sporadically in the CI machine.

Modifications:

- Refactor SocketSslEchoTest so that it is easier to retrieve the
  information about renegotiation and the current progress

Result:

We will get more information when the test fails.
2014-12-16 17:40:00 +09:00
Trustin Lee
0b9fff906e Compress the heap dump after tests are finished
Motivation:

Tests sometimes time out because it took too long to compress the
generated heap dump.

Modifications:

- Move the compression logic to a new method 'compressHeapDumps()'
- Call TestUtils.compressHeapDumps() at the end of the tests, so that
  the tests do not fail because of timeout

Result:

JUnit reports the real cause of the test failure instead of timeout
exception.
2014-12-16 16:16:03 +09:00
Trustin Lee
2668944e0b Generate non-test JAR for netty-testsuite
Motivation:

So far, we generated and deployed test JARs to Maven repositories. The
deployed JAR had the classifier 'test-jar'.  The test JAR is consumed by
transport-native-epoll as a test dependency.

The problem is, when netty-transport-native-epoll pulls the test JAR as
a dependency, that Maven resolves its transitive dependencies at
'compile' and 'runtime' scope only, which is incorrect.

I was bitten by this problem recently while trying to add a new
dependency to netty-testsuite.  Because I added a new dependency at the
'test' scope, the new dependency was not pulled transitively by
transport-native-epoll and caused an unexpected build failure.

- d6160208c3
- bf77bb4c3a

Modifications:

- Move all classes in netty-testsuite from src/test to src/main
- Update the 'compile' scope dependencies of netty-testsuite
- Override the test directory configuration properties of the surefire
  plugin
- Do not generate the test JAR anymore
- Update the dependency of netty-transport-native-epoll

Result:

It is less error-prone to add a new dependency to netty-testsuite.
2014-12-15 09:17:51 +09:00
Trustin Lee
bf77bb4c3a Fix build errors due to missing dependency 2014-12-14 21:29:26 +09:00
Trustin Lee
d6160208c3 Compress the heap dump generated by TestUtils.dump()
Motivation:

It takes too long to download the heap dump from the CI server.

Modifications:

Compress the heap dump as much as possible.

Result:

When heap dump is generated by certain test failure, the generated heap
dump file is about 3 times smaller than before, although the compression
time will increase the build time when the test fails.
2014-12-14 12:01:17 +09:00
Trustin Lee
9a0be053c4 Add more assertions related with TLS renegotiation 2014-12-12 18:01:36 +09:00
Trustin Lee
d838b19b6f Test TLS renegotiation with explicit cipher suite change
Motivation:

So far, our TLS renegotiation test did not test changing cipher suite
during renegotiation explicitly.

Modifications:

- Switch the cipher suite during renegotiation

Result:

We are now sure the cipher suite change works.
2014-12-12 17:43:23 +09:00
Trustin Lee
61eddd5b74 Add log messages when dump starts
.. to make it easier to find the right dump file for a test when there
are multiple dump files.
2014-12-12 11:55:34 +09:00
Trustin Lee
94a64c09cd Make sure to notify handshake success even if SSLEngine is closed
Related:

e9685ea45a

Motivation:

SslHandler.unwrap() does not evaluate the handshake status of
SSLEngine.unwrap() when the status of SSLEngine.unwrap() is CLOSED.

It is not correct because the status does not reflect the state of the
handshake currently in progress, accoding to the API documentation of
SSLEngineResult.Status.

Also, sslCloseFuture can be notified earlier than handshake notification
because we call sslCloseFuture.trySuccess() before evaluating handshake
status.

Modifications:

- Notify sslCloseFuture after the unwrap loop is finished
- Add more assertions to SocketSslEchoTest

Result:

Potentially fix the regression caused by:
- e9685ea45a
2014-12-12 11:55:34 +09:00
Trustin Lee
ed7cc5bee6 Generate heap and thread dump when some tests fail
Motivation:

We have a few sporadic test failures which are only easily reproduceable
in our CI machine.  To get more information about the failure, we need
heap and full thread dump at the moment of failure.

Modifications:

- Add TestUtils.dump() method to dump heap and threads
- Modify SocketGatheringWriteTest and SocketSslEchoTest to call
  TestUtils.dump() on failure

Result:

We get more information about the test failure.
2014-12-12 10:35:19 +09:00
Trustin Lee
d1612f67ad Add SslHandler.renegotiate()
Related: #3125

Motivation:

We did not expose a way to initiate TLS renegotiation and to get
notified when the renegotiation is done.

Modifications:

- Add SslHandler.renegotiate() so that a user can initiate TLS
  renegotiation and get the future that's notified on completion
- Make SslHandler.handshakeFuture() return the future for the most
  recent handshake so that a user can get the future of the last
  renegotiation
- Add the test for renegotiation to SocketSslEchoTest

Result:

Both client-initiated and server-initiated renegotiations are now
supported properly.
2014-12-10 18:47:53 +09:00
Trustin Lee
f793f395d6 Replace HttpHeaders.Names/Values with HttpHeaderNames/Values
Related: 4ce994dd4f

Motivation:

In 4.1, we were not able to change the type of the HTTP header name and
value constants from String to AsciiString due to backward compatibility
reasons.

Instead of breaking backward compatibility in 4.1, we introduced new
types called HttpHeaderNames and HttpHeaderValues which provides the
AsciiString version of the constants, and then deprecated
HttpHeaders.Names/Values.

We should make the same changes while deleting the deprecated classes
activaly.

Modifications:

- Remove HttpHeaders.Names/Values and RtspHeaders
- Add HttpHeaderNames/Values and RtspHeaderNames/Values
  - Make HttpHeaderValues.WEBSOCKET lowercased because it's actually
    lowercased in all WebSocket versions but the oldest one
- Do not use AsciiString.equalsIgnoreCase(CharSeq, CharSeq) if one of
  the parameters are AsciiString
  - Avoid using AsciiString.toString() repetitively
    - Change the parameter type of some methods from String to
      CharSequence

Result:

A user who upgraded from 4.0 to 4.1 first and removed the references to
the deprecated classes and methods can easily upgrade from 4.1 to 5.0.
2014-11-01 02:41:56 +09:00
Trustin Lee
0eb059bf58 Fix an infinite loop when writing a zero-length FileRegion
Related: #2964

Motivation:

Writing a zero-length FileRegion to an NIO channel will lead to an
infinite loop.

Modification:

- Do not write a zero-length FileRegion by protecting with proper 'if'.
- Update the testsuite

Result:

Another bug fixed
2014-10-17 16:04:37 +09:00
Trustin Lee
582f2e82ec Make TestUtils.getFreePort() check both TCP and UDP
Motivation:

We see occational failures in the datagram tests saying 'address already
in use' when we attempt to bind on a port returned by
TestUtils.getFreePort().

It turns out that TestUtils.getFreePort() only checks if TCP port is
available.

Modifications:

Also check if UDP port is available, so that the datagram tests do not
fail because of the 'address already in use' error during a bind
attempt.

Result:

Less chance of datagram test failures
2014-10-17 15:01:51 +09:00
Trustin Lee
fa248cecb5 Name resolver API and DNS-based name resolver
Motivation:

So far, we relied on the domain name resolution mechanism provided by
JDK.  It served its purpose very well, but had the following
shortcomings:

- Domain name resolution is performed in a blocking manner.
  This becomes a problem when a user has to connect to thousands of
  different hosts. e.g. web crawlers
- It is impossible to employ an alternative cache/retry policy.
  e.g. lower/upper bound in TTL, round-robin
- It is impossible to employ an alternative name resolution mechanism.
  e.g. Zookeeper-based name resolver

Modification:

- Add the resolver API in the new module: netty-resolver
- Implement the DNS-based resolver: netty-resolver-dns
  .. which uses netty-codec-dns
- Make ChannelFactory reusable because it's now used by
  io.netty.bootstrap, io.netty.resolver.dns, and potentially by other
  modules in the future
  - Move ChannelFactory from io.netty.bootstrap to io.netty.channel
  - Deprecate the old ChannelFactory
  - Add ReflectiveChannelFactory

Result:

It is trivial to resolve a large number of domain names asynchronously.
2014-10-16 17:10:36 +09:00
Jakob Buchgraber
b72a05edb4 Shutdown Executor on MultithreadEventLoopGroup shutdown. Fixes #2837
Motivation:

Currently the Executor created by (Nio|Epoll)EventLoopGroup is not correctly shutdown.
This might lead to resource shortages, due to resources not being freed asap.

Modifications:

If (Nio|Epoll)EventLoopGroup create their internal Executor via a constructor
provided `ExecutorServiceFactory` object or via
MultithreadEventLoopGroup.newDefaultExecutorService(...) the ExecutorService.shutdown()
method will be called after (Nio|Epoll)EventLoopGroup is shutdown.

ExecutorService.shutdown() will not be called if the Executor object was passed
to the (Nio|Epoll)EventLoopGroup (that is, it was instantiated outside of Netty).

Result:

Correctly release resources on (Nio|Epoll)EventLoopGroup shutdown.
2014-09-12 12:17:29 +09:00
Norman Maurer
3fd3cae8c3 Fix buffer leak in DatagramUnicastTest caused by incorrect usage of CompositeByteBuf
Motivation:

Due incorrect usage of CompositeByteBuf a buffer leak was introduced.

Modifications:

Correctly handle tests with CompositeByteBuf.

Result:

No more buffer leaks
2014-09-10 11:43:11 +02:00
Norman Maurer
d5b9f58f1f Add support for sendmmsg(...) and so allow to write multiple DatagramPackets with one syscall. Related to [#2719]
Motivation:

On linux with glibc >= 2.14 it is possible to send multiple DatagramPackets with one syscall. This can be a huge performance win and so we should support it in our native transport.

Modification:

- Add support for sendmmsg by reuse IovArray
- Factor out ThreadLocal support of IovArray to IovArrayThreadLocal for better separation as we use IovArray also without ThreadLocal in NativeDatagramPacketArray now
- Introduce NativeDatagramPacketArray which is used for sendmmsg(...)
- Implement sendmmsg(...) via jni
- Expand DatagramUnicastTest to test also sendmmsg(...)

Result:

Netty now automatically use sendmmsg(...) if it is supported and we have more then 1 DatagramPacket in the ChannelOutboundBuffer and flush() is called.
2014-09-09 10:03:17 +02:00
Norman Maurer
f1f14f524a Allow to write CompositeByteBuf directly via EpollDatagramChannel. Related to [#2719]
Motivation:

On linux it is possible to use the sendMsg(...) system call to write multiple buffers with one system call when using datagram/udp.

Modifications:

- Implement the needed changes and make use of sendMsg(...) if possible for max performance
- Add tests that test sending datagram packets with all kind of different ByteBuf implementations.

Result:

Performance improvement when using CompoisteByteBuf and EpollDatagramChannel.
2014-09-09 09:50:42 +02:00
Frédéric Brégier
5aa6bcde9e Better fix for TrafficShapingHandlerTest
Motivation:
The test procedure is unstable when testing quick time (factor less or equal to 1). Changing to default 10ms in this case will force time to be correct and time to be checked only when factor is >= 2.

Modifications:
When factor is <= 1, minimalWaitBetween is 10ms

Result:
Hoping this version is finally stable.
2014-08-16 18:08:43 +02:00
Frédéric Brégier
4c30067b58 Better fix for TrafficShapingHandlerTest
Motivation:
It seems that in certain conditions, the write back from the server is so quick that the handler has no time to compute traffic shaping. So 10ms of wait before acknowledging is added in server side.

Modifications:
Add 10ms waiting before server ackonwledge the client.

Result:
The timing is now suppsed to be stable.
2014-08-16 10:54:00 +02:00
Frédéric Brégier
4d20f03b6d Update TrafficShapingHandlerTest for Master
Motivation:
Test seems not yet stable (20ms instead of 40ms), so this small fix provides a "lightest" check.
If this is not passing again in master, one could deactivate this test by commenting the assertTrue in line 431.

Modification:
Divide by 2 the to be compared elapse time.

Result:
Locally this test is now passing.
2014-08-16 10:44:20 +02:00
Trustin Lee
138b366b00 Fix checkstyle 2014-08-15 11:09:42 -07:00
Trustin Lee
98dfc91471 Fix for issue #2765 relative to unstable trafficshaping test procedure
Motivation:

The test procedure is unstable due to not enough precise timestamping
during the check.

Modifications:

Reducing the test cases and cibling "stable" test ("timestamp-able")
bring more stability to the tests.

Result:

Tests for TrafficShapingHandler seem more stable (whatever using JVM 6,
7 or 8).
2014-08-15 10:21:11 -07:00
Norman Maurer
050e7ed933 [#2769] Fix regression when writing different message types
Motivation:

Due a regression NioSocketChannel.doWrite(...) will throw a ClassCastException if you do something like:

channel.write(bytebuf);
channel.write(fileregion);
channel.flush();

Modifications:

Correctly handle writing of different message types by using the correct message count while loop over them.

Result:

No more ClassCastException
2014-08-15 11:56:27 +02:00
Trustin Lee
2ded0dd6fe Fix data corruption in FileRegion transfer with epoll transport
Related issue: #2764

Motivation:

EpollSocketChannel.writeFileRegion() does not handle the case where the
position of a FileRegion is non-zero properly.

Modifications:

- Improve SocketFileRegionTest so that it tests the cases where the file
  transfer begins from the middle of the file
- Add another jlong parameter named 'base_off' so that we can take the
  position of a FileRegion into account

Result:

Improved test passes. Corruption is gone.
2014-08-13 16:58:35 -07:00
Frédéric Brégier
8b23ff1171 Update TrafficShapingTest for Master
Motivation:
Master test seems not ok as in 4.1 in some tests so relax the internal time check but keeping the outside check.

Modifications:
Comment the assertTrue regarding the internal time test and extending to 15s the tests using Exec.

Result:
No more false negative tests (until Master is stable enough to recheck the condition of validation).
2014-08-13 19:20:42 +02:00
fbregier
ba58f949e8 [#2721] Improve Traffic Shaping handler
Motivation:
Currently Traffic Shaping is using 1 timer only and could lead to
"partial" wrong bandwidth computation when "short" time occurs between
adding used bytes and when the TrafficCounter updates itself and finally
when the traffic is computed.
Indeed, the TrafficCounter is updated every x delay and it is at the
same time saved into "lastXxxxBytes" and set to 0. Therefore, when one
request the counter, it first updates the TrafficCounter with the added
used bytes. If this value is set just before the TrafficCounter is
updated, then the bandwidth computation will use the TrafficCounter with
a "0" value (this value being reset once the delay occurs). Therefore,
the traffic shaping computation is wrong in rare cases.

Secondly the traffic shapping should avoid if possible the "Timeout"
effect by not stopping reading or writing more than a maxTime, this
maxTime being less than the TimeOut limit.

Thirdly the traffic shapping in read had an issue since the readOp
was not set but should, turning in no read blocking from socket
point of view.

Modifications:
The TrafficCounter has 2 new methods that compute the time to wait
according to read or write) using in priority the currentXxxxBytes (as
before), but could used (if current is at 0) the lastXxxxxBytes, and
therefore having more chance to take into account the real traffic.

Moreover the Handler could change the default "max time to wait", which
is by default set to half of "standard" Time Out (30s:2 = 15s).

Finally we add the setAutoRead(boolean) accordingly to the situation,
as proposed in #2696 (this pull request is in error for unknown reason).

Result:
The Traffic Shaping is better take into account (no 0 value when it
shouldn't) and it tries to not block traffic more than Time Out event.

Moreover the read is really stopped from socket point of view.

This version is similar to #2388 and #2450.
This version is for Master, and includes the #2696 pull request
to ease the merge process.

Including also #2748

The test minimizes time check by reducing to 66ms steps (55s).

Conflicts:
	handler/src/main/java/io/netty/handler/traffic/AbstractTrafficShapingHandler.java
2014-08-13 02:21:07 +02:00
Jakob Buchgraber
f8bee2e94c Make Nio/EpollEventLoop run on a ForkJoinPool
Related issue: #2250

Motivation:

Prior to this commit, Netty's non blocking EventLoops
were each assigned a fixed thread by which all of the
EventLoop's I/O and handler logic would be performed.

While this is a fine approach for most users of Netty,
some advanced users require more flexibility in
scheduling the EventLoops.

Modifications:

Remove all direct usages of threads in MultithreadEventExecutorGroup,
SingleThreadEventExecutor et al., and introduce an Executor
abstraction instead.

The way to think about this change is, that each
iteration of an eventloop is now a task that gets scheduled
in a ForkJoinPool.

While the ForkJoinPool is the default, one also has the
ability to plug in his/her own Executor (aka thread pool)
into a EventLoop(Group).

Result:

Netty hands off thread management to a ForkJoinPool by default.
Users can also provide their own thread pool implementation and
get some control over scheduling Netty's EventLoops
2014-08-11 15:28:46 -07:00
Norman Maurer
578068c7e6 [#2647] Respect IOV_MAX when call writev in native transport
Motivation:

epoll transport fails on gathering write of more then 1024 buffers. As linux supports max. 1024 iov entries when calling writev(...) the epoll transport throws an exception.

Thanks again to @blucas to provide me with a reproducer and so helped me to understand what the issue is.

Modifications:

Make sure we break down the writes if to many buffers are uses for gathering writes.

Result:

Gathering writes work with any number of buffers
2014-07-09 12:18:48 +02:00
Trustin Lee
330404da07 Fix most inspector warnings
Motivation:

It's good to minimize potentially broken windows.

Modifications:

Fix most inspector warnings from our profile

Result:

Cleaner code
2014-07-02 19:04:11 +09:00
Trustin Lee
24e17d1832 Remove 'get' prefix from all HTTP/SPDY messages
Motivation:

Persuit for the consistency in method naming

Modifications:

- Remove the 'get' prefix from all HTTP/SPDY message classes
- Fix some inspector warnings

Result:

Consistency
2014-06-24 18:41:27 +09:00
Trustin Lee
de2872f7f7 Introduce TextHeaders and AsciiString
Motivation:

We have quite a bit of code duplication between HTTP/1, HTTP/2, SPDY,
and STOMP codec, because they all have a notion of 'headers', which is a
multimap of string names and values.

Modifications:

- Add TextHeaders and its default implementation
- Add AsciiString to replace HttpHeaderEntity
  - Borrowed some portion from Apache Harmony's java.lang.String.
- Reimplement HttpHeaders, SpdyHeaders, and StompHeaders using
  TextHeaders
- Add AsciiHeadersEncoder to reuse the encoding a TextHeaders
  - Used a dedicated encoder for HTTP headers for better performance
    though
- Remove shortcut methods in SpdyHeaders
- Remove shortcut methods in SpdyHttpHeaders
- Replace SpdyHeaders.getStatus() with HttpResponseStatus.parseLine()

Result:

- Removed quite a bit of code duplication in the header implementations.
- Slightly better performance thanks to improved header validation and
  hash code calculation
2014-06-14 17:14:30 +09:00
Trustin Lee
2e98566916 Introduce MessageAggregator and DecoderResultProvider
Motivation:

We have different message aggregator implementations for different
protocols, but they are very similar with each other.  They all stems
from HttpObjectAggregator.  If we provide an abstract class that provide
generic message aggregation functionality, we will remove their code
duplication.

Modifications:

- Add MessageAggregator which provides generic message aggregation
- Reimplement all existing aggregators using MessageAggregator
- Add DecoderResultProvider interface and extend it wherever possible so
  that MessageAggregator respects the state of the decoded message

Result:

Less code duplication
2014-06-05 16:39:59 +09:00
Trustin Lee
a72230061d Add an OpenSslEngine and the universal API for enabling SSL
Motivation:

Some users already use an SSLEngine implementation in finagle-native. It
wraps OpenSSL to get higher SSL performance.  However, to take advantage
of it, finagle-native must be compiled manually, and it means we cannot
pull it in as a dependency and thus we cannot test our SslHandler
against the OpenSSL-based SSLEngine.  For an instance, we had #2216.

Because the construction procedures of JDK SSLEngine and OpenSslEngine
are very different from each other, we also need to provide a universal
way to enable SSL in a Netty application.

Modifications:

- Pull netty-tcnative in as an optional dependency.
  http://netty.io/wiki/forked-tomcat-native.html
- Backport NativeLibraryLoader from 4.0
- Move OpenSSL-based SSLEngine implementation into our code base.
  - Copied from finagle-native; originally written by @jpinner et al.
  - Overall cleanup by @trustin.
- Run all SslHandler tests with both default SSLEngine and OpenSslEngine
- Add a unified API for creating an SSL context
  - SslContext allows you to create a new SSLEngine or a new SslHandler
    with your PKCS#8 key and X.509 certificate chain.
  - Add JdkSslContext and its subclasses
  - Add OpenSslServerContext
- Add ApplicationProtocolSelector to ensure the future support for NPN
  (NextProtoNego) and ALPN (Application Layer Protocol Negotiation) on
  the client-side.
- Add SimpleTrustManagerFactory to help a user write a
  TrustManagerFactory easily, which should be useful for those who need
  to write an alternative verification mechanism. For example, we can
  use it to implement an unsafe TrustManagerFactory that accepts
  self-signed certificates for testing purposes.
- Add InsecureTrustManagerFactory and FingerprintTrustManager for quick
  and dirty testing
- Add SelfSignedCertificate class which generates a self-signed X.509
  certificate very easily.
- Update all our examples to use SslContext.newClient/ServerContext()
- SslHandler now logs the chosen cipher suite when handshake is
  finished.

Result:

- Cleaner unified API for configuring an SSL client and an SSL server
  regardless of its internal implementation.
- When native libraries are available, OpenSSL-based SSLEngine
  implementation is selected automatically to take advantage of its
  performance benefit.
- Examples take advantage of this modification and thus are cleaner.
2014-05-18 02:33:26 +09:00
Norman Maurer
3508960781 Fix compile error 2014-04-30 15:58:33 +02:00
Norman Maurer
e920dc57db Correctly write pending data after ssl handshake completes. Related to [#2437]
Motivation:
When writing data from a server before the ssl handshake completes may not be written at all to the remote peer
if nothing else is written after the handshake was done.

Modification:
Correctly try to write pending data after the handshake was complete

Result:
Correctly write out all pending data
2014-04-30 14:23:34 +02:00
Trustin Lee
d2614cfc01 Synchronized between 4.1 and master
Motivation:

4 and 5 were diverged long time ago and we recently reverted some of the
early commits in master.  We must make sure 4.1 and master are not very
different now.

Modification:

Fix found differences

Result:

4.1 and master got closer.
2014-04-25 00:36:01 +09:00
Norman Maurer
48f2e705d9 Resurrect channel deregistration and constructor changes
Motivation:

Due to the complexity of handling deregistration and re-registration of
a channel, we previously decided to remove the deregister() operation
completely to simplify our code.  However, we realized that it shouldn't
be that complicated to implement it during our discussion about making
I/O scheduling more flexible and more customizable [1], and thus the
removal of deregistration and re-registration is unnecessary now.

Modification:

- Revert commit c149f4bcc0
- Revert commit e743a27e75
- Make some additional adjustments

Result:

- deregister(), fireChannelUnregistered(), and channelRegistered() were
  added back..
- Channel constructors do not require an EventLoop anymore.

[1] https://github.com/netty/netty/issues/2250
2014-04-24 20:54:50 +09:00
Norman Maurer
541abb8515 [#2375] [#2404] Fix bug in respecting ChannelConfig.setAutoRead(false) and also fix Channel.read() for OIO
Motivation:
At the moment ChanneConfig.setAutoRead(false) only is guaranteer to not have an extra channelRead(...) triggered when used from within the channelRead(...) or channelReadComplete(...) method. This is not the correct behaviour as it should also work from other methods that are triggered from within the EventLoop. For example a valid use case is to have it called from within a ChannelFutureListener, which currently not work as expected.

Beside this there is another bug which is kind of related. Currently Channel.read() will not work as expected for OIO as we will stop try to read even if nothing could be read there after one read operation on the socket (when the SO_TIMEOUT kicks in).

Modifications:
Implement the logic the right way for the NIO/OIO/SCTP and native transport, specific to the transport implementation. Also correctly handle Channel.read() for OIO transport by trigger a new read if SO_TIMEOUT was catched.

Result:
It is now also possible to use ChannelConfig.setAutoRead(false) from other methods that are called from within the EventLoop and have direct effect.
2014-04-17 08:19:20 +02:00
Jeff Pinner
350c8ef42a SPDY: refactor frame codec implementation
Motivation:

Currently, the SPDY frame encoding and decoding code is based upon
the ChannelHandler abstraction. This requires maintaining multiple
versions for 3.x and 4.x (and possibly 5.x moving forward).

Modifications:

The SPDY frame encoding and decoding code is separated from the
ChannelHandler and SpdyFrame abstractions. Also test coverage is
improved.

Result:

SpdyFrameCodec now implements the ChannelHandler abstraction and is
responsible for creating and handling SpdyFrame objects.
2014-04-15 18:42:08 +02:00
Trustin Lee
6490c2bb8b Perform cross-tests between NIO and epoll transport
Motivation:

The epoll testsuite tests the epoll transport only against itself (i.e. epoll x epoll only).  We should test the epoll transport also against the well-tested NIO transport, too.

Modifications:

- Make SocketTestPermutation extensible and reusable so that the epoll testsuite can take advantage of it.
- Rename EpollTestUtils to EpollSocketTestPermutation and make it extend SocketTestPermutation.
- Overall clean-up of SocketTestPermutation
  - Use Arrays.asList() for simplicity
  - Add combo() method to remove code duplication

Result:

The epoll transport is now also tested against the NIO transport.  SocketTestPermutation got cleaner.
2014-03-17 10:49:31 +09:00
Trustin Lee
92f0a0310d Set timeout for SocketSslEchoTest
Motivation:

We are seeing EpollSocketSslEchoTest does not finish itself while its I/O thread is busy.  Jenkins should have terminated them when the global build timeout reaches, but Jenkins seems to fail to do so.  What's more interesting is that Jenkins will start another job before the EpollSocketSslEchoTest is terminated, and Linux starts to oom-kill them, impacting the uptime of the CI service.

Modifications:

- Set timeout for all test cases in SocketSslEchoTest so that all SSL tests terminate themselves when they take too long.
- Fix a bug where the epoll testsuite uses non-daemon threads which can potentially prevent JVM from quitting.
- (Cleanup) Separate boss group and worker group just like we do for NIO/OIO transport testsuite.

Result:

Potentially more stable CI machine.
2014-03-17 09:31:27 +09:00
Norman Maurer
6de58f5ad1 Ensure the HttpResponseEncoder is always placed before the HttpObjectAggregator. Part of [#2219] 2014-03-05 06:58:54 +01:00
Norman Maurer
bea810707c [#2254] Fix regression in handling autoRead and Channel.read()
This regression was introduced by e0b39159657c9eb711047bc32367537c4870d467
2014-02-21 09:46:28 +01:00