Commit Graph

6536 Commits

Author SHA1 Message Date
Norman Maurer
528415bc29 Update javadocs to highlight that derived buffers will not increment the reference count.
Motivation:

We not explain the derived buffers will not retain the parent buffer.

Modifications:

Add docs.

Result:

Correctly document behaviour
2015-06-06 19:54:03 +02:00
Norman Maurer
1913fcd5f4 [#3848] Respect EPOLLERR event
Motivation:

Some glibc/kernel versions will trigger an EPOLLERR event to notify
about failed connect and not an EPOLLOUT. Also EPOLLERR may be triggered
when a connection is broke.

Modification:

React on EPOLLERR like if an EPOLLOUT / EPOLLIN was received, this will work in
all cases as we handle errors in EPOLLOUT / EPOLLIN anyway.

Result:

Correctly detect errors.
2015-06-06 10:34:36 +02:00
Trustin Lee
263e6979a6 Replace SpdyOrHttpChooser and Http2OrHttpChooser with ApplicationProtocolNegotiationHandler
Motivation:

SpdyOrHttpChooser and Http2OrHttpChooser duplicate fair amount code with each other.

Modification:

- Replace SpdyOrHttpChooser and Http2OrHttpChooser with ApplicationProtocolNegotiationHandler
- Add ApplicationProtocolNames to define the known application-level protocol names
- Deprecate SpdyOrHttpChooser

Result:

- Less code duplication
- A user can perform dynamic pipeline configuration that follows ALPN/NPN for any protocols.
2015-06-05 14:51:21 +09:00
Trustin Lee
67e02dad0a Improve the API design of Http2OrHttpChooser and SpdyOrHttpChooser
Related: #3641 and #3813

Motivation:

When setting up an HTTP/1 or HTTP/2 (or SPDY) pipeline, a user usually
ends up with adding arbitrary set of handlers.

Http2OrHttpChooser and SpdyOrHttpChooser have two abstract methods
(create*Handler()) that expect a user to return a single handler, and
also have add*Handlers() methods that add the handler returned by
create*Handler() to the pipeline as well as the pre-defined set of
handlers.

The problem is, some users (read: I) don't need all of them or the
user wants to add more than one handler. For example, take a look at
io.netty.example.http2.tiles.Http2OrHttpHandler, which works around
this issue by overriding addHttp2Handlers() and making
createHttp2RequestHandler() a no-op.

Modifications:

- Replace add*Handlers() and create*Handler() with configure*()
- Rename getProtocol() to selectProtocol() to make what it does clear
- Provide the default implementation of selectProtocol()
- Remove SelectedProtocol.UNKNOWN and use null instead, because
  'UNKNOWN' is not a protocol
- Proper exception handling in the *OrHttpChooser so that the
  exception is logged and the connection is closed when failed to
  select a protocol
- Make SpdyClient example always use SSL. It was always using SSL
  anyway.
- Implement SslHandshakeCompletionEvent.toString() for debuggability
- Remove an orphaned class: JettyNpnSslSession
- Add SslHandler.applicationProtocol() to get the name of the
  application protocol
  - SSLSession.getProtocol() now returns transport-layer protocol name
    only, so that it conforms to its contract.

Result:

- *OrHttpChooser have better API.
- *OrHttpChooser handle protocol selection failure properly.
- SSLSession.getProtocol() now conforms to its contract.
- SpdyClient example works with SpdyServer example out of the box
2015-06-05 14:51:20 +09:00
Norman Maurer
9e125d8456 Expose SSL_CTX and SSL pointers
Motivation:

For advanced use-cases it an be helpful to be able to directly access the SSL_CTX and SSL pointers of the underlying openssl objects. This for example allows to register custom C callbacks.

Modifications:

- Expose the SSL_CTX and SSL pointers
- Cleanup the shutdown code

Result:

It's now possible to obtain the c pointes and set native callbacks.
2015-06-05 07:24:57 +02:00
Norman Maurer
2c479b77c1 Backport codec-haproxy
Motivation:

The codec-haproxy is very useful and standalone. So it should be very safe to backport it and make it usable by 4.0 users.

Modifications:

Backport codec-haproxy.

Result:

codec-haproxy is now included in 4.0.
2015-06-04 19:26:50 +02:00
Jean-Rémi Desjardins
d95e2c18bc Fix incoherence in WebSocket example
Motivation:

The logic in the current websocket example is confusing and misleading

Modifications:

Remove occurrences of "http" and "https" and replace them with "ws" and "wss"

Result:

The example code is now coherent and is easier to understand for a new user.
2015-06-04 19:24:23 +02:00
Norman Maurer
5d4e34b021 [#3837] Null out ByteBuffer[] array once done
Motivation:

the ByteBuffer[] that we keep in the ThreadLocal are never nulled out which can lead to have ByteBuffer instances sit there forever.
This is even a bigger problem if nioBuffer() of ByteBuffer returns a new ByteBuffer that can not be destroyed by ByteBuffer.release().

Modifications:

Null out ByteBuffer array after processing.

Result:

No more dangling references after done.
2015-06-04 12:33:13 +02:00
Trustin Lee
a1afaf4c35 More meaningful assertion failure message
We see sporadic failure in EpollSocketConnectionAttemptTest. Generate
more useful failure message using hamcrest to know more.
2015-06-04 12:08:49 +09:00
Scott Mitchell
9484687e13 Linux EPOLL Channel Configuration test unsupported options
Motivation:
The unit tests should not fail due to using a channel option which is not supported by the underlying kernel.

Modifications:
- Ignore RuntimeExceptions which are thrown by JNI code when setsockopt or getsockopt fails.

Result:
Unit tests pass if socket option is not supported by kernel.
2015-06-02 12:55:04 -07:00
Scott Mitchell
779e923302 TCP_NOTSENT_LOWAT older kernel compatibility
Motiviation:
TCP_NOTSENT_LOWAT is only supported in linux kernel 3.12 or newer. The addition of this socket option prevents older kernels from building.

Modifications:
- Conditionally define TCP_NOTSENT_LOWAT if it is not defined

Result:
Kernels older than 3.12 can still compile the EPOLL module.
2015-06-02 12:19:32 -07:00
Jean-Rémi Desjardins
1f3e6be32b Fix typo 2015-06-02 13:02:10 +02:00
Scott Mitchell
a17e1c89b6 TCP_NOTSENT_LOWAT socket option support
Motiviation:
Linux provides the TCP_NOTSENT_LOWAT socket option. This can be used to control how much unsent data is queued in the tcp kernel buffers. This can be important when application level protocols (SPDY, HTTP/2) have their own priority mechanism and don't want data queued in the kernel.

Modifications:
- The epoll module will have an additional socket option TCP_NOTSENT_LOWAT
- There will be JNI methods to control the underlying linux socket option mechanism

Result:
Linux EPOLL module exposes the TCP_NOTSENT_LOWAT socket option.
2015-06-01 13:36:28 -07:00
Trustin Lee
3adc6566d4 Fix sporadic failures in DatagramUnicastTest
Motivation:

DatagramUnitcastTest sometimes fails with BindException for an unknown reason.

Modifications:

Retry up to 3 times with a new free port when bind() fails with BindException

Result:

More build stability
2015-06-01 17:31:14 +09:00
Trustin Lee
add630c957 Fix sporadic assertion failure in SingleThreadEventLoopTest
Motivation:

SingleThreadEventLoopTest.testScheduleTaskAtFixedRate() fails often due to:

- too little tolerance
- incorrect assertion (it compares only with the previous timestamp)

Modifications:

- Increase the timestamp difference tolerance from 10ms to 20ms
- Improve the timestamp assertion so that the comparison is performed against the first recorded timestamp
- Misc: Fix broken Javadoc tag

Result:

More build stability
2015-06-01 14:44:49 +09:00
Trustin Lee
684bcaa07e Fix sporadic assertion failure in SocketSslEchoTest
Motivation:

SocketSslEchoTest.testSslEcho() has a race condition where a renegotiation future can be done before:

    assertThat(renegoFuture.isDone(), is(false));

Modifications:

Remove the offending assertion.

Result:

More build stability
2015-06-01 13:56:01 +09:00
Clebert Suconic
ea0f84e72d fixing small leak on exception on the transport-epoll-native allocation
Motivation:

the JNI function ThrowNew won't release any allocated memory.
The method exceptionMessage is allocating a new string concatenating 2 constant strings
What is creating a small leak in case of these exceptions are happening.

Modifications:

Added new methods that will use exceptionMessage and free resources accordingly.
I am also removing the inline definition on these methods as they could be reused by
other added modules (e.g. libaio which should be coming soon)

Result:

No more leaks in case of failures.
2015-05-31 09:36:27 +02:00
Norman Maurer
76d51b514a Mention correct order in SimplechannelPool javadocs 2015-05-29 20:56:40 +02:00
Trustin Lee
1dac4e44ec Remove the verbose:gc flag from the build
Motivation:

When a faulty never-ending test keeps producing a lot of garbage doing
nothing but generating CPU load, our CI fails to detect the stalled
build, because it determines the 'inactivity time' from console
activity and GC keeps producing console output.

Modifications:

Remove the -verbose:gc flag from pom.xml

Result:

Stalled builds are terminated by our CI server.
2015-05-29 10:47:55 +09:00
Norman Maurer
7e328fd1c3 Fix regression introduced by f765053ae7 by use Entry after it is recycled 2015-05-27 16:56:05 +02:00
Norman Maurer
f765053ae7 Let PoolThreadCache work even if allocation and deallocation Thread are different
Motivation:

PoolThreadCache did only cache allocations if the allocation and deallocation Thread were the same. This is not optimal as often people write from differen thread then the actual EventLoop thread.

Modification:

- Add MpscArrayQueue which was forked from jctools and lightly modified.
- Use MpscArrayQueue for caches and always add buffer back to the cache that belongs to the allocation thread.

Result:

ThreadPoolCache is now also usable and so gives performance improvements when allocation and deallocation thread are different.

Performance when using same thread for allocation and deallocation is noticable worse then before.
2015-05-27 14:35:22 +02:00
Norman Maurer
7706d2309c Reduce calls to System.nanoTime() and object creation in IdleStateHandler. Related to [#3808]
Motivation:

Calling System.nanoTime() for each channelRead(...) is very expensive. See [#3808] for more detailed description.
Also we always do extra work for each write and read even if read or write idle states should not be handled.

Modifications:

- Move System.nanoTime() call to channelReadComplete(...).
- Reuse ChannelFutureListener for writes
- Only add ChannelFutureListener to writes if write and all idle states should be handled.
- Only call System.nanoTime() for reads if idle state events for read and all states should be handled.

Result:

Less overhead when using the IdleStateHandler.
2015-05-27 13:55:01 +02:00
Norman Maurer
bb230579f8 Do not try to init TrustManagerFactory if trustCertChainFile is null.
Motivation:

We called TrustManagerFactory.init(...) even when the trustCertChainFile is null. This could lead to exceptions during the handshake.

Modifications:

Correctly only call TurstManagerFactory.init() if trustCertcChainFail is not null.

Result:

Correct behavior.
2015-05-27 13:45:34 +02:00
Norman Maurer
c9512480f4 [#3785] Correctly handle connection refused with native transport
Motivation:

Due a bug we not correctly handled connection refused errors and so failed the connect promise with the wrong exception.
Beside this we some times even triggered fireChannelActive() which is not correct.

Modifications:

- Add testcase
- correctly detect connect errors

Result:

Correct and consistent handling.
2015-05-27 13:38:38 +02:00
Norman Maurer
f18990a8a5 [#3654] Synchronize on PoolSubpage head when allocate / free PoolSubpages
Motivation:

Currently we hold a lock on the PoolArena when we allocate / free PoolSubpages, which is wasteful as this also affects "normal" allocations. The same is true vice-verse.

Modifications:

Ensure we synchronize on the head of the PoolSubPages pool. This is done per size and so it is possible to concurrently allocate / deallocate PoolSubPages with different sizes, and also normal allocations.

Result:

Less condition and so faster allocation/deallocation.

Before this commit:
xxx:~/wrk $ ./wrk -H 'Connection: keep-alive' -d 120 -c 256 -t 16 -s scripts/pipeline-many.lua  http://xxx:8080/plaintext
Running 2m test @ http://xxx:8080/plaintext
  16 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    17.61ms   29.52ms 689.73ms   97.27%
    Req/Sec   278.93k    41.97k  351.04k    84.83%
  530527460 requests in 2.00m, 71.64GB read
Requests/sec: 4422226.13
Transfer/sec:    611.52MB

After this commit:
xxx:~/wrk $ ./wrk -H 'Connection: keep-alive' -d 120 -c 256 -t 16 -s scripts/pipeline-many.lua  http://xxx:8080/plaintext
Running 2m test @ http://xxx:8080/plaintext
  16 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    15.85ms   24.50ms 681.61ms   97.42%
    Req/Sec   287.14k    38.39k  360.33k    85.88%
  547902773 requests in 2.00m, 73.99GB read
Requests/sec: 4567066.11
Transfer/sec:    631.55MB

This is reproducable every time.
2015-05-27 10:32:52 +02:00
Norman Maurer
7e80e1bf97 [#3654] No need to hold lock while destroy a chunk
Motiviation:

At the moment we sometimes hold the lock on the PoolArena during destroy a PoolChunk. This is not needed.

Modification:

- Ensure we not hold the lock during destroy a PoolChunk
- Move all synchronized usage in PoolArena
- Cleanup

Result:

Less condition.
2015-05-27 09:47:34 +02:00
Scott Mitchell
08cb650b64 Consistent use of SSLHandshakeException for ALPN
Motiviation:
The OpenSSL engine uses SSLHandshakeException in the event of failures that occur during the handshake process. The alpn-boot project's getSSLException will also map the no_application_protocol to a SSLHandshakeException exception. We should be consistent and use SSLHandshakeException for handshake failure events.

Modifications:
-Update JdkAlpnSslEngine to propagate an SSLHandshakeException in the event of a failure.

Result:
Consistent usage of SSLHandshakeException during a handshake failure event.
2015-05-26 16:10:43 -07:00
Roelof Naude
41b0080fcc Support empty http responses when using compression
Motivation:

Found a bug in that netty would generate a 20 byte body when returing a response
to an HTTP HEAD. the 20 bytes seems to be related to the compression footer.

RFC2616, section 9.4 states that responses to an HTTP HEAD MUST not return a message
body in the response.

Netty's own client implementation expected an empty response. The extra bytes lead to a
2nd response with an error decoder result:
java.lang.IllegalArgumentException: invalid version format: 14

Modifications:

Track the HTTP request method. When processing the response we determine if the response
is passthru unnchanged. This decision now takes into account the request method and passthru
responses related to HTTP HEAD requests.

Result:

Netty's http client works and better RFC conformance.
2015-05-26 10:06:25 +02:00
Norman Maurer
a80f0d69a7 [#3805] Fix incorrect javadoc in PlatformDependent 2015-05-25 21:42:21 +02:00
johnou
0decf2f647 Allow writing with void promise if IdleStateHandler is configured in pipeline.
Motivation:

Allow writing with void promise if IdleStateHandler is configured in the pipeline for read timeout events.

Modifications:

Better performance.

Result:

No more ChannelFutureListeners are created if IdleStateHandler is only configured for read timeouts allowing for writing to the channel with void promise.
2015-05-25 21:12:51 +02:00
Norman Maurer
6703318010 Only call System.nanoTime() if no read batch is ongoing. Related to [#3808]
Motivation:

[#3808] introduced some improvements to reduce the calls to System.nanoTime() but missed one possible optimization.

Modifications:

Only call System.nanoTime() if no reading patch is in process.

Result:

Less System.nanoTime() calls.
2015-05-25 18:20:50 +02:00
Jestan Nirojan
74f67ddca7 SCTP echo client/server multi-homing examples
Motivation:
There are no Netty SCTP examples on multi-homing.

Modifications:
- Added new example classes based on echo client/server example

Result:
Better documentation
2015-05-22 22:24:18 +02:00
nmittler
2ca6a70211 Upgrading Jetty alpn-api version
Motivation:

Discussion is in https://github.com/jetty-project/jetty-alpn/issues/8. The new API allows protocol negotiation to properly throw SSLHandshakeException.

Modifications:

Updated the parent pom.xml with the new version.

Result:

Upgraded alpn-api now allows throwing SSLHandshakeException.
2015-05-22 13:15:21 -07:00
Norman Maurer
90adae7b32 Not try to write more then Integer.MAX_VALUE / SSIZE_MAX via writev
Motivation:

When trying to write more then Integer.MAX_VALUE / SSIZE_MAX via writev(...) the OS may return EINVAL depending on the kernel or the actual OS (bsd / osx always return EINVAL). This will trigger an IOException.

Modifications:

Never try to write more then Integer.MAX_VALUE / SSIZE_MAX when using writev.

Result:

No more IOException when write more data then Integer.MAX_VALUE / SSIZE_MAX via writev.
2015-05-21 12:01:51 +02:00
Ruslan Sennov
2cb7991aee QueryStringDecoder's javadoc fix 2015-05-21 11:46:58 +02:00
Robert Varga
addc4413aa Do not call System.nanoTime() in ReadTimeoutHandler.channelRead()
Motivation:

We mitigate callouts to System.nanoTime() in SingleThreadEventExecutor
as it is 'relatively expensive'. On a modern system, tak translates to
about 20ns per call. With channelReadComplete() we can side-step this in
channelRead().

Modifications:

Introduce a boolean flag, which indicates that a read batch is currently
on-going, which acts as a flush guard for lastReadTime. Update
lastReadTime in channelReadComplete() just before setting the flag to
false. We set the flag to true in channelRead().

The periodic task examines the flag, and if it observes it to be true,
it will reschedule the task for the full duration. If it observes as
false, it will read lastReadTime and adjust the delay accordingly.

Result:

ReadTimeoutHandler calls System.nanoTime() only once per read batch.
2015-05-21 07:13:39 +02:00
Norman Maurer
8fed94eee3 Expose metrics for PooledByteBufAllocator
Motivation:

The PooledByteBufAllocator is more or less a black-box atm. We need to expose some metrics to allow the user to get a better idea how to tune it.

Modifications:

- Expose different metrics via PooledByteBufAllocator
- Add *Metrics interfaces

Result:

It is now easy to gather metrics and detail about the PooledByteBufAllocator and so get a better understanding about resource-usage etc.
2015-05-20 21:01:37 +02:00
Norman Maurer
8a1a0b2c1e Add PooledSlicedByteBuf and PooledDuplicatedByteBuf
Motivation:

At the moment when calling slice(...) or duplicate(...) on a Pooled*ByteBuf a new SlicedByteBuf or DuplicatedByteBuf. This can create a lot of GC.

Modifications:

Add PooledSlicedByteBuf and PooledDuplicatedByteBuf which will be used when a PooledByteBuf is used.

Result:

Less GC.
2015-05-20 11:37:41 +02:00
Norman Maurer
981292ffc0 Clarify ByteBuf.duplicate() semantics.
Motivation:

From the javadocs of ByteBuf.duplicate() it is not clear if the reader and writer marks will be duplicated.

Modifications:

Add sentence to clarify that marks will not be duplicated.

Result:

Clear semantics.
2015-05-20 08:32:43 +02:00
Norman Maurer
979fdfd3d3 Reset markers when obtain PooledByteBuf.
Motivation:

When allocate a PooledByteBuf we need to ensure to also reset the markers for the readerIndex and writerIndex.

Modifications:

- Correct reset the markers
- Add test-case for it

Result:

Correctly reset markers.
2015-05-20 07:29:21 +02:00
Norman Maurer
dc73c15df3 No need to release lock and acquire again when allocate normal size.
Motiviation:

When tried to allocate tiny and small sized and failed to serve these out of the PoolSubPage we exit the synchronization
block just to enter it again when call allocateNormal(...).

Modification:

Not exit the synchronized block until allocateNormal(...) is done.

Result:

Better performance.
2015-05-18 10:23:29 +02:00
Robin Stocker
214e889937 Fix typo in FingerprintTrustManagerFactory docs 2015-05-18 08:31:00 +02:00
Norman Maurer
b02f52a680 [#3784] Support hostname verification when using OpenSSLEngine
Motivation:

At the moment hostname verification is not supported with OpenSSLEngine.

Modifications:

- Allow to create OpenSslEngine with peerHost and peerPort informations.
- Respect endPointIdentificationAlgorithm and algorithmConstraints when set and get SSLParamaters.

Result:

hostname verification is supported now.
2015-05-18 08:01:38 +02:00
Stephane Landelle
59c882cb51 Minor ClientCookieDecoder improvements
Motivation:

* Path attribute should be null, not empty String, if it's passed as "Path=".
* Only extract attribute value when the name is recognized.
* Only extract Expires attribute value String if MaxAge is undefined as it has precedence.

Modification:

Modify ClientCookieDecoder.
Add "testIgnoreEmptyPath" test in ClientCookieDecoderTest.

Result:

More idyomatic Path behavior (like Domain).
Minor performance improvement in some corner cases.
2015-05-12 11:26:35 +02:00
Eric Anderson
b44fe3fd68 Add missing SslContextBuilder.forServer(KeyManagerFactory)
Motivation:

keyManager() is required on server-side, and so there is a forServer()
method for each override of keyManager(). However, one of the
forServer() overrides was missing, which meant that if you wanted to use
a KeyManagerFactory you were forced to provide garbage configuration
just to get past null checks.

Modifications:

Add missing override.

Result:

No hacks to use SslContextBuilder on server-side with KeyManagerFactory.
Resolves #3775
2015-05-11 22:09:50 +02:00
Frederic Bregier
d0c81604b6 Proposal to fix issue #3768 (3.10)
Motivations:
When using HttpPostRequestEncoder and trying to set an attribute if a
charset is defined, currenlty implicit Charset.toStrng() is used, given
wrong format.
As in Android for UTF-16 = "com.ibm.icu4jni.charset.CharsetICU[UTF-16]".

Modifications:
Each time charset is used to be printed as its name, charset.name() is
used to get the canonical name.

Result:
Now get "UTF-16" instead.
(3.10 version)
2015-05-11 06:16:40 +02:00
Scott Mitchell
ef95fbf6bb ALPN/NPN Examples Not using failure behavior compatible with all SslProviders
Motivation:
Examples that are using ALPN/NPN are using a failure mode which is not supported by the JDK SslProvider. The examples fail to run and throw an exception if the JDK SslProvider is used.

Modifications:
- Use SelectorFailureBehavior.NO_ADVERTISE
- Use SelectedListenerFailureBehavior.ACCEPT

Result:
Examples can be run with both OpenSsl and JDK SslProviders.
2015-05-08 09:40:25 -07:00
Norman Maurer
d1c46ca987 [maven-release-plugin] prepare for next development iteration 2015-05-07 11:33:47 -04:00
Norman Maurer
005d4a42fc [maven-release-plugin] prepare release netty-4.0.28.Final 2015-05-07 11:33:09 -04:00
Norman Maurer
0f4d3a981e Revert "[maven-release-plugin] prepare for next development iteration"
This reverts commit 3c10ffab5e.
2015-05-07 11:02:03 -04:00