Commit Graph

6191 Commits

Author SHA1 Message Date
Norman Maurer
6f2401ac40 Allow the user to call slice().retain() or duplicate.retain() in his/her ByteToMessageDecoder.decode(...) method.
Motivation:
At the moment a user can not safetly call slice().retain() or duplicate.retain()in the ByteToMessageDecoder.decode(...) implementation without the risk to see coruption because we may call discardSomeReadBytes() to make room on the buffer once the handling is done.

Modifications:
Check for the refCnt() before call discardSomeReadBytes() and also check before call decode(...) to create a copy if needed.

Result:
The user can safetly call slice().retain() or duplicate.retain() in his/her ByteToMessageDecoder.decode(...) method.
2014-04-07 11:33:52 +02:00
Norman Maurer
2aa35922b4 [#2363] SelectedSelectionKeySet may hold strong reference to SelectionKey after Channel is closed
Motivation:
Because we not null out the array entry in the SelectionKey[] which is produced by SelectedSelectionKeySet.flip() we may end up with a few SelectionKeyreferences still hanging around here even after the Channel was closed. As these entries may be present at the end of the SelectionKey[] which is never updated for a long time as not enough SelectionKeys are ready.

Modifications:
Once we access the SelectionKey out of the SelectionKey[] we directly null it out.

Result:
Reference can be GC'ed right away once the Channel was closed.
2014-04-05 19:29:58 +02:00
Norman Maurer
a4c80211ea [#2359] EpollSocketChannel.remoteAddress0() is always null on accepted EpollSocketChannels
Motivation:
EpollSocketChannel.remoteAddress0() is always null on accepted EpollSocketChannels as we not set it excplicit.

Modifications:
Correctly retrieve the local and remote address when accept new channel and store it

Result:
EpollSocketchannel.remoteAddress0() and EpollSocketChannel.localAddress0() return correct addresses
2014-04-04 15:23:07 +02:00
Norman Maurer
5cd939f634 [#2362] AbstractChannel.AbstractUnsafe.write(...) is slow
Motivation:
At the moment we do a Channel.isActive() check in every AbstractChannel.AbstractUnsafe.write(...) call which gives quite some overhead as shown in the profiler when you write fast enough. We can eliminate the check and do something more smart here.

Modifications:
Remove the isActive() check and just check if the ChannelOutboundBuffer was set to null before, which means the Channel was closed. The rest will be handled in flush0() anyway.

Result:
Less overhead when doing many write calls
2014-04-04 09:45:03 +02:00
Norman Maurer
9d695e03db [#2361] Native.epollCreate(...) fails on systems using a kernel < 2.6.27 / glibc < 2.9
Motivation:
Native.epollCreate(...) fails on systems using a kernel < 2.6.27 / glibc < 2.9 because it uses epoll_create1(...) without checking if it is present

Modifications:
Check if epoll_create1(...) exists abd if not fall back to use epoll_create(...)

Result:
Works even on systems with kernel < 2.6.27 / glibc < 2.9
2014-04-04 07:54:49 +02:00
Norman Maurer
7966c42636 [#2358] SslHandler.safeClose(...) may not notify the ChannelPromise
Motivation:
In SslHandler.safeClose(...) we attach a ChannelFutureListener to the flushFuture and will notify the ChannelPromise which was used for close(...) in it. The problem here is that we only call ChannelHandlerContext.close(ChannelPromise) if Channel.isActive() is true and otherwise not notify it at all. We should just call ChannelHandlerContext.close(ChannelPromise) in all cases.

Modifications:
Always call ChannelHandlerContext.close(ChannelPromise) in the ChannelFutureListeiner

Result:
ChannelPromise used for close the Channel is notified in all cases
2014-04-03 13:29:54 +02:00
Norman Maurer
816165c96a [maven-release-plugin] prepare for next development iteration 2014-04-01 07:21:40 +02:00
Norman Maurer
1512a4dcca [maven-release-plugin] prepare release netty-4.0.18.Final 2014-04-01 07:20:16 +02:00
Norman Maurer
aa74f00006 [#2349] Correctly handle cancelled ChannelPromise in DefaultChannelHandlerContext
Motivation:
At the moment an IllegalArgumentException will be thrown if a ChannelPromise is cancelled while propagate through the ChannelPipeline. This is not correct, we should just stop to propagate it as it is valid to cancel at any time.

Modifications:
Stop propagate the operation through the ChannelPipeline once a ChannelPromise is cancelled.

Result:
No more IllegalArgumentException when cancel a ChannelPromise while moving through the ChannelPipeline.
2014-03-31 07:29:31 +02:00
Daniel Bevenius
14b962a434 Adding origins whitelist support for CORS
Motivation:
Currently the CORS support only handles a single origin, or a wildcard
origin. This task should enhance Netty's CORS support to allow multiple
origins to be specified. Just being allowed to specify one origin is
particulary limiting when a site support both http and https for
example.

Modifications:
- Updated CorsConfig and its Builder to accept multiple origins.

Result:
Users are now able to configure multiple origins for CORS.

[https://github.com/netty/netty/issues/2346]
2014-03-30 19:35:48 +02:00
Ian Barfield
5f9993ee1f Deleting redundant needsFlush boolean
Motivation:

In ChunkedWriteHandler, there is a redundant variable that servers
no purpose. It implies that under some conditions you might not want
to flush.

Modifications:

Removed the variable and the if condition that read it. The boolean
was always true so just removing the if statement was fine.

Result:

Slightly less misleading code.
2014-03-29 20:21:33 +01:00
Alexey Diomin
ab2119feac [#2339] Reduce memory usage in ProtobufVarint32LengthFieldPrepender
Motivation:

Reduce memory usage in ProtobufVarint32LengthFieldPrepender.

Modifications:

Explicit set the buffer size that is needed for the header (between 1 and 5 bytes).

Result:

Less memory usage in ProtobufVarint32LengthFieldPrepender.
2014-03-28 19:46:54 +01:00
CoNDoRip
68670ba195 Allow specifying SelectorProvider when constructing an NIO channel #2311
Motivation:

At the moment we use the system-wide default selector provider for this invocation of the Java virtual machine when constructing a new NIO channel, which makes using an alternative SelectorProvider practically useless.
This change allows user specify his/her preferred SelectorProvider.

Modifications:

Add SelectorProvider as a param for current `private static *Channel newSocket` method of NioSocketChannel, NioServerSocketChannel and NioDatagramChannel.
Change default constructors of NioSocketChannel, NioServerSocketChannel and NioDatagramChannel to use DEFAULT_SELECTOR_PROVIDER when calling newSocket(SelectorProvider).
Add new constructors for NioSocketChannel, NioServerSocketChannel and NioDatagramChannel which allow user specify his/her preferred SelectorProvider.

Result:

Now users can specify his/her preferred SelectorProvider when constructing an NIO channel.
2014-03-26 21:08:15 +01:00
Norman Maurer
76d091a884 Make sure the local / remote InetSocketAddres can be obtained. Part of [#2262]
Motivation:
Make sure the remote/local InetSocketAddress can be obtained correctly

Modifications:
Set the remote/local InetSocketAddress after a bind/connect operation was performed

Result:
It is possible to still access the informations even after the fd became invalid. This mirror the behaviour of NIO.
2014-03-22 15:01:49 +01:00
Norman Maurer
6f0e740e19 [#2234] Use QueryStringDecoder.decodeComponent to decode url-encoded data instead of Java's URLDecoder. Backport of 98bc7b7859
Motivation:
Previously, we used URLDecoder.decode(...) to decode url-encoded data. This generates a lot of garbage and takes a considerable amount of time.

Modifications:
Replace URLDecoder.decode(...) with QueryStringDecoder.decodeComponent(...)

Result:
Less garbage to GC and faster decode processing.
2014-03-22 14:14:01 +01:00
Daniel Bevenius
7c4aea4bc3 Fixing CorsConfigTest failure under Java 8.
Motivation:
When running the build with Java 8 the following error occurred:

java: reference to preflightResponseHeader is ambiguous
  both method
  <T>preflightResponseHeader(java.lang.CharSequence,java.lang.Iterable<T>)
  in io.netty.handler.codec.http.cors.CorsConfig.Builder and method
  <T>preflightResponseHeader(java.lang.String,java.util.concurrent.Callable<T>)
  in io.netty.handler.codec.http.cors.CorsConfig.Builder match

The offending class was CorsConfigTest and its shouldThrowIfValueIsNull
which contained the following line:
withOrigin("*").preflightResponseHeader("HeaderName", null).build();

Modifications:
Updated the offending method with to supply a type, and object array, to
avoid the error.

Result:
After this I was able to build with Java 7 and Java 8
2014-03-22 07:27:55 +01:00
Daniel Bevenius
8d8bbec274 Adding support for specifying preflight response headers.
Motivation:

An intermediary like a load balancer might require that a Cross Origin
Resource Sharing (CORS) preflight request have certain headers set.
As a concrete example the Elastic Load Balancer (ELB) requires the
'Date' and 'Content-Length' header to be set or it will fail with a 502
error code.

This works is an enhancement of https://github.com/netty/netty/pull/2290

Modifications:

CorsConfig has been extended to make additional HTTP response headers
configurable for preflight responses. Since some headers, like the
'Date' header need to be generated each time, m0wfo suggested using a
Callable.

Result:

By default, the 'Date' and 'Content-Lenght' headers will be sent in a
preflight response. This can be overriden and users can specify
any headers that might be required by different intermediaries.
2014-03-21 15:19:55 +01:00
Trustin Lee
50f685fa49 Use SecureRandom.generateSeed() to generate ThreadLocalRandom's initialSeedUniquifier
Motivation:

Previously, we used SecureRandom.nextLong() to generate the initialSeedUniquifier.  This required more entrophy than necessary because it has to 1) generate the seed of SecureRandom first and then 2) generate a random long integer.  Instead, we can use generateSeed() to skip the step (2)

Modifications:

Use generateSeed() instead of nextLong()

Result:

ThreadLocalRandom requires less amount of entrphy to start up
2014-03-21 13:43:57 +09:00
Trustin Lee
feb352c6df Reduce the time taken by NetUtil class initialization
Motivation:

As reported in #2331, some query operations in NetworkInterface takes much longer time than we expected.  For example, specifying -Djava.net.preferIPv4Stack=true option in Window increases the execution time by more than 4 times.  Some Windows systems have more than 20 network interfaces, and this problem gets bigger as the number of unused (virtual) NICs increases.

Modification:

Use NetworkInterface.getInetAddresses() wherever possible.
Before iterating over all NICs reported by NetworkInterface, filter the NICs without proper InetAddresses.  This reduces the number of candidates quite a lot.
NetUtil does not query hardware address of NIC in the first place but uses InetAddress.isLoopbackAddress().
Do not call unnecessary query operations on NetworkInterface.  Just get hardware address and compare.

Result:

Significantly reduced class initialization time
2014-03-21 13:02:37 +09:00
Norman Maurer
13fd69e871 Implement Thread caches for pooled buffers to minimize conditions. This fixes [#2264] and [#808].
Motivation:
Remove the synchronization bottleneck in PoolArena and so speed up things

Modifications:

This implementation uses kind of the same technics as outlined in the jemalloc paper and jemalloc
blogpost https://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919.

At the moment we only cache for "known" Threads (that powers EventExecutors) and not for others to keep the overhead
minimal when need to free up unused buffers in the cache and free up cached buffers once the Thread completes. Here
we use multi-level caches for tiny, small and normal allocations. Huge allocations are not cached at all to keep the
memory usage at a sane level. All the different cache configurations can be adjusted via system properties or the constructor
directly where it makes sense.

Result:
Less conditions as most allocations can be served by the cache itself
2014-03-20 09:18:04 -07:00
Trustin Lee
2afe58f5f1 Fix and simplify freeing a direct buffer / Fix Android support
Motivation:

6e8ba291cf introduced a regression in Android because Android does not have sun.nio.ch.DirectBuffer (see #2330.)  I also found PlatformDependent0.freeDirectBuffer() and freeDirectBufferUnsafe() are pretty much same after the commit and the unsafe version should be removed.

Modifications:

- Merge PlatformDependent0.freeDirectBuffer() and freeDirectBufferUnsafe() into one method.
- Make the Unsafe unavailable when sun.nio.ch.DirectBuffer is unavailable.  We could keep the Unsafe available and handle the sun.nio.ch.DirectBuffer case separately, but I don't want to complicate our code just because of that.  All supported JDK versions have sun.nio.ch.DirectBuffer if the Unsafe is available.

Result:

Simpler code. Fixes Android support (#2330)
2014-03-20 11:13:25 +09:00
Norman Maurer
cf2c8b40ae Replace usage of System.currentTimeMillis() with System.nanoTime()
Motivation:

Currently we use System.currentTimeMillis() in our timeout handlers this is bad
for various reasons like when the clock adjusts etc.

Modifications:

Replace System.currentTimeMillis() with System.nanoTime()

Result:

More robust timeout handling
2014-03-18 16:05:51 +09:00
Jakob Buchgraber
17ba35b6d0 Bit tricks to check for and calculate power of two.
Motivation:
I was studying the code and thought this was simpler and easier to
understand.

Modifications:
Replaced the for loop and if conditions, with a simple implementation.

Result:
Code is easier to understand.
2014-03-18 15:59:58 +09:00
Trustin Lee
1d73df906f Add -verbose:gc option for test runs
Motivation:

While investigating the recent CI machine crashes, I observed that the
JVM processes spawned by surefire sometimes take up to 1 GiB RAM.
Consuming large amount of memory isn't really a problem, but we need to
make sure no GC trashing is occuring during the tests.

Modifications:

Add -verbose:gc option to the test JVM arguments

Result:

We can determine if there is any GC anomalies going on in our CI
machine.
2014-03-17 14:20:32 +09:00
Trustin Lee
5b13e9058b Reduce SO_TIMEOUT of testsuite so it finishes sooner
Motivation:

Testing the OIO transport takes longer time than other transports because it has to wait for SO_TIMEOUT if there is nothing to read.  In production, it's not a good idea to decrease this value (1000ms) because it will result in so many SocketTimeoutExceptions internally, but doing so in the testsuite should be fine.

Modifications:

Reduce the default SO_TIMEOUT of OIO channels to 10 ms.

Result:

Our testsuite finishes sooner.
2014-03-17 10:53:49 +09:00
Trustin Lee
5acb8759e4 Perform cross-tests between NIO and epoll transport
Motivation:

The epoll testsuite tests the epoll transport only against itself (i.e. epoll x epoll only).  We should test the epoll transport also against the well-tested NIO transport, too.

Modifications:

- Make SocketTestPermutation extensible and reusable so that the epoll testsuite can take advantage of it.
- Rename EpollTestUtils to EpollSocketTestPermutation and make it extend SocketTestPermutation.
- Overall clean-up of SocketTestPermutation
  - Use Arrays.asList() for simplicity
  - Add combo() method to remove code duplication

Result:

The epoll transport is now also tested against the NIO transport.  SocketTestPermutation got cleaner.
2014-03-17 10:38:18 +09:00
Trustin Lee
bcd11d1ac2 Fix 'incompatible event loop' regression
Motivation:

Previous commit (2de65e25e9) introduced a regression that makes the epoll testsuite fail with an 'incompatible event loop' error.

Modifications:

Use the correct event loop type.

Result:

Build doesn't fail anymore.
2014-03-17 09:52:10 +09:00
Trustin Lee
eda702b6ab Set timeout for SocketSslEchoTest
Motivation:

We are seeing EpollSocketSslEchoTest does not finish itself while its I/O thread is busy.  Jenkins should have terminated them when the global build timeout reaches, but Jenkins seems to fail to do so.  What's more interesting is that Jenkins will start another job before the EpollSocketSslEchoTest is terminated, and Linux starts to oom-kill them, impacting the uptime of the CI service.

Modifications:

- Set timeout for all test cases in SocketSslEchoTest so that all SSL tests terminate themselves when they take too long.
- Fix a bug where the epoll testsuite uses non-daemon threads which can potentially prevent JVM from quitting.
- (Cleanup) Separate boss group and worker group just like we do for NIO/OIO transport testsuite.

Result:

Potentially more stable CI machine.
2014-03-17 09:31:48 +09:00
Norman Maurer
140f963d60 Replace usage of UnknownHostException with UnresolveableAddressException. Part of [#2262]
Motivation:
We better use UnresolveableAddressException as NIO does the same.

Modifications:
Replace usage of UnknownHostException with UnresolveableAddressException

Result:
epoll transport and nio transport behave the same way
2014-03-16 08:29:45 -07:00
Norman Maurer
a17d28605a [#2326] Add constructor to NioServerSocketChannel which accepts a ServerSocketChannel
Motivation:
Allow the user to create a NioServerSocketChannel from an existing ServerSocketChannel.

Modifications:
Add an extra constructor

Result:
Now the user is be able to create a NioServerSocketChannel from an existing ServerSocketChannel, like he can do with all the other Nio*Channel implemntations.
2014-03-16 07:00:44 -07:00
Norman Maurer
535b024913 [#2323] Make it clear a Channel must be closed to release all resources
Motivation:
Ensure the user know the Channel must be closed to release resources like filehandles.

Modifications:
Add some extra javadoc.

Result:
More clear documentation
2014-03-16 06:56:49 -07:00
Norman Maurer
b22d92057b [#2262] Fix NPE triggered by unresolveable InetSocketAddress in epoll transport
Motivation:
At the moment when an unresolvable InetSocketAddress is passed into the epoll transport a NPE is thrown

Modifications:
Add check in place which will throw an UnknownHostException if an InetSocketAddress could not been resolved.

Result:
Proper handling of unresolvable InetSocketAddresses
2014-03-16 06:25:52 -07:00
Frederic Bregier
55bafbd9c2 [#2305] Fix issue related to decoding post request raized an exception due to a split of information by chunk not correctly taken into account by the decoder
Motivation:

If the last item analyzed in a previous received HttpChunk/HttpContent was a part of an attribute's name, the read index was not set to the new right place and therefore raizing an exception in some case (since the "new" name analyzed is empty, which is not allowed so the exception).

What appears there is that the read index should be reset to the last valid position encountered whatever the case. Currently it was set when only when there is an attribute not already finished (name is ok, but content is possibly not).

Therefore the issue is that elements could be rescanned multiple times (including completed elements) and moreover some bad decoding can occur such as when in a middle of an attribute's name.

Modifications:

To fix this issue, since "firstpos" contains the last "valid" read index of the decoding (when finding a '&', '=', 'CR/LF'), we should add the setting of the read index for the following cases:

'lastchunk' encountered, therefore finishing the current buffer
any other cases than current attribute is not finished (name not found yet in particular)
So adding for this 2 cases:

undecodedChunk.readerIndex(firstpos);

Result:

Now the decoding is done once, content is added from chunk/content to chunk/content, name is decoded correctly even if in the middle of 2 chunks/contents.
A Junit test code was added: testChunkCorrect that should not raized any exception.
2014-03-14 09:36:50 +01:00
Bourne, Geoff
1c074eabe5 Fix limit computation of NIO ByteBuffers obtained via ReadOnlyByteBufferBuf.nioBuffer
Motivation:

When starting with a read-only NIO buffer, wrapping it in a ByteBuf,
and then later retrieving a re-wrapped NIO buffer the limit was getting
too short.

Modifications:

Changed ReadOnlyByteBufferBuf.nioBuffer(int,int) to compute the
limit in the same manner as the internalNioBuffer method.

Result:

Round-trip conversion from NIO to ByteBuf to NIO will work reliably.
2014-03-14 08:07:19 +01:00
Norman Maurer
92037e8bea [#2307] Remove synchronized bottleneck in SingleThreadEventExecutor.execute(...)
Motivation:
Remove the synchronization bottleneck in startThread() which is called by each execute(..) call from outside the EventLoop.

Modifications:
Replace the synchronized block with the use of AtomicInteger and compareAndSet loops.

Result:
Less conditions during SingleThreadEventExecutor.execute(...)
2014-03-13 09:45:05 +01:00
Norman Maurer
d4d2085377 Remove sniffer whitelist entries for NIO.2
Motivation:
Cleanup pom.xml file.

Modifications:
Remove sniffer whitelist entries for NIO.2 as we not include a NIO.2 bases transport anymore.

Result:
Less entries in pom.xml
2014-03-13 06:58:36 +01:00
Norman Maurer
e0b2f34a37 [#2308] Use SelectorProvider.open*() to open NIO channels and so remove condition when create new NIO channels.
Motivation:
At the moment we use SocketChannel.open(), ServerSocketChannel.open() and DatagramSocketChannel.open(...) within the constructor of our
NIO channels. This introduces a bottleneck if you create a lot of connections as these calls delegate to SelectorProvider.provider() which
uses synchronized internal. This change removed the bottleneck.

Modifications:
Obtain a static instance of the SelectorProvider and use SelectorProvider.openSocketChannel(), SelectorProvider.openServerSocketChannel() and
SelectorProvider.openDatagramChannel(). This eliminates the bottleneck as SelectorProvider.provider() is not called on every channel creation.

Result:
Less conditions when create new channels.
2014-03-13 06:46:05 +01:00
Norman Maurer
f396f40db9 Fix checkstyle errors introduced by f0d1bbd63e 2014-03-12 13:57:01 +01:00
Norman Maurer
ef76907422 Remove condition in ChannelHandlerAdapter.isSharable() by caching the result of the annotation lookup.
Motivation:
Remove the synchronization bottleneck and so speed up things

Modifications:
Introduce a ThreadLocal cache that holds mappings between classes of ChannelHandlerAdapater implementations and the result of checking if the @Sharable annotation is present.
This way we only will need to do the real check one time and server the other calls via the cache. A ThreadLocal and WeakHashMap combo is used to implement the cache
as this way we can minimize the conditions while still be sure we not leak class instances in containers.

Result:
Less conditions during adding ChannelHandlerAdapter to the ChannelPipeline
2014-03-12 12:31:22 +01:00
Trustin Lee
8d2ddae4a7 Add capacity limit to Recycler / Optimize when assertion is off
Motivation:

- As reported recently [1], Recycler's thread-local object pool has unbounded capacity which is a potential problem.
- It accesses a hash table on each push and pop for debugging purposes.  We don't really need it besides debugging Netty itself.

Modifications:

- Introduced the maxCapacity constructor parameter to Recycler.  The default default maxCapacity is retrieved from the system property whose default is 256K, which should be plenty for most cases.
- Recycler.Stack.map is now created and accessed only when assertion is enabled for Recycler.

Result:

- Recycler does not grow infinitely anymore.
- If assertion is disabled, Recycler should be much faster.

[1] https://github.com/netty/netty/issues/1841
2014-03-12 18:17:24 +09:00
Norman Maurer
ef69c4592b Revert "ipfilter implementation for netty 4/5 [#2129]"
Motivation:
ipfilter implementation may need a bit more time to stabilize.

Modifications:
Revert the commit that added it to 4.0 branch (it is still included in 4.1 and master)

Result:
ipfilter will not be included in the next 4.0.x release (just as before9
2014-03-12 06:26:50 +01:00
Trustin Lee
8e2de4fcb2 Move the pull request guide to the developer guide
Motivation:

CONTRIBUTING.md is useful only because it lets Github show a user the
link to it so the user can check what information we need before
submitting a bug report.  However, Github does not do the same for a
pull request submission form, and thus there's no reason to keep the
information about how to submit a good pull request in CONTRIBUTING.md.

Modification:

Replace the section about issuing a pull request with the link to the
official developer guide.

Result:

CONTRIBUTING.md is easier to maintain.
2014-03-12 13:18:14 +09:00
Norman Maurer
072c5e3cc6 Use bitwise operations to choose next EventExecutor if number of EventExecutors is power of two 2014-03-10 20:46:59 +01:00
Jakob Buchgraber
ae20f278ff ipfilter implementation for netty 4/5 [#2129] 2014-03-10 20:46:05 +01:00
Norman Maurer
fc5d20ff2d [#2297] Correctly close all registered Channels on EpollEventLoop.closeAll() 2014-03-10 20:27:44 +01:00
David Dossot
3c7ef6ffce added support for empty query parameters 2014-03-10 06:49:00 +01:00
Norman Maurer
ae818b3be6 Fix buffer leak in test which was introduced while implement ZLIB_OR_NONE support. Related to [#2269] 2014-03-10 06:24:53 +01:00
Aaron Riekenberg
223deb255e [#2280] Correct logic in Native.finishConnect. Fix use of optval parameter in c getOption function. In epoll event loop, check that channel is open before processing event. 2014-03-09 19:37:32 +01:00
Chris Mowforth
3c1bcb7279 Add content length and date headers to CORS response 2014-03-06 21:00:10 +01:00
Norman Maurer
835c446f5f Fix buffer leak in test which was introduced while implement ZLIB_OR_NONE support. Related to [#2269] 2014-03-06 20:03:40 +01:00