Commit Graph

6310 Commits

Author SHA1 Message Date
Norman Maurer
35061a4332 [#2692] Allows notify ChannelFutureProgressListener on complete writes
Motivation:

We have some inconsistency when handling writes. Sometimes we call ChannelOutboundBuffer.progress(...) also for complete writes and sometimes not. We should call it always.

Modifications:

Correctly call ChannelOuboundBuffer.progress(...) for complete and incomplete writes.

Result:

Consistent behavior
2014-07-28 04:12:59 -07:00
Norman Maurer
d989b24351 [#2662] Fix race in cancellation of TimerTasks which could let to NPE
Motivation:

Due some race-condition while handling canellation of TimerTasks it was possibleto corrupt the linked-list structure that is represent by HashedWheelBucket and so produce a NPE.

Modification:

Fix the problem by adding another MpscLinkedQueue which holds the cancellation tasks and process them on each tick. This allows to use no synchronization / locking at all while introduce a latency of max 1 tick before the TimerTask can be GC'ed.

Result:

No more NPE
2014-07-25 06:34:49 +02:00
Norman Maurer
f5faada77c [#2705] Call fireChannelReadComplete() if channelActive(...) decodes messages in ReplayingDecoder / ByteToMessageDecoder
Motivation:

In ReplayingDecoder / ByteToMessageDecoder channelInactive(...) method we try to decode a last time and fire all decoded messages throw the pipeline before call ctx.fireChannelInactive(...). To keep the correct order of events we also need to call ctx.fireChannelReadComplete() if we read anything.

Modifications:

- Channel channelInactive(...) to call ctx.fireChannelReadComplete() if something was decoded
- Move out.recycle() to finally block

Result:

Correct order of events.
2014-07-24 14:33:56 +02:00
Willem Jiang
ce069e2dc4 Updated the ChannelGroup JavaDoc by removing b.releaseExternalResources(); 2014-07-24 10:55:22 +02:00
Trustin Lee
08c87c6256 Reduce the default initial capacity of ChannelOutboundBuffer
Motivation:

ChannelOutboundBuffer is basically a circular array queue of its entry
objects.  Once an entry is created in the array, it is never nulled out
to reduce the allocation cost.

However, because it is a circular queue, the array almost always ends up
with as many entry instances as the size of the array, regardless of the
number of pending writes.

At worst case, a channel might have only 1 pending writes at maximum
while creating 32 entry objects, where 32 is the initial capacity of the
array.

Modifications:

- Reduce the initial capacity of the circular array queue to 4.
- Make the initial capacity of the circular array queue configurable

Result:

We spend 4 times less memory for entry objects under certain
circumstances.
2014-07-22 13:38:24 -07:00
Trustin Lee
ed304b42f5 Add a link to the 'native transports' page 2014-07-21 12:54:43 -07:00
Osvaldo Doederlein
0ae7f2f5e1 Fixes and improvements to IntObjectHashMap. Related to [#2659]
- Rewrite with linear probing, no state array, compaction at cleanup
- Optimize keys() and values() to not use reflection
- Optimize hashCode() and equals() for efficient iteration
- Fixed equals() to not return true for equals(null)
- Optimize iterator to not allocate new Entry at each next()
- Added toString()
- Added some new unit tests
2014-07-21 16:44:19 +02:00
Norman Maurer
e25891964c [#2675] Replace synchronization performed on util.concurrent instance in TrafficCounter
Motivation:

Message from FindBugs:
This method performs synchronization an object that is an instance of a class from the java.util.concurrent package (or its subclasses). Instances of these classes have their own concurrency control mechanisms that are orthogonal to the synchronization provided by the Java keyword synchronized. For example, synchronizing on an AtomicBoolean will not prevent other threads from modifying the AtomicBoolean.
Such code may be correct, but should be carefully reviewed and documented, and may confuse people who have to maintain the code at a later date.

Modification:

Use synchronized(this)

Result:

Less confusing code
2014-07-21 08:21:27 +02:00
Jeff Pinner
6d0233d3bd SPDY: fix pushed response NullPointerException 2014-07-21 07:59:25 +02:00
Norman Maurer
eb34cbb2d8 [#2685] Epoll transport should use GetPrimitiveArrayCritical / ReleasePrimitiveArrayCritical
Motivation:

At the moment we use Get*ArrayElement all the time in the epoll transport which may be wasteful as the JVM may do a memory copy for this. For code-path that will get executed fast (without blocking) we should better make use of GetPrimitiveArrayCritical and ReleasePrimitiveArrayCritical as this signal the JVM that we not want to do any memory copy if not really needed. It is important to only do this on non-blocking code-path as this may even suspend the GC to disallow the JVM to move the arrays around.

See also http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/functions.html#GetPrimitiveArrayCritical

Modification:

Make use of GetPrimitiveArrayCritical / ReleasePrimitiveArrayCritical as replacement for Get*ArrayElement / Release*ArrayElement where possible.

Result:

Better performance due less memory copies.
2014-07-21 07:07:09 +02:00
Norman Maurer
028128993d [#2684] EpollSocketChannel gathering writes should take fast-path if possible
Motivation:

In EpollSocketchannel.writeBytesMultiple(...) we loop over all buffers to see if we need to adjust the readerIndex for incomplete writes. We can skip this if we know that everything was written (a.k.a complete write).

Modification:

Use fast-path if all bytes are written and so no need to loop over buffers

Result:

Fast write path for the average use.
2014-07-21 06:43:49 +02:00
Norman Maurer
e7310cc4c0 Only try gathering writes if we have more then one buffer to write. Part of [#2680]. 2014-07-20 20:00:21 +02:00
Norman Maurer
cb2246ee07 [#2680] ChannelOutboundBuffer.nioBuffers() should always return non-null array as stated in javadocs
Motivation:

At the moment ChannelOutboundBuffer.nioBuffers() returns null if something is contained in the ChannelOutboundBuffer which is not a ByteBuf. This is a problem for two reasons:
 1 - In the javadocs we state that it will never return null
 2 - We may do a not optimal write as there may be things that could be written via gathering writes

Modifications:

Change ChannelOutboundBuffer.nioBuffers() to never return null but have it contain all ByteBuffer that were found before the non ByteBuf. This way we can do a gathering write and also conform to the javadocs.

Result:

Better speed and also correct implementation in terms of the api.
2014-07-20 19:13:48 +02:00
Idel Pivnitskiy
dd026eb60a Fix NPE problems
Motivation:

Now Netty has a few problems with null values.

Modifications:

- Check File in DiskFileUpload.toString().
If File is null we will get NPE when calling toString() method.
- Check Result<String> in MqttDecoder.decodeConnectionPayload(...).
- Check Unsafe before calling unsafe.getClass() in PlatformDependent0 static block.
- Removed unnecessary null check in WebSocket08FrameEncoder.encode(...).
Because msg.content() can not return null.
- Removed unnecessary null checks in ConcurrentHashMapV8.removeTreeNode(TreeNode<K,V>).
- Removed unnecessary null check in OioDatagramChannel.doReadMessages(List<Object>).
Because tmpPacket.getSocketAddress() always returns new SocketAddress instance.
- Removed unnecessary null check in OioServerSocketChannel.doReadMessages(List<Object>).
Because socket.accept() always returns new Socket instance.
- Pass Unpooled.buffer(0) instead of null inside CloseWebSocketFrame(boolean, int) constructor.
If we will pass null we will get NPE in super class constructor.
- Added throw new IllegalStateException in GlobalEventExecutor.awaitInactivity(long, TimeUnit) if it will be called before GlobalEventExecutor.execute(Runnable).
Because now we will get NPE. IllegalStateException will be better in this case.
- Fixed null check in OpenSslServerContext.setTicketKeys(byte[]).
Now we throw new NPE if byte[] is not null.

Result:

Added new null checks when it is necessary, removed unnecessary null checks and fixed some NPE problems.
2014-07-20 12:56:21 +02:00
Idel Pivnitskiy
13569481bf Small fixes and improvements
Motivation:

Fix some typos in Netty.

Modifications:

- Fix potentially dangerous use of non-short-circuit logic in Recycler.transfer(Stack<?>).
- Removed double 'the the' in javadoc of EmbeddedChannel.
- Write to log an exception message if we can not get SOMAXCONN in the NetUtil's static block.
2014-07-20 09:36:57 +02:00
Idel Pivnitskiy
a7f0fdfa9c Fixes for compression codecs
Motivation:

Fixed founded mistakes in compression codecs.

Modifications:

- Changed return type of ZlibUtil.inflaterException() from CompressionException to DecompressionException
- Updated @throws in javadoc of JZlibDecoder to throw DecompressionException instead of CompressionException
- Fixed JdkZlibDecoder to throw DecompressionException instead of CompressionException
- Removed unnecessary empty lines in JdkZlibEncoder and JZlibEncoder
- Removed public modifier from Snappy class
- Added MAX_UNCOMPRESSED_DATA_SIZE constant in SnappyFramedDecoder
- Used in.readableBytes() instead of (in.writerIndex() - in.readerIndex()) in SnappyFramedDecoder
- Added private modifier for enum ChunkType in SnappyFramedDecoder

Result:

Fixed sum overflow in Bzip2HuffmanAllocator, improved exceptions in ZlibDecoder implementations, hid Snappy class
2014-07-20 09:34:31 +02:00
Idel Pivnitskiy
01b11ca2cb Small performance improvements
Modifications:

- Added a static modifier for CompositeByteBuf.Component.
This class is an inner class, but does not use its embedded reference to the object which created it. This reference makes the instances of the class larger, and may keep the reference to the creator object alive longer than necessary.
A boxed primitive is created from a String, just to extract the unboxed primitive value.
- Removed unnecessary checks if file exists before call mkdirs() in NativeLibraryLoader and PlatformDependent.
Because the method mkdirs() has this check inside.

Conflicts:
	codec-http/src/main/java/io/netty/handler/codec/http/multipart/DiskAttribute.java
	codec-stomp/src/main/java/io/netty/handler/codec/stomp/StompSubframeAggregator.java
	codec-stomp/src/main/java/io/netty/handler/codec/stomp/StompSubframeDecoder.java
2014-07-20 09:29:33 +02:00
Idel Pivnitskiy
f62777af11 Close ObjectInputStream in ObjectDecoder.decode(...)
Motivation:

We create a new CompactObjectInputStream with ByteBufInputStream in ObjectDecoder.decode(...) method and don't close this InputStreams before return statement.

Modifications:

Save link to the ObjectInputStream and close it before return statement.

Result:

Close InputStreams and clean up unused resources. It will be better for GC.
2014-07-20 09:23:17 +02:00
Norman Maurer
72173f0d16 [#2667] Write until EAGAIN in native transport and only call setEpollOut() in this case
Motivation:

In the previous fix for #2667 I did introduce a bit overhead by calling setEpollOut() too often.

Modification:

Only call setEpollOut() if really needed and remove unused code.

Result:

Less overhead when saturate network.
2014-07-18 20:31:19 +02:00
Norman Maurer
530badb239 [#2665] Continue writing on IOException when using DatagramChannel
Motivation:

As a DatagramChannel supports to write to multiple remote peers we must not close the Channel once a IOException accours as this error may be only valid for one remote peer.

Modification:

Continue writing on IOException.

Result:

DatagramChannel can be used even after an IOException accours during writing.
2014-07-18 12:35:18 +02:00
Norman Maurer
4835ff2d85 [#2667] Write until EAGAIN in native transport
Motivation:

We need to continue write until we hit EAGAIN to make sure we not see an starvation

Modification:

Write until EAGAIN is returned

Result:

No starvation when using native transport with ET.
2014-07-18 09:45:08 +02:00
Norman Maurer
460bf37387 [#2666] Fix possible NPE when try to fullfill connect ChannelPromise
Motivation:

Because of a missing return statement we may produce a NPE when try to fullfill the connect ChannelPromise when it was fullfilled before.

Modification:

Add missing return statement.

Result:

No more NPE.
2014-07-18 07:10:29 +02:00
Norman Maurer
db790123fd [#2647] Handle IOV_MAX in java code
Motivation:

The handling of IOV_MAX was done in JNI code base which makes stuff really complicated to maintain etc.

Modifications:

Move handling of IOV_MAX to java code to simplify stuff

Result:

Cleaner code.
2014-07-17 16:00:53 +02:00
Norman Maurer
bc10131f71 [#2664] Support write-spinning also in native transport
Motivation:

In our nio implementation we use write-spinning for maximize throughput, but in the native implementation this is not used.

Modification:

Respect writeSpinCount in native transport.

Result:

Better throughput
2014-07-17 13:10:38 +02:00
Gernot Pansy
4a937b7ca0 Fixed NPE in WebSocket00FrameDecoder if end couldn't be found in text frame
Motivation:
When we receive an incomplete WebSocketFrame we need to make sure to wait for more data. Because we not did this we could produce a NPE.

Modification:
Make sure we not try to add null into the RecyclableArrayList

Result:
no more NPE on incomplete frames.
2014-07-16 19:58:23 +02:00
Norman Maurer
8fe3d97d8d [#2656] Minimize ByteBuf.writeBytes(...) calls by pack the separator into the HttpHeaderEntity.
Motivation:
Currently we do 4 ByteBuf.writeBytes(...) calls per header line. This is can be improved.

Modification:
Introduce two new HttpHeaders methods to allow create HttpHeaderEntity which contains the separator. With this we can minimize it to 2 ByteBuf.writeBytes(...) calls per header line

Result:
Performance improvement.
2014-07-15 13:59:31 +02:00
Norman Maurer
c4d0a87e19 [#2653] Remove unnecessary ensureAccessible() calls
Motivation:

I introduced ensureAccessible() class as part of 6c47cc9711 in some places. Unfortunally I also added some where these are not needed and so caused a performance regression.

Modification:

Remove calls where not needed.

Result:

Fixed performance regression.
2014-07-14 21:02:07 +02:00
Norman Maurer
ccd88596f3 [#2653] Remove uncessary range checks for performance reasons
Motivation:

I introduced range checks as part of 6c47cc9711 in some places. Unfortunally I also added some where these are not needed and so caused a performance regression.

Modification:

Remove range checks where not needed

Result:

Fixed performance regression.
2014-07-14 11:40:27 +02:00
Norman Maurer
17b9e01780 Fix over-sensible testcase 2014-07-13 17:19:15 +02:00
Norman Maurer
08426d5554 [#2651] Fix possible infinite-loop when cancel tasks
Motivations:
In our new version of HWT we used some kind of lazy cancelation of timeouts by put them back in the queue and let them pick up on the next tick. This  multiple problems:
 - we may corrupt the MpscLinkedQueue if the task is used as tombstone
 - this sometimes lead to an uncessary delay especially when someone did executed some "heavy" logic in the TimeTask

Modifications:
Use a Lock per HashedWheelBucket for save and fast removal.

Modifications:
Cancellation of tasks can be done fast and so stuff can be GC'ed and no more infinite-loop possible
2014-07-11 15:41:22 +02:00
Norman Maurer
8f0d03998c [#2650] Allow to disable http header validation in SpdyHttpDecoder and SpdyHttpCodec
Motivation:

HTTP header validation can be expensive so we should allow to disable it like we do in HttpObjectDecoder.

Modification:

Add constructor argument to disable validation.

Result:
Performance improvement
2014-07-11 08:35:22 +02:00
Norman Maurer
d9d906c54e Reuse previous created HttpHeaders by HttpObjectAggregator
Motivation:

HttpObjectAggregator currently creates a new FullHttpResponse / FullHttpRequest for each message it needs to aggregate. While doing so it also creates 2 DefaultHttpHeader instances (one for the headers and one for the trailing headers). This is bad for two reasons:
  - More objects are created then needed and also populate the headers is not for free
  - Headers may get validated even if the validation was disabled in the decoder

Modification:

- Wrap the previous created HttpResponse / HttpRequest and so reuse the original HttpHeaders
- Reuse the previous created trailing HttpHeader.
- Fix a bug where the trailing HttpHeader was incorrectly mixed in the headers.

Result:

- Less GC
- Faster HttpObjectAggregator implementation
2014-07-11 06:41:03 +02:00
Norman Maurer
93b5e832b6 [#2643] Throw TooLongFrameException instead of using fireExceptionCaught
Motivation:

It's not always the case that there is another handler in the pipeline that will intercept the exceptionCaught event because sometimes users just sub-class. In this case the exception will just hit the end of the pipeline.

Modification:
Throw the TooLongFrameException so that sub-classes can handle it in the exceptionCaught(...) method directly.

Result:
Sub-classes can correctly handle the exception,
2014-07-10 06:57:08 +02:00
Norman Maurer
b440fa840b [#2647] Respect IOV_MAX when call writev in native transport
Motivation:

epoll transport fails on gathering write of more then 1024 buffers. As linux supports max. 1024 iov entries when calling writev(...) the epoll transport throws an exception.

Thanks again to @blucas to provide me with a reproducer and so helped me to understand what the issue is.

Modifications:

Make sure we break down the writes if to many buffers are uses for gathering writes.

Result:

Gathering writes work with any number of buffers
2014-07-09 13:32:43 +02:00
Brendt Lucas
5061be1b03 [#2642] CompositeByteBuf.deallocate memory/GC improvement
Motivation:

CompositeByteBuf.deallocate generates unnecessary GC pressure when using the 'foreach' loop, as a 'foreach' loop creates an iterator when looping.

Modification:

Convert 'foreach' loop into regular 'for' loop.

Result:

Less GC pressure (and possibly more throughput) as the 'for' loop does not create an iterator
2014-07-08 21:08:34 +02:00
Norman Maurer
21aa3d8997 [#2644] Correctly release buffer when exception happens during send DatagramPacket or SctpMessage
Motivation:

When an exception is thrown during try to send DatagramPacket or SctpMessage a buffer may leak.

Modification:

Correctly handle allocated buffers in case of exception

Result:

No more leaks
2014-07-08 20:15:33 +02:00
Norman Maurer
26c20c91bd Move generic code to HttpOrSpdyChooser to simplify implementations
Motivation:

HttpOrSpdyChooser can be simplified so the user not need to implement getProtocol(...) method.

Modification:

Add implementation for the method. The user can override it if necessary.

Result:

Easier usage of HttpOrSpdyChooser.
2014-07-07 09:34:33 +02:00
Trustin Lee
d0bc3be79c Add missing m2eclipse life cycle mapping
Also, use ignore instead of execution because those plugins are not
really useful when building from Eclipse anyway.
2014-07-06 16:53:53 +09:00
Trustin Lee
0623df12a5 Log the time taken for generating the initialSeedUniquifier
- Sometimes useful to know it how long it takes from the log, to make
  sure it's not something else that is blocking.
2014-07-04 13:27:18 +09:00
Norman Maurer
cd688c5cde [#2586] Use correct EventLoop to notify delayed bind failures
Motivation:

When a bind fails AbstractBootstrap will use the GlobalEventExecutor to notify the ChannelPromise. We should use the EventLoop of the Channel if possible.

Modification:

Use EventLoop of the Channel if possible to use the correct Thread to notify and so guaranteer the right order of events.

Result:

Use the correct EventLoop for notification
2014-07-03 21:24:15 +02:00
Trustin Lee
1464abe765 Let OkResponseHandler extend SimpleChannelInboundHandler
Motivation:

OkResponseHandler is the last handler in the pipeline of the HTTP CORS
example.  It is responsible for releasing all messages it handled.

Modification:

Extend SimpleChannelInboundHandler instead of
ChannelInboundHandlerAdapter

Result:

Fixed a leak
2014-07-03 18:13:03 +09:00
Trustin Lee
3d8d843198 Fix the build timeout when 'leak' profile is active
Motivation:

AbstractByteBufTest.testInternalBuffer() uses writeByte() operations to
populate the sample data.  Usually, this isn't a problem, but it starts
to take a lot of time when the resource leak detection level gets
higher.

In our CI machine, testInternalBuffer() takes more than 30 minutes,
causing the build timeout when the 'leak' profile is active (paranoid
level resource detection.)

Modification:

Populate the sample data using ThreadLocalRandom.nextBytes() instead of
using millions of writeByte() operations.

Result:

Test runs much faster when leak detection level is high.
2014-07-03 17:55:18 +09:00
Trustin Lee
0a8ff3b52d Fix most inspector warnings
Motivation:

It's good to minimize potentially broken windows.

Modifications:

Fix most inspector warnings from our profile
Update IntObjectHashMap

Result:

Cleaner code
2014-07-02 20:21:30 +09:00
Norman Maurer
893bc04eee [#2558] Define SO_REUSEPORT if not defined
Motivation:

Currently it is impossible to build netty on linux system that not define SO_REUSEPORT even if it is supported.

Modification:

Define SO_REUSEPORT if not defined.

Result:

Possible to build on more linux dists.
2014-07-02 09:39:27 +02:00
Norman Maurer
016bdfbf66 Correctly return from selector loop one a scheduled task is ready for processing
Motivation:

We use the nanoTime of the scheduledTasks to calculate the milli-seconds to wait for a select operation to select something. Once these elapsed we check if there was something selected or some task is ready for processing. Unfortunally we not take into account scheduled tasks here so the selection loop will continue if only scheduled tasks are ready for processing. This will delay the execution of these tasks.

Modification:

- Check if a scheduled task is ready after selecting
- also make a tiny change in NioEventLoop to not trigger a rebuild if nothing was selected because the timeout was reached a few times in a row.

Result:

Execute scheduled tasks on time.
2014-07-02 08:31:47 +02:00
Norman Maurer
e8f4def2a3 [maven-release-plugin] prepare for next development iteration 2014-06-30 14:31:08 +02:00
Norman Maurer
25e3c8ce3d [maven-release-plugin] prepare release netty-4.0.21.Final 2014-06-30 14:29:15 +02:00
Norman Maurer
8427aa723a [#2623] Release local references to guard against StackOverflow in JNI
Motivation:

When we do a (env*)->GetObjectArrayElement(...) call we may created many local references which will only be cleaned up once we exist the native method. Thus a lot of memory can be used and so a StackOverFlow may be triggered. Beside this the JNI specification only say that an implementation must cope with 16 local references.

Modification:

Call (env*)->ReleaseLocalRef(...) to release the resource once not needed anymore.

Result:

Less memory usage and guard against StackOverflow
2014-06-30 10:40:52 +02:00
Norman Maurer
6c47cc9711 [#2622] Correctly check reference count before try to work on the underlying memory
Motivation:

Because of how we use reference counting we need to check for the reference count before each operation that touches the underlying memory. This is especially true as we use sun.misc.Cleaner.clean() to release the memory ASAP when possible. Because of this the user may cause a SEGFAULT if an operation is called that tries to access the backing memory after it was released.

Modification:

Correctly check the reference count on all methods that access the underlying memory or expose it via a ByteBuffer.

Result:

Safer usage of ByteBuf
2014-06-30 07:10:12 +02:00
Jay
0332fd1589 Maintain decoder result in HttpObjectAggregator
Motivation:
DecodeResult is dropped when aggregate HTTP messages.

Modification:

Make sure we not drop the DecodeResult while aggregate HTTP messages.

Result:

Correctly include the DecodeResult for later processing.
2014-06-28 21:47:49 +02:00