Commit Graph

6205 Commits

Author SHA1 Message Date
Idel Pivnitskiy
c6b2c5a320 Implemented LZMA frame encoder
Motivation:

LZMA compression algorithm has a very good compression ratio.

Modifications:

- Added `lzma-java` library which implements LZMA algorithm.
- Implemented LzmaFrameEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Added tests to verify the LzmaFrameEncoder and how it can compress data for the next uncompression using the original library.

Result:

LZMA encoder which can compress data using LZMA algorithm.
2014-09-15 15:14:15 +02:00
Barber, Francis
b24a749ae8 Add support for ExtensionRegistryLite in ProtobufDecoder
Motivation:

ExtensionRegistry is a subclass of ExtensionRegistryLite.  The ProtobufDecoder
doesn't use the registry directly, it simply passes it through to the Protobuf
API.  The Protobuf calls in question are themselves written in terms
ExtensionRegistryLite not ExtensionRegistry.

Modifications:

Require ExtensionRegistryLite instead of ExtensionRegistry in ProtobufDecoder.

Result:

Consumers can use ExtensionRegistryLite with ProtobufDecoder.
2014-09-15 07:05:47 +02:00
Scott Mitchell
2cf6ed9460 HTTP/2 Unit Test Leak Fixes
Motivation:
The HTTP/2 tests do not always clean up ByteBuf resources reliably. There are issues with the refCnt, over allocating buffers, and potentially not waiting long enough to reclaim resources for stress tests.

Modifications:
Scrub the HTTP/2 unit tests for ByteBuf leaks.

Result:
Less leaks (hopefully none) in the HTTP/2 unit tests. No OOME from HTTP/2 unit tests.
2014-09-14 17:10:31 -04:00
Scott Mitchell
96a044fabe HTTP/2 Data Decompression
Motivation:
The HTTP/2 codec does not provide a way to decompress data. This functionality is supported by the HTTP codec and is expected to be a commonly used feature.

Modifications:
-The Http2FrameReader will be modified to allow hooks for decompression
-New classes which detect the decompression from HTTP/2 header frames and uses that decompression when HTTP/2 data frames come in
-New unit tests

Result:
The HTTP/2 codec will provide a means to support data decompression
2014-09-14 08:43:13 -04:00
Scott Mitchell
9b811a24e8 Correcting HttpDecoder SuppressWarnings parameter
Motiviation:
The HttpContentDecoder.getTargetContentEncoding has a SuppressWarnings(unused) on its parameter.
This should be SuppressWarnings(UnusedParameters).

Modifications:
SuppressWarnings(unused) -> SuppressWarnings(UnusedParameters)

Result:
Correctly suppressing warnings due to HttpContentDecoder.getTargetContentEncoding
2014-09-12 07:25:25 +02:00
Jakob Buchgraber
b72a05edb4 Shutdown Executor on MultithreadEventLoopGroup shutdown. Fixes #2837
Motivation:

Currently the Executor created by (Nio|Epoll)EventLoopGroup is not correctly shutdown.
This might lead to resource shortages, due to resources not being freed asap.

Modifications:

If (Nio|Epoll)EventLoopGroup create their internal Executor via a constructor
provided `ExecutorServiceFactory` object or via
MultithreadEventLoopGroup.newDefaultExecutorService(...) the ExecutorService.shutdown()
method will be called after (Nio|Epoll)EventLoopGroup is shutdown.

ExecutorService.shutdown() will not be called if the Executor object was passed
to the (Nio|Epoll)EventLoopGroup (that is, it was instantiated outside of Netty).

Result:

Correctly release resources on (Nio|Epoll)EventLoopGroup shutdown.
2014-09-12 12:17:29 +09:00
Jakob Buchgraber
3dba1ef8dd Add missing calls to childGroup().shutdownGraceFully()
Motivation:

The ServerBootrap's child group would not be shutdown.

Modification:

Add missing shutdownGracefully() call.

Result:

The child group is shutdown correctly.
2014-09-12 12:15:49 +09:00
Scott Mitchell
d022ac1016 HTTP/2 Cipher Suite Support
Motivation:
The HTTP/2 specification places restrictions on the cipher suites that can be used. There is no central place to pull the ciphers that are allowed by the specification, supported by different java versions, and recommended by the community.

Modifications:
-HTTP/2 will have a security utility class to define supported ciphers
-netty-handler will be modified to support filtering the supplied list of ciphers to the supported ciphers for the current SSLEngine

Result:
-Netty provides unified support for HTTP/2 cipher lists and ciphers can be pruned by currently supported ciphers
2014-09-11 10:19:48 -04:00
Brad Fritz
fbe75ed637 Correct javadoc typo in IdleStateHandler example code 2014-09-10 20:55:08 +02:00
Scott Mitchell
6c148456ba HTTP Content Decoder Cleanup Bug
Motiviation:
The HTTP content decoder's cleanup method is not cleaning up the decoder correctly.
The cleanup method is currently doing a readOutbound on the EmbeddedChannel but
for decoding the call should be readInbound.

Modifications:
-Change readOutbound to readInbound in the cleanup method

Result:
The cleanup method should be correctly releaseing unused resources
2014-09-10 08:48:10 -04:00
Norman Maurer
2ba9f824c2 Directly write CompositeByteBuf if possible without memory copy. Related to [#2719]
Motivation:

In linux it is possible to write more then one buffer withone syscall when sending datagram messages.

Modifications:

Not copy CompositeByteBuf if it only contains direct buffers.

Result:

More performance due less overhead for copy.
2014-09-10 14:33:29 +02:00
Norman Maurer
3fd3cae8c3 Fix buffer leak in DatagramUnicastTest caused by incorrect usage of CompositeByteBuf
Motivation:

Due incorrect usage of CompositeByteBuf a buffer leak was introduced.

Modifications:

Correctly handle tests with CompositeByteBuf.

Result:

No more buffer leaks
2014-09-10 11:43:11 +02:00
Norman Maurer
d5b9f58f1f Add support for sendmmsg(...) and so allow to write multiple DatagramPackets with one syscall. Related to [#2719]
Motivation:

On linux with glibc >= 2.14 it is possible to send multiple DatagramPackets with one syscall. This can be a huge performance win and so we should support it in our native transport.

Modification:

- Add support for sendmmsg by reuse IovArray
- Factor out ThreadLocal support of IovArray to IovArrayThreadLocal for better separation as we use IovArray also without ThreadLocal in NativeDatagramPacketArray now
- Introduce NativeDatagramPacketArray which is used for sendmmsg(...)
- Implement sendmmsg(...) via jni
- Expand DatagramUnicastTest to test also sendmmsg(...)

Result:

Netty now automatically use sendmmsg(...) if it is supported and we have more then 1 DatagramPacket in the ChannelOutboundBuffer and flush() is called.
2014-09-09 10:03:17 +02:00
Norman Maurer
f1f14f524a Allow to write CompositeByteBuf directly via EpollDatagramChannel. Related to [#2719]
Motivation:

On linux it is possible to use the sendMsg(...) system call to write multiple buffers with one system call when using datagram/udp.

Modifications:

- Implement the needed changes and make use of sendMsg(...) if possible for max performance
- Add tests that test sending datagram packets with all kind of different ByteBuf implementations.

Result:

Performance improvement when using CompoisteByteBuf and EpollDatagramChannel.
2014-09-09 09:50:42 +02:00
Norman Maurer
7e48801f71 [#2867] Workaround performance issue with IPv4-mapped-on-IPv6 addresses
Motivation:

InetAddress.getByName(...) uses exceptions for control flow when try to parse IPv4-mapped-on-IPv6 addresses. This is quite expensive.

Modifications:

Detect IPv4-mapped-on-IPv6 addresses in the JNI level and convert to IPv4 addresses before pass to InetAddress.getByName(...) (via InetSocketAddress constructor).

Result:

Eliminate performance problem causes by exception creation when parsing IPv4-mapped-on-IPv6 addresses.
2014-09-09 07:24:17 +02:00
Scott Mitchell
ec604a3464 Handler Unit Test Bug
Motivation:
There is a bug in the JettySslEngineTest where the interface receiving a message does not do a latch.countDown().
This causes each test to be subject to the CountDownLatch timeout period instead of being notified right when an event occurs.

Modifications:
- The JettySslEngineTest message handler will call the appropriate latch.countDown after a message is received

Result:
JettySslEngineTest will not be subject to waiting the timeout period even if the message is correctly received
2014-09-07 17:37:31 -04:00
nmittler
d35ddf0648 Fixing queuing bug in outbound flow control.
Motivation:

Outbound flow control does not properly remove the head of queue after
it's written. This will cause streams with multiple frames to get stuck
and not send all of the data.

Modifications:

Modified the DefaultHttp2OutboundFlowController to properly remove the
head of the pending write queue once a queued frame has been written.

Added an integration test that sends a large message to verify that all
DATA frames are properly collected at the other end.

Result:

Outbound flow control properly handles several queued messages.
2014-09-05 15:05:55 -07:00
Jakob Buchgraber
c40b0d2e07 Reset interrupted flag in SingleThreadEventLoopTest. Fixes #2841
Motivation:

Once the interrupted flag in SingleThreadEventLoopB was set it was never reset.
That way the eventloop thread was always in an interrupted state and all calls
to Thread.sleep/LockSupport.parkNanos would fail.

Modifications:

Reset interrupted flag.

Result:

Number of GC runs when executing the test is down to zero (from 20 - 50).
Looking at the heap dump shows >10x decrease in memory consumption e.g.
the number of ForkJoinTasks is down to 25k from 350k previously. Similar
is true for BlockingQueue.Node objects.
2014-09-01 16:33:15 +02:00
Norman Maurer
6104e44c76 [#2843] Add test-case to show correct behavior of ByteBuf.refCnt() and ByteBuf.release(...)
Motivation:

We received a bug-report that the ByteBuf.refCnt() does sometimes not show the correct value when release() and refCnt() is called from different Threads.

Modifications:

Add test-case which shows that all is working like expected

Result:

Test-case added which shows everything is ok.
2014-09-01 08:50:30 +02:00
Norman Maurer
a765753ae6 [#2847] Correctly encode HTTP to SPDY if X-SPDY-Associated-To-Stream-ID is not present
Motivation:

Because of a bug a NPE was thrown when tried to encode HTTP to SPDY and no X-SPDY-Associated-To-Stream-ID was present.

Modifications:

Use 0 as default value when X-SPDY-Associated-To-Stream-ID not present.

Result:

No NPE anymore.
2014-08-31 21:08:25 +02:00
Amir Szekely
bdd950dff8 Don't ignore maxCapacity if it's not a power of 2
Motivation:

This fixes bug #2848 which caused Recycler to become unbounded and cache infinite number of objects with maxCapacity that's not a power of two. This can result in general sluggishness of the application and OutOfMemoryError.

Modifications:

The test for maxCapacity has been moved out of test to check if the buffer has filled. The buffer is now also capped at maxCapacity and cannot grow over it as it jumps from one power of two to the other.

Additionally, a unit test was added to verify maxCapacity is honored even when it's not a power of two.

Result:

With these changes the user is able to use a custom maxCapacity number and not have it ignored. The unit test assures this bug will not repeat itself.
2014-08-31 09:06:10 +02:00
Norman Maurer
cc55a71b14 Correctly release buffers on protocol errors
Motivation:

We failed to release buffers on protocolErrors which caused buffer leaks when using HTTP/2

Modifications:

Release buffer on protocol errors

Result:

No more buffer leaks
2014-08-30 20:53:07 +02:00
Norman Maurer
ce8230f455 Correctly release buffer in DelegatingHttp2ConnectionHandlerTest
Motivation:

Because we did not release the buffer correctly in DelegatingHttp2ConnectionHandlerTest the CI failed because of a buffer leak.
See https://secure.motd.kr/jenkins/job/pr_2849_kichik_netty_master/4/consoleFull

Modifications:

Correctly release buffer

Result:

No more leak error.
2014-08-30 20:43:23 +02:00
Scott Mitchell
5f232b2220 ALPN java implementation
Motivation:

Netty only supports a java NPN implementation provided by npn-api and npn-boot.
There is no java implementation for ALPN.
ALPN is needed to be compliant with the HTTP/2 spec.

Modifications:
-SslContext and JdkSslContext to support ALPN
-JettyNpn* class restructure for NPN and ALPN common aspects
-Pull in alpn-api and alpn-boot optional dependencies for ALPN java implementation

Result:

-Netty provides access to a java implementation of APLN
2014-08-30 13:41:07 -04:00
Jakob Buchgraber
0d549706f1 Reduce Memory Usage of SingleThreadEventLoopTest
Motivation:

The SingleThreadEventLoopTest allocated up to half a gigabyte of memory per run,
causing up to 50 GC runs in ~20 seconds.

Modification:

Charlie Hunt identified TLAB allocations to be the root cause of this excessive
memory usage [1]. By reusing the Executor in every test we can reduce the
memory usage by >50%.

Result:

Lower Memory Usage and fewer Garbage Collector runs.
Helps to resolve GitHub issue #2841.

[1] https://twitter.com/charlesjhunt/status/505351389317722112
2014-08-29 19:31:31 +02:00
Norman Maurer
3b114a51df Cleanup test
Motivation:

Saw a lot of inspector warnings

Modifications:

Fix inspector warnings

Result:

Cleaner code
2014-08-29 16:01:54 +02:00
Norman Maurer
5b0f60cff9 [#2841] Fix SingleThreadEventLoopTest that was failing because of GC pressure
Motivation:

Sometimes the SingleThreadEventLoopTest did fail on our CI. This was because of GC pressure produced by Thread.sleep(...) when interrupted as it creates a new InterruptedException all the time (and need to fill it).

Modifications:

Replace Thread.sleep(...) with LockSupport.parkNanos(...) to eliminate exception overhead.

Result:

SingleThreadEventLoopTest produce a lot less garbage.
2014-08-29 14:57:48 +02:00
nmittler
4599b7a0ba Fixing HTTP/2 data write completion bug
Motivation:

A recent refactoring of the outbound flow controller interface
introduced a bug when writing data. We're no longer properly handling
the completion of the write (i.e. updating stream state/handling error).

Modifications:

Updated AbstractHttp2ConnectionHandler.writeData to properly handle the
completion of the write future.

Result:

DATA writes now perform post-write cleanup.
2014-08-28 09:58:00 -07:00
Scott Mitchell
2128753484 Exclude bin directory from git Motivation:
The 'bin/' directory is currently not in the .gitignore file.
Eclipse creates this directory to support IDE operations.
These directory is generated and not related to source code and therefore does not belong in git.

Modifications:
-Add 'bin/' to .gitignore

Result:
-No more 'bin/' directory confusion for eclipse users.
2014-08-27 06:33:22 +02:00
nmittler
102e781715 Renaming "Observer" to "Listener" for consistency in HTTP/2
Motivation:

We currently have a mix of "Observer" and "Listener" for interface
names. We should make them all "Listener" to be consistent.

Modifications:

Renamed Http2DataObserver->Http2DataListener and
Http2FrameObserver->Http2FrameListener.

Result:

Listener interface names are consistent.
2014-08-26 18:52:05 -07:00
Scott Mitchell
63d5925d6e HTTP/2 to HTTP priority translation
Motivation:

The priority information reported by the HTTP/2 to HTTP tranlsation layer is not correct in all situations.
The HTTP translation layer is not using the Http2Connection.Listener interface to track tree restructures.
This incorrect information is being sent up to clients and is misleading.

Modifications:
-Restructure InboundHttp2ToHttpAdapter to allow a default data/header mode
-Extend this interface to provide an optional priority translation layer

Result:
-Priority information being correctly reported in HTTP/2 to HTTP translation layer
-Cleaner code with seperation of concerns (optional priority conversion).
2014-08-26 14:06:02 -07:00
nmittler
6409a5a1d5 Adding window maintenance flag to HTTP/2 inbound flow control
Motivation:

Currently, window maintenance is automatically performed when a flow
control window drops below half its initial size. We should provide a
way for advanced applications to determine whether or not this should be
done on a per-stream basis.

Modifications:

Modifying DefaultHttp2InboundFlowController to allow enabling/disabling
of window maintenance per stream.

Result:

Inbound flow control window maintenance will be dynamically
configurable.
2014-08-26 12:02:05 -07:00
Norman Maurer
e166780f0b [#2823] Writing DefaultFileRegion with EpollSocketChannel may cause hang
Motivation:

In EpollSocketchannel.doWriteFileRegion(...) we need to make sure we write until sendFile(...) returns either 0 or all is written. Otherwise we may not get notified once the Channel is writable again.

This is the case as we use EPOLL_ET.

Modifications:

Always write until either sendFile returns 0 or all is written.

Result:

No more hangs when writing DefaultFileRegion can happen.
2014-08-26 15:09:27 +02:00
Norman Maurer
0b5df22aec [#2797] Let EventExecutorGroup extend AutoCloseable
Motivation:

To make it more easy to shutdown an EventExecutorGroup / EventLoopGroup we should let both of them extend AutoCloseable.

Modifications:

Let EventExecutorGroup extend AutoCloseable and impement it.

Result:

Easier shutdown of EventExecutorGroup and EventLoopGroup
2014-08-26 14:04:27 +02:00
Jakob Buchgraber
0381fa67a8 Undo AbstractChannelHandlerContext extend PausableChannelEventExecutor. Fixes #2814
Motivation:
Our last-minute change that made AbstractChannelHandlerContext implement PausableChannelEventExecutor broke the socksproxy example.

Modifications:
AbstractChannelHandlerContext does not inherit from PausableChannelEventExecutor anymore. Instead we'll allocate an extra object on demand.

Result:

AbstractChannelHandlerContext.executor().newPromise() returns the correct type.
2014-08-26 13:50:40 +02:00
Trustin Lee
de6e73e3ea Check noJdkZlibEncoder before comparing other parameters
.. for consistency
2014-08-26 16:12:38 +09:00
Trustin Lee
239371abaf Add io.netty.noJdkZlibEncoder system property
Related issue: #2821

Motivation:

There's no way for a user to change the default ZlibEncoder
implementation.

It is already possible to change the default ZlibDecoder implementation.

Modification:

Add a new system property 'io.netty.noJdkZlibEncoder'.

Result:

A user can disable JDK ZlibEncoder, just like he or she can disable JDK
ZlibDecoder.
2014-08-26 15:49:29 +09:00
Scott Mitchell
892944eba8 HTTP/2 draft 14 HTTP message flow
Motivation:

HTTP/2 draft 14 came out a couple of weeks ago and we need to keep up
with the spec.

Modifications:
-Revert back to dispatching FullHttpMessage objects instead of individual HttpObjects
-Corrections to HttpObject comparitors to support test cases
-New test cases to support sending headers immediatley
-Bug fixes cleaned up to ensure the message flow is terminated properly

Result:
Netty HTTP/2 to HTTP/1.x translation layer will support the HTTP/2 draft message flow.
2014-08-25 11:34:15 -07:00
nmittler
5b1d50fa7c Changing HTTP/2 inbound flow control to use Http2FrameWriter
Motivation:

This is just some general cleanup to get rid of the FrameWriter inner
interface withing Http2InboundFlowController.  It's not necessary since
the flow controller can just use the Http2FrameWriter to send
WINDOW_UPDATE frames.

Modifications:

Updated DefaultHttp2InboundFlowController to use Http2FrameWriter.

Result:

The inbound flow control code is somewhat less smelly :).
2014-08-25 07:28:07 -07:00
Norman Maurer
8785ff5256 Move duplicated code into CodecUtil
Motivation:

We have some duplicated code that can be reused.

Modifications:

Create package private class called CodecUtil that now contains the shared code / helper method.

Result:

Less code-duplication
2014-08-25 07:37:35 +02:00
Norman Maurer
488f36d009 [#2812] Ensure we call checkForSharableAnnotation in all constructors of ByteToMessageCodec
Motivation:

ByteToMessageCodec miss to check for @Sharable annotation in one of its constructors.

Modifications:

Ensure we call checkForSharableAnnotation in all constructors.

Result:

After your change, what will change.
2014-08-23 21:03:57 +02:00
nmittler
3ffd205f57 Adding padding to HTTP/2 flow control.
Motivation:

We're currently out-of-spec with HTTP/2 in that we don't include padding
in the flow control logic.

Modifications:

Modified both DefaultHttp2InboundFlowController and
DefaultHttp2OutboundFlowController to properly take padding into
account. Outbound is more complicated since padding has to be properly
accounted for when splitting frames.

Result:

HTTP/2 codec properly flow controls padding.
2014-08-22 13:22:21 -07:00
Scott Mitchell
d3538dee2e HTTP/2 Priority Tree Restructure
Motivation:

The current interface and implementation for HTTP/2 priority tree events does not notify listeners of all parent change events. As a result operations which depend upon knowing about parent change events may be missing events and result in stale data. This interface also allows new listeners to easily consume priority tree change events.

Modifications:

-Http2Connection.Listener interface will change to support notifications on every node after the priority has changed and tree events have settled
-This will affect the outbound flow controller, DefaultHttp2Connection, and other listeners using the old interface

Result:
A modified (hopefully simplified and correct) Listener interface to get priority tree change event notification
2014-08-22 13:07:22 -07:00
nmittler
21bc279700 Refactoring outbound flow controller to use frame writer.
Motivation:

This is addressing a TODO in the outbound flow controller. We currently
have a separate writer interface passed into the outbound flow
controller. This is confusing and limiting as to how the flow controller
can perform its writes (e.g. no control over flushing). Instead it would
be better to just let the flow controller use the Http2FrameWriter
directly.

Modifications:

- Added a new Http2DataWriter interface, which is extended by
Http2FrameWriter and Http2OutboundFlowController.

- Removed automatic flushing from Http2DataWriter in order to facilitate
optimizing the case where there are multiple writes.

- Updated DefaultHttp2OutboundFlowController to properly optimize
flushing of the ChannelHandlerContext when multiple writes occur.

Result:

Code is greatly simplified WRT outbound flow control and flushes are
optimized for flow-controlled DATA frames.
2014-08-21 11:49:12 -07:00
Norman Maurer
d332c00e2f Reduce memory copies in spdy compression implementation.
Motivation:

Currently we do more memory copies then needed.

Modification:

- Directly use heap buffers to reduce memory copy
- Correctly release buffers to fix buffer leak

Result:

Less memory copies and no leaks
2014-08-21 11:32:36 +02:00
Norman Maurer
ee198d7cfc Allow efficient writing of CompositeByteBuf when using native epoll transport.
Motivation:

There were no way to efficient write a CompositeByteBuf as we always did a memory copy to a direct buffer in this case. This is not needed as we can just write a CompositeByteBuf as long as all the components are buffers with a memory address.

Modifications:

- Write CompositeByteBuf which contains only direct buffers without memory copy
- Also handle CompositeByteBuf that have more components then 1024.

Result:

More efficient writing of CompositeByteBuf.
2014-08-21 10:58:42 +02:00
Norman Maurer
a32652197b Avoid redundant reads of head in peakNode
Motivation:

There is not need todo redunant reads of head in peakNode as we can just spin on next() until it becomes visible.

Modifications:

Remove redundant reads of head in peakNode. This is based on @nitsanw's patch for akka.
See https://github.com/akka/akka/pull/15596

Result:

Less volatile access.
2014-08-21 09:01:34 +02:00
Norman Maurer
16a8f8b5aa Code-inspection fixes
Motivation:

Saw some code-inspection warnings

Modifications:

Fix warnings

Result:

Less code-inspection warnings
2014-08-21 07:20:00 +02:00
Norman Maurer
ceaab70153 [#2800] Fix NPE in SingleThreadEventExecutor on Android
Motivation:

The recent changes f8bee2e94c to use a FJP introduced a regression on Android that cause a NPE.

Modifications:

- Use AtomicReferenceFieldUpdater so it works also on Android
- Fix inspection warnings

Result:

Netty works again on Android too.
2014-08-21 06:31:56 +02:00
Norman Maurer
62e26a2818 [#2586] Use correct EventExecutor to notify for bind failures on late registration
Motivation:

We used the wrong EventExecutor to notify for bind failures if a late registration was done.

Modifications:

Use the correct EventExecutor to notify and only use the GlobelEventExecutor if the registration fails itself.

Result:

The correct Thread will do the notification.
2014-08-20 16:35:06 +02:00