Commit Graph

6119 Commits

Author SHA1 Message Date
Idel Pivnitskiy
0b307fe083 Implemented FastLZ compression codec
Motivation:

FastLZ compression codec provides sending and receiving data encoded by fast FastLZ algorithm using block mode.

Modifications:

- Added part of `jfastlz` library which implements FastLZ algorithm. See FastLz class.
- Implemented FastLzFramedEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Implemented FastLzFramedDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages.
- Added integration tests for `FastLzFramedEncoder/Decoder`.

Result:

Full FastLZ compression codec which can compress/uncompress data using FastLZ algorithm.
2014-08-12 15:15:54 -07:00
Scott Mitchell
b41b11c53d Introduced HTTP/2 frame to HTTP/1.x translation layer.
Motivation:

The HTTP/2 codec currently provides direct callbacks to access stream events/data. The HTTP/2 codec provides the protocol support for HTTP/2 but it does not pass messages up the context pipeline. It would be nice to have a decoder which could collect the data framed by HTTP/2 and translate this into traditional HTTP type objects. This would allow the traditional Netty context pipeline to be used to separate processing concerns (i.e. HttpContentDecompressor).  It would also be good to have a layer which can translate FullHttp[Request|Response] objects into HTTP/2 frame outbound events.

Modifications:

Introduce a new InboundHttp2ToHttpAdapter and supporting classes which will translate HTTP/2 stream events/data into HttpObject objects. Introduce a new DelegatingHttp2HttpConnectionHandler which will translate FullHttp[Request|Response] objects to HTTP/2 frame events.

Result:

Introduced HTTP/2 frame events to HttpObject layer.
Introduced FullHttp[Request|Response] to HTTP/2 frame events.
Introduced new unit tests to support new code.
Updated HTTP/2 client example to use new code.
Miscelaneous updates and bug fixes made to support new code.
2014-08-12 12:31:26 -07:00
Trustin Lee
9b755d79e7 Do not throw an exception when failed to get a header
Motivation:

It is often very expensive to instantiate an exception. TextHeader
should not raise an exception when it failed to find a header or when
its header value is not valid.

Modification:

- Change the return type of the getter methods to Integer and Long so
  that null is returned when no header is found or its value is invalid
- Update Javadoc

Result:

- Fixes #2758
- No unnecessary instantiation of exceptions
2014-08-12 11:15:11 -07:00
jxu
7607eb8761 Add TextHeaders.getAndRemove(...) and its variants
Related issue: #2649 and #2745

Motivation:

At the moment there is no way to get and remove a header with one call.
This means you need to search the headers two times. We should add
getAndRemove(...) to allow doing so with one call.

Modifications:

Add getAndRemove(...) and getUnconvertedAndRemove(...) and their
variants

Result:

More efficient API
2014-08-12 10:33:17 -07:00
Norman Maurer
c5b6808645 Allow to obtain RecvByteBufAllocator.Handle to allow more flexible implementations
Motivation:

At the moment it's only possible for a user to set the RecvByteBufAllocator for a Channel but not access the Handle once it is assigned. This makes it hard to write more flexible implementations.

Modifications:

Add a new method to the Channel.Unsafe to allow access the the used Handle for the Channel. The RecvByteBufAllocator.Handle is created lazily.

Result:

It's possible to write more flexible implementatons that allow to adjust stuff on the fly for a Handle that is used by a Channel
2014-08-12 06:54:29 +02:00
Norman Maurer
7764b0e3de [#2752] Add PendingWriteQueue for queue up writes
Motivation:

Sometimes ChannelHandler need to queue writes to some point and then process these. We currently have no datastructure for this so the user will use an Queue or something like this. The problem is with this Channel.isWritable() will not work as expected and so the user risk to write to fast. That's exactly what happened in our SslHandler. For this purpose we need to add a special datastructure which will also take care of update the Channel and so be sure that Channel.isWritable() works as expected.

Modifications:

- Add PendingWriteQueue which can be used for this purpose
- Make use of PendingWriteQueue in SslHandler

Result:

It is now possible to queue writes in a ChannelHandler and still have Channel.isWritable() working as expected. This also fixes #2752.
2014-08-12 06:41:08 +02:00
Jakob Buchgraber
220660e351 Deregistration of a Channel from an EventLoop
Motivation:

After a channel is created it's usually assigned to an
EventLoop. During the lifetime of a Channel the
EventLoop is then responsible for processing all I/O
and compute tasks of the Channel.

For various reasons (e.g. load balancing) a user might
require the ability for a Channel to be assigned to
another EventLoop during its lifetime.

Modifications:

Introduce under the hood changes that ensure that Netty's
thread model is obeyed during and after the deregistration
of a channel.

Ensure that tasks (one time and periodic) are executed by
the right EventLoop at all times.

Result:

A Channel can be deregistered from one and re-registered with
another EventLoop.
2014-08-11 15:28:46 -07:00
Jakob Buchgraber
f8bee2e94c Make Nio/EpollEventLoop run on a ForkJoinPool
Related issue: #2250

Motivation:

Prior to this commit, Netty's non blocking EventLoops
were each assigned a fixed thread by which all of the
EventLoop's I/O and handler logic would be performed.

While this is a fine approach for most users of Netty,
some advanced users require more flexibility in
scheduling the EventLoops.

Modifications:

Remove all direct usages of threads in MultithreadEventExecutorGroup,
SingleThreadEventExecutor et al., and introduce an Executor
abstraction instead.

The way to think about this change is, that each
iteration of an eventloop is now a task that gets scheduled
in a ForkJoinPool.

While the ForkJoinPool is the default, one also has the
ability to plug in his/her own Executor (aka thread pool)
into a EventLoop(Group).

Result:

Netty hands off thread management to a ForkJoinPool by default.
Users can also provide their own thread pool implementation and
get some control over scheduling Netty's EventLoops
2014-08-11 15:28:46 -07:00
Trustin Lee
59cf8ffcb4 Fix a resource leak in StompSubframeAggregatorTest 2014-08-11 10:46:17 -07:00
Jeff Pinner
f7e183f8e4 SPDY: fix SpdySessionHandler::updateSendWindowSize
In Netty 3, downstream writes of SPDY data frames and upstream reads of
SPDY window udpate frames occur on different threads.

When receiving a window update frame, we synchronize on a java object
(SpdySessionHandler::flowControlLock) while sending any pending writes
that are now able to complete.

When writing a data frame, we check the send window size to see if we
are allowed to write it to the socket, or if we have to enqueue it as a
pending write. To prevent races with the window update frame, this is
also synchronized on the same SpdySessionHandler::flowControlLock.

In Netty 4, upstream and downstream operations on any given channel now
occur on the same thread. Since java locks are re-entrant, this now
allows downstream writes to occur while processing window update frames.

In particular, when we receive a window update frame that unblocks a
pending write, this write completes which triggers an event notification
on the response, which in turn triggers a write of a data frame. Since
this is on the same thread it re-enters the lock and modifies the send
window. When the write completes, we continue processing pending writes
without knowledge that the window size has been decremented.
2014-08-11 11:13:22 +02:00
Trustin Lee
3fc7518367 Fix resource leaks in StompSubframeDecoderTest 2014-08-08 11:26:10 -07:00
Norman Maurer
7022dcd4a1 [#2744] Fix flakey HashedWheelTimerTest.testExecutionOnTime()
Motivation:

The calculation of the max wait time for HashedWheelTimerTest.testExecutionOnTime() was wrong and so the test sometimes failed.

Modifications:

Fix the max wait time.

Result:

No more test-failures
2014-08-06 07:01:24 +02:00
Trustin Lee
653262d2af Fix resource leaks in StompSubframeAggregatorTest 2014-08-05 18:11:45 -07:00
Trustin Lee
168e81e9cd Fix a bug where SpdySession.getActiveStreams() returns incorrect set
Related issue: #2743

Motivation:

When there are more than one stream with the same priority, the set
returned by SpdySession.getActiveStream() will not include all of them,
because it uses TreeSet and only compares the priority of streams. If
two different streams have the same priority, one of them will be
discarded by TreeSet.

Modification:

- Rename getActiveStreams() to activeStreams()
- Replace PriorityComparator with StreamComparator

Result:

Two different streams with the same priority are compared correctly.
2014-08-05 17:46:13 -07:00
Trustin Lee
6ecca2722e Add test cases for HttpContentCompressor
- Ported from 386a06dbfa
2014-08-05 15:48:14 -07:00
Idel Pivnitskiy
12c614a7eb Consider writerIndex when LzfDecoder writes into a new heap buffer
Motivation:

Now LzfDecoder do not consider writerIndex when it writes into array of a new heap buffer (when it decodes a compressed chuck of data)
2014-08-05 22:51:21 +02:00
Trustin Lee
e59ec5384d Fix a bug where ChannelFuture.setFailure(null) doesn't fail
Motivation:

We forgot to do a null check on the cause parameter of
ChannelFuture.setFailure(cause)

Modifications:

Add a null check

Result:

Fixed issue: #2728
2014-08-05 11:23:52 -07:00
Norman Maurer
4a3ef90381 Port ChannelOutboundBuffer and related changes from 4.0
Motivation:

We did various changes related to the ChannelOutboundBuffer in 4.0 branch. This commit port all of them over and so make sure our branches are synced in terms of these changes.

Related to [#2734], [#2709], [#2729], [#2710] and [#2693] .

Modification:
Port all changes that was done on the ChannelOutboundBuffer.

This includes the port of the following commits:
 - 73dfd7c01b
 - 997d8c32d2
 - e282e504f1
 - 5e5d1a58fd
 - 8ee3575e72
 - d6f0d12a86
 - 16e50765d1
 - 3f3e66c31a

Result:
 - Less memory usage by ChannelOutboundBuffer
 - Same code as in 4.0 branch
 - Make it possible to use ChannelOutboundBuffer with Channel implementation that not extends AbstractChannel
2014-08-05 15:00:56 +02:00
Norman Maurer
2258b32549 [#2732] HttpRequestEncoder may produce invalid uri if uri parameters are included.
Motivation:

If the requests contains uri parameters but not path the HttpRequestEncoder does produce an invalid uri while try to add the missing path.

Modifications:

Correctly handle the case of uri with paramaters but no path.

Result:

HttpRequestEncoder produce correct uri in all cases.
2014-08-05 10:14:06 +02:00
Trustin Lee
730bfd4fdb Add more utility methods to check the availability of the epoll transport
Related issue: #2733

Motivation:

Unlike OpenSsl, Epoll lacks a couple useful availability checker
methods:

- ensureAvailability()
- unavailabilityCause()

Modifications:

Add missing methods

Result:

More ways to check the availability and to get the cause of
unavailability programatically.
2014-08-04 15:05:13 -07:00
Trustin Lee
2506dc778c Clean-up d9cccccbb3
- Revert irrelevant formatting changes
- Rename resource files
  - Add .pem
  - Remove 'netty' from names
2014-08-04 10:54:13 -07:00
Trustin Lee
2b12640960 More brief somaxconn logging
- Consistent log message format
- Avoid unnecessary autoboxing when debug level is off
- Remove the duplication of somaxconn path
2014-08-04 10:27:22 -07:00
Peter Schulz
75b5b26ba3 [#2718] Added private key decryption to JDK SSL server context.
Motivation:

Currently it is not possible to load an encrypted private key when
creating a JDK based SSL server context.

Modifications:

- Added static method to JdkSslServerContext which handles key spec generation for (encrypted) private keys and make use of it.
-Added tests for creating a SSL server context based on a (encrypted)
private key.

Result:

It is now possible to create a JDK based SSL server context with an
encrypted (password protected) private key.
2014-08-04 14:09:26 +02:00
Norman Maurer
63b9fff963 Fix buffer leaks in DnsResponseDecoder and DnsResponseDecoderTest
Motivation:

There were two buffer leaks in the codec-dns.

Modifications:

- Fix buffer leak in DnsResponseTest.readResponseTest()
- Correctly release DnsResources on Exception

Result:

No more buffer leaks in the codec-dns module.
2014-08-04 14:05:02 +02:00
Idel Pivnitskiy
7791fbf0e2 Improve Bzip2BitReader/Writer
Motivation:

Before this changes Bzip2BitReader and Bzip2BitWriter accessed to ByteBuf byte by byte. So tests for Bzip2 compression codec takes a lot of time if we ran them with paranoid level of resource leak detection. For more information see comments to #2681 and #2689.

Modifications:

- Increased size of bit buffers from 8 to 64 bits.
- Improved reading and writing operations.
- Save link to incoming ByteBuf inside Bzip2BitReader.
- Added methods to check possible readable bits and bytes in Bzip2BitReader.
- Updated Bzip2 classes to use new API of Bzip2BitReader.
- Added new constants to Bzip2Constants.

Result:

Increased size of bit buffers and improved performance of Bzip2 compression codec (for general work by 13% and for tests with paranoid level of resource leak detection by 55%).
2014-08-04 08:01:19 +02:00
Norman Maurer
bc92ac6ba3 Remove duplicated code 2014-07-31 18:27:26 -07:00
Norman Maurer
045f9ff0ed [#2720] Check if /proc/sys/net/core/somaxconn exists before try to parse it
Motivation:

As /proc/sys/net/core/somaxconn does not exists on non-linux platforms you see a noisy stacktrace when debug level is enabled while the static method of NetUtil is executed.

Modifications:

Check if the file exists before try to parse it.

Result:

Less noisy logging on non-linux platforms.
2014-07-31 18:09:08 -07:00
Trustin Lee
c5ae3e3dcb Use our own URL shortener wherever possible 2014-07-31 17:06:30 -07:00
Trustin Lee
e6193fe0ef Remove duplicate range check in AbstractByteBuf.skipBytes() 2014-07-29 15:57:58 -07:00
Trustin Lee
1c9cb90655 Fix a ConstantPoolTest failure 2014-07-29 15:45:50 -07:00
Trustin Lee
e64644b21c Fix a bug where AbstractConstant.compareTo() returns 0 for different constants
Related issue: #2354

Motivation:

AbstractConstant.compareTo() can return 0 even if the specified constant
object is not the same instance with 'this'.

Modifications:

- Compare the identityHashCode of constant first. If that fails,
  allocate a small direct buffer and use its memory address as a unique
  value.  If the platform does not provide a way to get the memory
  address of a direct buffer, use a thread-local random value.
- Signal cannot extend AbstractConstant. Use delegation.

Result:

It is practically impossible for AbstractConstant.compareTo() to return
0 for different constant objects.
2014-07-29 14:56:28 -07:00
Norman Maurer
410b5623ff Use correct exception message when throw exception from native code
Motivation:

We sometimes not use the correct exception message when throw it from the native code.

Modifications:

Fixed the message.

Result:

Correct message in exception
2014-07-28 13:34:09 -07:00
Norman Maurer
148b23b28b Fix broken test after change the maximal value of the pid. Part of [#2706] 2014-07-28 10:39:01 -07:00
Norman Maurer
d82cc29d0c [#2706] Allow pid up to 4194304
Motivation:

The PID_MAX_LIMIT on 64bit linux systems is 4194304 and on osx it is 99998. At the moment we use 65535 as an upper-limit which is too small.

Modifications:

Use 4194304 as max possible value

Result:

No more false-positives when try to detect current pid.
2014-07-28 10:16:25 -07:00
Norman Maurer
88c8a4f393 [#2708] DnsResource.duplicate() should return DnsResource and not ByteBufHolder
Motivation:

DnsResource.duplicate() should return DnsResource and not ByteBufHolder

Modifications:

Change return type from ByteBufHolder to DnsResource

Result:

No need to cast to the correct type when using duplicate()
2014-07-28 04:24:26 -07:00
Norman Maurer
2131edadf2 [#2692] Allows notify ChannelFutureProgressListener on complete writes
Motivation:

We have some inconsistency when handling writes. Sometimes we call ChannelOutboundBuffer.progress(...) also for complete writes and sometimes not. We should call it always.

Modifications:

Correctly call ChannelOuboundBuffer.progress(...) for complete and incomplete writes.

Result:

Consistent behavior
2014-07-28 04:20:04 -07:00
Norman Maurer
30295138fe Correctly write single ByteBuf with memoryAddress
Motivation:

While optimize gathering writes I introduced a bug when writing single ByteBuf that have a memoryAddress. This regression was introduced by 88bd6e7a93.

Modifications:

Correctly use the writerIndex as argument when call Native.writeAddress(...)

Result:

No more corruption while write single buffers.
2014-07-25 17:29:31 +02:00
Norman Maurer
ca0571cc9d Optimize native transport for gathering writes
Motivation:

While benchmarking the native transport with gathering writes I noticed that it is quite slow. This is due the fact that we need to do a lot of array copies to get the buffers into the iov array.

Modification:

Introduce a new class calles IovArray which allows to fill buffers directly in a iov array that can be passed over to JNI without any array copies. This gives a nice optimization in terms of speed when doing gathering writes.

Result:

Big performance improvement when doing gathering writes. See the included benchmark...

Before:
[nmaurer@xxx]~% wrk/wrk -H 'Host: localhost' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -d 120 -c 256 -t 16 --pipeline 256  http://xxx:8080/plaintext
Running 2m test @ http://xxx:8080/plaintext
  16 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.44ms   16.37ms 259.57ms   91.77%
    Req/Sec   181.99k    31.69k  304.60k    78.12%
  346544071 requests in 2.00m, 46.48GB read
Requests/sec: 2887885.09
Transfer/sec:    396.59MB

With this change:
[nmaurer@xxx]~% wrk/wrk -H 'Host: localhost' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -d 120 -c 256 -t 16 --pipeline 256  http://xxx:8080/plaintext
Running 2m test @ http://xxx:8080/plaintext
  16 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.93ms   16.33ms 305.73ms   92.34%
    Req/Sec   194.56k    33.75k  309.33k    77.04%
  369617503 requests in 2.00m, 49.57GB read
Requests/sec: 3080169.65
Transfer/sec:    423.00MB
2014-07-25 09:54:41 +02:00
Norman Maurer
44ea769f53 [#2662] Fix race in cancellation of TimerTasks which could let to NPE
Motivation:

Due some race-condition while handling canellation of TimerTasks it was possibleto corrupt the linked-list structure that is represent by HashedWheelBucket and so produce a NPE.

Modification:

Fix the problem by adding another MpscLinkedQueue which holds the cancellation tasks and process them on each tick. This allows to use no synchronization / locking at all while introduce a latency of max 1 tick before the TimerTask can be GC'ed.

Result:

No more NPE
2014-07-25 06:29:28 +02:00
Norman Maurer
581374c111 Fix compile error 2014-07-24 14:52:04 +02:00
Norman Maurer
3bcd243ffc [#2705] Call fireChannelReadComplete() if channelActive(...) decodes messages in ReplayingDecoder / ByteToMessageDecoder
Motivation:

In ReplayingDecoder / ByteToMessageDecoder channelInactive(...) method we try to decode a last time and fire all decoded messages throw the pipeline before call ctx.fireChannelInactive(...). To keep the correct order of events we also need to call ctx.fireChannelReadComplete() if we read anything.

Modifications:

- Channel channelInactive(...) to call ctx.fireChannelReadComplete() if something was decoded
- Move out.recycle() to finally block

Result:

Correct order of events.
2014-07-24 14:38:54 +02:00
Willem Jiang
48b7c25e00 Updated the ChannelGroup JavaDoc by removing b.releaseExternalResources(); 2014-07-24 10:54:38 +02:00
Trustin Lee
2c1d8584db Fix a compilation error 2014-07-23 14:56:20 -07:00
Trustin Lee
00295b23c8 Overall cleanup of codec-dns
Related issue: #2688

- DnsClass and DnsType
  - Make DnsClass and DnsType implement Comparable
  - Optimize the message generation of IllegalArgumentException,
    by pre-populating the list of the expected parameters
  - Move the static methods up
  - Relax the validation rule of DnsClass so that a user can define a
    CLASS which is not listed in the RFC 1035
  - valueOf(int) does not throw IllegalArgumentException anymore as long
    as the specified value is an unsigned short.
  - Rename create() and forName() to valueOf() so that they look like a
    real enum
  - Rename type() and clazz() to intValue() so that they conform to our
    naming convention
- Add missing null checks in DnsEntry
2014-07-23 14:40:36 -07:00
Tim Boudreau
9605029323 Use value types for class and type in DNS entries to make them immune to parameter order bugs
Motivation:

DNS class and type were represented as integers rather than an enum or a
similar dedicated value type.  This can be a potential source of a
parameter order bug which might be difficult to track down.

Modifications:

Add DnsClass and DnsType to replace integer parameters

Result:

Type safety and less error-proneness
2014-07-23 14:40:36 -07:00
nmittler
46a6312cba Remove compressed flag from HTTP/2
Motivation:

There are still a few places in the HTTP/2 code that have the compressed
flag (from pre-draft 13). Need to remove this flag since it's no longer
used.

Modifications:

Various changes to remove the flag from the writing path.

Result:

No references to the compressed flag.
2014-07-23 19:47:11 +02:00
Idel Pivnitskiy
1a8b29cfb7 Refactor Bzip2 tests
Motivation:

Complicated code of Bzip2 tests with some unnecessary actions.

Modifications:

- Reduce size of BYTES_LARGE array of random test data for Bzip2  tests.
- Removed unnecessary creations of EmbeddedChannel instances in Bzip2 tests.
- Simplified tests in Bzip2DecoderTest which expect exception.
- Removed unnecessary testStreamInitialization() from Bzip2EncoderTest.

Result:

Reduced time to test the 'codec' package by 7 percent, simplified code of Bzip2 tests.
2014-07-23 19:45:46 +02:00
Idel Pivnitskiy
42e1cc23fe Refactor integration tests of compression codecs
Motivation:

Duplicated code of integration tests for different compression codecs.

Modifications:

- Added abstract class IntegrationTest which contains common tests for any compression codec.
- Removed common tests from Bzip2IntegrationTest and LzfIntegrationTest.
- Implemented abstract methods of IntegrationTest in Bzip2IntegrationTest, LzfIntegrationTest and SnappyIntegrationTest.

Result:

Removed duplicated code of integration tests for compression codecs and simplified an addition of integration tests for new compression codecs.
2014-07-23 19:44:27 +02:00
Trustin Lee
7e362277b9 Reduce the default initial capacity of ChannelOutboundBuffer
Motivation:

ChannelOutboundBuffer is basically a circular array queue of its entry
objects.  Once an entry is created in the array, it is never nulled out
to reduce the allocation cost.

However, because it is a circular queue, the array almost always ends up
with as many entry instances as the size of the array, regardless of the
number of pending writes.

At worst case, a channel might have only 1 pending writes at maximum
while creating 32 entry objects, where 32 is the initial capacity of the
array.

Modifications:

- Reduce the initial capacity of the circular array queue to 4.
- Make the initial capacity of the circular array queue configurable

Result:

We spend 4 times less memory for entry objects under certain
circumstances.
2014-07-22 13:32:01 -07:00
Idel Pivnitskiy
0062a871f4 Simplify Bzip2 tests
Motivation:

Sometimes we have a 'build time out' error because tests for bzip2 codec take a long time.

Modifications:

Removed cycles from Bzip2EncoderTest.testCompression(byte[]) and Bzip2DecoderTest.testDecompression(byte[]).

Result:

Reduced time to test the 'codec' package by 30 percent.
2014-07-22 17:57:24 +02:00