Commit Graph

6238 Commits

Author SHA1 Message Date
Idel Pivnitskiy
a0c466a276 Implemented FastLZ compression codec
Motivation:

FastLZ compression codec provides sending and receiving data encoded by fast FastLZ algorithm using block mode.

Modifications:

- Added part of `jfastlz` library which implements FastLZ algorithm. See FastLz class.
- Implemented FastLzFramedEncoder which extends MessageToByteEncoder and provides compression of outgoing messages.
- Implemented FastLzFramedDecoder which extends ByteToMessageDecoder and provides uncompression of incoming messages.
- Added integration tests for `FastLzFramedEncoder/Decoder`.

Result:

Full FastLZ compression codec which can compress/uncompress data using FastLZ algorithm.
2014-08-12 15:14:59 -07:00
Trustin Lee
60764200d7 Do not throw an exception when failed to get a header
Motivation:

It is often very expensive to instantiate an exception. TextHeader
should not raise an exception when it failed to find a header or when
its header value is not valid.

Modification:

- Change the return type of the getter methods to Integer and Long so
  that null is returned when no header is found or its value is invalid
- Update Javadoc

Result:

- Fixes #2758
- No unnecessary instantiation of exceptions
2014-08-12 11:14:06 -07:00
Trustin Lee
de724063f3 Reduce the initial capacity of the value list from 4 to 2
Motivation:

DefaultTextHeaders.getAll*() methods create an ArrayList whose initial
capacity is 4.  However, it is more likely that the actual number of
values is smaller than that.

Modifications:

Reduce the initial capacity of the value list from 4 to 2

Result:

Slightly reduced memory footprint
2014-08-12 10:36:13 -07:00
jxu
2d36caa9f6 Add TextHeaders.getAndRemove(...) and its variants
Related issue: #2649 and #2745

Motivation:

At the moment there is no way to get and remove a header with one call.
This means you need to search the headers two times. We should add
getAndRemove(...) to allow doing so with one call.

Modifications:

Add getAndRemove(...) and getUnconvertedAndRemove(...) and their
variants

Result:

More efficient API
2014-08-12 10:33:04 -07:00
Norman Maurer
286b89933c Allow to obtain RecvByteBufAllocator.Handle to allow more flexible implementations
Motivation:

At the moment it's only possible for a user to set the RecvByteBufAllocator for a Channel but not access the Handle once it is assigned. This makes it hard to write more flexible implementations.

Modifications:

Add a new method to the Channel.Unsafe to allow access the the used Handle for the Channel. The RecvByteBufAllocator.Handle is created lazily.

Result:

It's possible to write more flexible implementatons that allow to adjust stuff on the fly for a Handle that is used by a Channel
2014-08-12 06:53:57 +02:00
Norman Maurer
02e7e53cbb [#2752] Add PendingWriteQueue for queue up writes
Motivation:

Sometimes ChannelHandler need to queue writes to some point and then process these. We currently have no datastructure for this so the user will use an Queue or something like this. The problem is with this Channel.isWritable() will not work as expected and so the user risk to write to fast. That's exactly what happened in our SslHandler. For this purpose we need to add a special datastructure which will also take care of update the Channel and so be sure that Channel.isWritable() works as expected.

Modifications:

- Add PendingWriteQueue which can be used for this purpose
- Make use of PendingWriteQueue in SslHandler

Result:

It is now possible to queue writes in a ChannelHandler and still have Channel.isWritable() working as expected. This also fixes #2752.
2014-08-12 06:38:22 +02:00
Trustin Lee
486de44680 Fix a resource leak in StompSubframeAggregatorTest 2014-08-11 10:46:43 -07:00
Jeff Pinner
af26826348 SPDY: fix SpdySessionHandler::updateSendWindowSize
In Netty 3, downstream writes of SPDY data frames and upstream reads of
SPDY window udpate frames occur on different threads.

When receiving a window update frame, we synchronize on a java object
(SpdySessionHandler::flowControlLock) while sending any pending writes
that are now able to complete.

When writing a data frame, we check the send window size to see if we
are allowed to write it to the socket, or if we have to enqueue it as a
pending write. To prevent races with the window update frame, this is
also synchronized on the same SpdySessionHandler::flowControlLock.

In Netty 4, upstream and downstream operations on any given channel now
occur on the same thread. Since java locks are re-entrant, this now
allows downstream writes to occur while processing window update frames.

In particular, when we receive a window update frame that unblocks a
pending write, this write completes which triggers an event notification
on the response, which in turn triggers a write of a data frame. Since
this is on the same thread it re-enters the lock and modifies the send
window. When the write completes, we continue processing pending writes
without knowledge that the window size has been decremented.
2014-08-11 11:21:24 +02:00
Trustin Lee
23d3b84273 Fix resource leaks in StompSubframeDecoderTest 2014-08-08 11:26:40 -07:00
Norman Maurer
d5fd57262b [#2744] Fix flakey HashedWheelTimerTest.testExecutionOnTime()
Motivation:

The calculation of the max wait time for HashedWheelTimerTest.testExecutionOnTime() was wrong and so the test sometimes failed.

Modifications:

Fix the max wait time.

Result:

No more test-failures
2014-08-06 07:03:31 +02:00
Trustin Lee
a9ae80fc8f Fix resource leaks in StompSubframeAggregatorTest 2014-08-05 18:12:12 -07:00
Trustin Lee
8e1007d693 Fix a bug where SpdySession.getActiveStreams() returns incorrect set
Related issue: #2743

Motivation:

When there are more than one stream with the same priority, the set
returned by SpdySession.getActiveStream() will not include all of them,
because it uses TreeSet and only compares the priority of streams. If
two different streams have the same priority, one of them will be
discarded by TreeSet.

Modification:

- Rename getActiveStreams() to activeStreams()
- Replace PriorityComparator with StreamComparator

Result:

Two different streams with the same priority are compared correctly.
2014-08-05 17:46:06 -07:00
Trustin Lee
fd813f76fa Add test cases for HttpContentCompressor
- Ported from 386a06dbfa
2014-08-05 15:48:43 -07:00
Idel Pivnitskiy
073ec8d10a Consider writerIndex when LzfDecoder writes into a new heap buffer
Motivation:

Now LzfDecoder do not consider writerIndex when it writes into array of a new heap buffer (when it decodes a compressed chuck of data)
2014-08-05 22:51:02 +02:00
Trustin Lee
8fce6316ad Fix a bug where ChannelFuture.setFailure(null) doesn't fail
Motivation:

We forgot to do a null check on the cause parameter of
ChannelFuture.setFailure(cause)

Modifications:

Add a null check

Result:

Fixed issue: #2728
2014-08-05 11:23:45 -07:00
Norman Maurer
869687bd71 Port ChannelOutboundBuffer and related changes from 4.0
Motivation:

We did various changes related to the ChannelOutboundBuffer in 4.0 branch. This commit port all of them over and so make sure our branches are synced in terms of these changes.

Related to [#2734], [#2709], [#2729], [#2710] and [#2693] .

Modification:
Port all changes that was done on the ChannelOutboundBuffer.

This includes the port of the following commits:
 - 73dfd7c01b
 - 997d8c32d2
 - e282e504f1
 - 5e5d1a58fd
 - 8ee3575e72
 - d6f0d12a86
 - 16e50765d1
 - 3f3e66c31a

Result:
 - Less memory usage by ChannelOutboundBuffer
 - Same code as in 4.0 branch
 - Make it possible to use ChannelOutboundBuffer with Channel implementation that not extends AbstractChannel
2014-08-05 15:00:45 +02:00
Norman Maurer
e33f12f5b8 [#2732] HttpRequestEncoder may produce invalid uri if uri parameters are included.
Motivation:

If the requests contains uri parameters but not path the HttpRequestEncoder does produce an invalid uri while try to add the missing path.

Modifications:

Correctly handle the case of uri with paramaters but no path.

Result:

HttpRequestEncoder produce correct uri in all cases.
2014-08-05 10:13:53 +02:00
Trustin Lee
cc33417c0e Add more utility methods to check the availability of the epoll transport
Related issue: #2733

Motivation:

Unlike OpenSsl, Epoll lacks a couple useful availability checker
methods:

- ensureAvailability()
- unavailabilityCause()

Modifications:

Add missing methods

Result:

More ways to check the availability and to get the cause of
unavailability programatically.
2014-08-04 15:05:05 -07:00
Trustin Lee
8263a62014 Clean-up d9cccccbb3
- Revert irrelevant formatting changes
- Rename resource files
  - Add .pem
  - Remove 'netty' from names
2014-08-04 10:54:05 -07:00
Trustin Lee
a5ccec5ef3 More brief somaxconn logging
- Consistent log message format
- Avoid unnecessary autoboxing when debug level is off
- Remove the duplication of somaxconn path
2014-08-04 10:27:13 -07:00
Peter Schulz
20dffa8ee6 [#2718] Added private key decryption to JDK SSL server context.
Motivation:

Currently it is not possible to load an encrypted private key when
creating a JDK based SSL server context.

Modifications:

- Added static method to JdkSslServerContext which handles key spec generation for (encrypted) private keys and make use of it.
-Added tests for creating a SSL server context based on a (encrypted)
private key.

Result:

It is now possible to create a JDK based SSL server context with an
encrypted (password protected) private key.
2014-08-04 14:19:40 +02:00
Norman Maurer
74dd295e59 Fix buffer leaks in DnsResponseDecoder and DnsResponseDecoderTest
Motivation:

There were two buffer leaks in the codec-dns.

Modifications:

- Fix buffer leak in DnsResponseTest.readResponseTest()
- Correctly release DnsResources on Exception

Result:

No more buffer leaks in the codec-dns module.
2014-08-04 14:06:45 +02:00
Idel Pivnitskiy
c13419750d Improve Bzip2BitReader/Writer
Motivation:

Before this changes Bzip2BitReader and Bzip2BitWriter accessed to ByteBuf byte by byte. So tests for Bzip2 compression codec takes a lot of time if we ran them with paranoid level of resource leak detection. For more information see comments to #2681 and #2689.

Modifications:

- Increased size of bit buffers from 8 to 64 bits.
- Improved reading and writing operations.
- Save link to incoming ByteBuf inside Bzip2BitReader.
- Added methods to check possible readable bits and bytes in Bzip2BitReader.
- Updated Bzip2 classes to use new API of Bzip2BitReader.
- Added new constants to Bzip2Constants.

Result:

Increased size of bit buffers and improved performance of Bzip2 compression codec (for general work by 13% and for tests with paranoid level of resource leak detection by 55%).
2014-08-04 07:52:40 +02:00
Norman Maurer
ff9cc74bf6 Remove duplicated code 2014-07-31 18:11:12 -07:00
Norman Maurer
1f95e5db4c [#2720] Check if /proc/sys/net/core/somaxconn exists before try to parse it
Motivation:

As /proc/sys/net/core/somaxconn does not exists on non-linux platforms you see a noisy stacktrace when debug level is enabled while the static method of NetUtil is executed.

Modifications:

Check if the file exists before try to parse it.

Result:

Less noisy logging on non-linux platforms.
2014-07-31 18:08:54 -07:00
Trustin Lee
3c4321ce43 Use our own URL shortener wherever possible 2014-07-31 17:06:19 -07:00
Trustin Lee
9a654d8a61 Remove duplicate range check in AbstractByteBuf.skipBytes() 2014-07-29 15:58:28 -07:00
Trustin Lee
a35233a4d4 Fix a ConstantPoolTest failure 2014-07-29 15:46:15 -07:00
Trustin Lee
77609cf6ed Fix a bug where AbstractConstant.compareTo() returns 0 for different constants
Related issue: #2354

Motivation:

AbstractConstant.compareTo() can return 0 even if the specified constant
object is not the same instance with 'this'.

Modifications:

- Compare the identityHashCode of constant first. If that fails,
  allocate a small direct buffer and use its memory address as a unique
  value.  If the platform does not provide a way to get the memory
  address of a direct buffer, use a thread-local random value.
- Signal cannot extend AbstractConstant. Use delegation.

Result:

It is practically impossible for AbstractConstant.compareTo() to return
0 for different constant objects.
2014-07-29 15:01:47 -07:00
Norman Maurer
3207fac88e Use correct exception message when throw exception from native code
Motivation:

We sometimes not use the correct exception message when throw it from the native code.

Modifications:

Fixed the message.

Result:

Correct message in exception
2014-07-28 13:33:54 -07:00
Norman Maurer
750eed1804 Fix broken test after change the maximal value of the pid. Part of [#2706] 2014-07-28 10:39:44 -07:00
Norman Maurer
168e2dde05 [#2706] Allow pid up to 4194304
Motivation:

The PID_MAX_LIMIT on 64bit linux systems is 4194304 and on osx it is 99998. At the moment we use 65535 as an upper-limit which is too small.

Modifications:

Use 4194304 as max possible value

Result:

No more false-positives when try to detect current pid.
2014-07-28 10:12:49 -07:00
Norman Maurer
e09d2f32fb [#2708] DnsResource.duplicate() should return DnsResource and not ByteBufHolder
Motivation:

DnsResource.duplicate() should return DnsResource and not ByteBufHolder

Modifications:

Change return type from ByteBufHolder to DnsResource

Result:

No need to cast to the correct type when using duplicate()
2014-07-28 04:26:09 -07:00
Norman Maurer
f88cd62354 [#2692] Allows notify ChannelFutureProgressListener on complete writes
Motivation:

We have some inconsistency when handling writes. Sometimes we call ChannelOutboundBuffer.progress(...) also for complete writes and sometimes not. We should call it always.

Modifications:

Correctly call ChannelOuboundBuffer.progress(...) for complete and incomplete writes.

Result:

Consistent behavior
2014-07-28 04:19:23 -07:00
Norman Maurer
d5b7c131dd Correctly write single ByteBuf with memoryAddress
Motivation:

While optimize gathering writes I introduced a bug when writing single ByteBuf that have a memoryAddress. This regression was introduced by 88bd6e7a93.

Modifications:

Correctly use the writerIndex as argument when call Native.writeAddress(...)

Result:

No more corruption while write single buffers.
2014-07-25 17:27:22 +02:00
Norman Maurer
88bd6e7a93 Optimize native transport for gathering writes
Motivation:

While benchmarking the native transport with gathering writes I noticed that it is quite slow. This is due the fact that we need to do a lot of array copies to get the buffers into the iov array.

Modification:

Introduce a new class calles IovArray which allows to fill buffers directly in a iov array that can be passed over to JNI without any array copies. This gives a nice optimization in terms of speed when doing gathering writes.

Result:

Big performance improvement when doing gathering writes. See the included benchmark...

Before:
[nmaurer@xxx]~% wrk/wrk -H 'Host: localhost' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -d 120 -c 256 -t 16 --pipeline 256  http://xxx:8080/plaintext
Running 2m test @ http://xxx:8080/plaintext
  16 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.44ms   16.37ms 259.57ms   91.77%
    Req/Sec   181.99k    31.69k  304.60k    78.12%
  346544071 requests in 2.00m, 46.48GB read
Requests/sec: 2887885.09
Transfer/sec:    396.59MB

With this change:
[nmaurer@xxx]~% wrk/wrk -H 'Host: localhost' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -d 120 -c 256 -t 16 --pipeline 256  http://xxx:8080/plaintext
Running 2m test @ http://xxx:8080/plaintext
  16 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.93ms   16.33ms 305.73ms   92.34%
    Req/Sec   194.56k    33.75k  309.33k    77.04%
  369617503 requests in 2.00m, 49.57GB read
Requests/sec: 3080169.65
Transfer/sec:    423.00MB
2014-07-25 09:55:02 +02:00
Norman Maurer
5b2bdd844d [#2662] Fix race in cancellation of TimerTasks which could let to NPE
Motivation:

Due some race-condition while handling canellation of TimerTasks it was possibleto corrupt the linked-list structure that is represent by HashedWheelBucket and so produce a NPE.

Modification:

Fix the problem by adding another MpscLinkedQueue which holds the cancellation tasks and process them on each tick. This allows to use no synchronization / locking at all while introduce a latency of max 1 tick before the TimerTask can be GC'ed.

Result:

No more NPE
2014-07-25 06:34:35 +02:00
Norman Maurer
e1cc1fbabc [#2705] Call fireChannelReadComplete() if channelActive(...) decodes messages in ReplayingDecoder / ByteToMessageDecoder
Motivation:

In ReplayingDecoder / ByteToMessageDecoder channelInactive(...) method we try to decode a last time and fire all decoded messages throw the pipeline before call ctx.fireChannelInactive(...). To keep the correct order of events we also need to call ctx.fireChannelReadComplete() if we read anything.

Modifications:

- Channel channelInactive(...) to call ctx.fireChannelReadComplete() if something was decoded
- Move out.recycle() to finally block

Result:

Correct order of events.
2014-07-24 14:38:46 +02:00
Willem Jiang
b029800b68 Updated the ChannelGroup JavaDoc by removing b.releaseExternalResources(); 2014-07-24 10:55:09 +02:00
Trustin Lee
4bde044957 Overall cleanup of codec-dns
Related issue: #2688

- DnsClass and DnsType
  - Make DnsClass and DnsType implement Comparable
  - Optimize the message generation of IllegalArgumentException,
    by pre-populating the list of the expected parameters
  - Move the static methods up
  - Relax the validation rule of DnsClass so that a user can define a
    CLASS which is not listed in the RFC 1035
  - valueOf(int) does not throw IllegalArgumentException anymore as long
    as the specified value is an unsigned short.
  - Rename create() and forName() to valueOf() so that they look like a
    real enum
  - Rename type() and clazz() to intValue() so that they conform to our
    naming convention
- Add missing null checks in DnsEntry
2014-07-23 14:40:52 -07:00
Tim Boudreau
9734170b7d Use value types for class and type in DNS entries to make them immune to parameter order bugs
Motivation:

DNS class and type were represented as integers rather than an enum or a
similar dedicated value type.  This can be a potential source of a
parameter order bug which might be difficult to track down.

Modifications:

Add DnsClass and DnsType to replace integer parameters

Result:

Type safety and less error-proneness
2014-07-23 14:40:52 -07:00
Idel Pivnitskiy
4816533638 Refactor Bzip2 tests
Motivation:

Complicated code of Bzip2 tests with some unnecessary actions.

Modifications:

- Reduce size of BYTES_LARGE array of random test data for Bzip2  tests.
- Removed unnecessary creations of EmbeddedChannel instances in Bzip2 tests.
- Simplified tests in Bzip2DecoderTest which expect exception.
- Removed unnecessary testStreamInitialization() from Bzip2EncoderTest.

Result:

Reduced time to test the 'codec' package by 7 percent, simplified code of Bzip2 tests.
2014-07-23 19:46:00 +02:00
Idel Pivnitskiy
99cf6f0732 Refactor integration tests of compression codecs
Motivation:

Duplicated code of integration tests for different compression codecs.

Modifications:

- Added abstract class IntegrationTest which contains common tests for any compression codec.
- Removed common tests from Bzip2IntegrationTest and LzfIntegrationTest.
- Implemented abstract methods of IntegrationTest in Bzip2IntegrationTest, LzfIntegrationTest and SnappyIntegrationTest.

Result:

Removed duplicated code of integration tests for compression codecs and simplified an addition of integration tests for new compression codecs.
2014-07-23 19:44:10 +02:00
Trustin Lee
830091c260 Reduce the default initial capacity of ChannelOutboundBuffer
Motivation:

ChannelOutboundBuffer is basically a circular array queue of its entry
objects.  Once an entry is created in the array, it is never nulled out
to reduce the allocation cost.

However, because it is a circular queue, the array almost always ends up
with as many entry instances as the size of the array, regardless of the
number of pending writes.

At worst case, a channel might have only 1 pending writes at maximum
while creating 32 entry objects, where 32 is the initial capacity of the
array.

Modifications:

- Reduce the initial capacity of the circular array queue to 4.
- Make the initial capacity of the circular array queue configurable

Result:

We spend 4 times less memory for entry objects under certain
circumstances.
2014-07-22 13:32:25 -07:00
Idel Pivnitskiy
ca87cc887e Simplify Bzip2 tests
Motivation:

Sometimes we have a 'build time out' error because tests for bzip2 codec take a long time.

Modifications:

Removed cycles from Bzip2EncoderTest.testCompression(byte[]) and Bzip2DecoderTest.testDecompression(byte[]).

Result:

Reduced time to test the 'codec' package by 30 percent.
2014-07-22 18:00:26 +02:00
Trustin Lee
923a0e71ac Raise a meaningful exception instead of NPE
Motivation:

When decoding the NAME field in a DNS Resource Record, DnsResponseDecoder
can raise a NullPointerException if the NAME field contains a loop.

Modification:

Instead of raising an NPE, raise CorruptedFrameException so that the
exception itself has meaning.

Result:

Less confusing when a malformed DNS RR is received
2014-07-21 16:51:38 -07:00
Trustin Lee
ae9963a40c Fix NPE while decoding authority section of a DNS response
Motivation:

NullPointerException is raised when a DNS response conrains a resource
record whose NAME is empty, which is the case for the authority section.

Modification:

Allow an empty name for DnsEntry. Only disallow an empty name for
DnsQuestion.

Result:

Fixes #2686
2014-07-21 16:40:26 -07:00
Trustin Lee
530750d606 Add a link to the 'native transports' page 2014-07-21 12:54:24 -07:00
Osvaldo Doederlein
07024a4e4b Fixes and improvements to IntObjectHashMap. Related to [#2659]
- Rewrite with linear probing, no state array, compaction at cleanup
- Optimize keys() and values() to not use reflection
- Optimize hashCode() and equals() for efficient iteration
- Fixed equals() to not return true for equals(null)
- Optimize iterator to not allocate new Entry at each next()
- Added toString()
- Added some new unit tests
2014-07-21 16:43:15 +02:00
Norman Maurer
c85319213a [#2675] Replace synchronization performed on util.concurrent instance in TrafficCounter
Motivation:

Message from FindBugs:
This method performs synchronization an object that is an instance of a class from the java.util.concurrent package (or its subclasses). Instances of these classes have their own concurrency control mechanisms that are orthogonal to the synchronization provided by the Java keyword synchronized. For example, synchronizing on an AtomicBoolean will not prevent other threads from modifying the AtomicBoolean.
Such code may be correct, but should be carefully reviewed and documented, and may confuse people who have to maintain the code at a later date.

Modification:

Use synchronized(this)

Result:

Less confusing code
2014-07-21 08:23:24 +02:00