Commit Graph

775 Commits

Author SHA1 Message Date
Norman Maurer
afafadd3d7 [#5505] Enforce Recycler limit when recycling from different threads
Motivation:

Currently, the recycler max capacity it's only enforced on the
thread-local stack which is used when the recycling happens on the
same thread that requested the object.

When the recycling happens in a different thread, then the objects
will be queued into a linked list (where each node holds N objects,
default=16). These objects are then transfered into the stack when
new objects are requested and the stack is empty.

The problem is that the queue doesn't have a max capacity and that
can lead to bad scenarios. Eg:

- Allocate 1M object from recycler
- Recycle all of them from different thread
- Recycler WeakOrderQueue will contain 1M objects
- Reference graph will be very long to traverse and GC timeseems to be negatively impacted
- Size of the queue will never shrink after this

Modifications:

Add some shared counter which is used to manage capacity limits when recycle from different thread then the allocation thread. We modify the counter whenever we allocate a new Link to reduce the overhead of increment / decrement it.

Result:

More predictable number of objects mantained in the recycler pool.
2016-07-14 07:56:03 +02:00
Nitesh Kant
77770374fb Ability to run a task at the end of an eventloop iteration.
Motivation:

This change is part of the change done in PR #5395 to provide an `AUTO_FLUSH` capability.
Splitting this change will enable to try other ways of implementing `AUTO_FLUSH`.

Modifications:



Two methods:

```java
void executeAfterEventLoopIteration(Runnable task);


boolean removeAfterEventLoopIterationTask(Runnable task);
```
are added to `SingleThreadEventLoop` class for adding/removing a task to be executed at the end of current/next iteration of this `eventloop`.

In order to support the above, a few methods are added to `SingleThreadEventExecutor`

```java
protected void afterRunningAllTasks() { }
```

This is invoked after all tasks are run for this executor OR if the passed timeout value for `runAllTasks(long timeoutNanos)` is expired.

Added a queue of `tailTasks` to `SingleThreadEventLoop` to hold all tasks to be executed at the end of every iteration.


Result:



`SingleThreadEventLoop` now has the ability to execute tasks at the end of an eventloop iteration.
2016-07-12 10:22:15 +02:00
Jason Tedor
cfd6db7915 Avoid missed signals on a default promise
Motivation:

Today when awaiting uninterruptibly on a default promise, a race
condition can lead to a missed signal. Quite simply, the check for
whether the condition holds is not made inside a lock before
waiting. This means that the waiting thread can enter the wait after the
promise has completed and will thus not be notified, thus missing the
signal. This leads to the waiting thread to enter a timed wait that will
only trip with the timeout elapses leading to unnecessarily long waits
(imagine a connection timeout, and the waiting thread missed the signal
that the connection is ready).

Modification:

This commit fixes this missed signal by checking the condition inside a
lock. We also add a test that reliably fails without the non-racy
condition check.

Result:

Timed uninterruptible waits on default promise will not race against the
condition and possibly wait longer than necessary.
2016-07-12 09:57:14 +02:00
Norman Maurer
ce95c50444 [#5507] SingleThreadEventExecutor should reject call invoke*() from within the EventLoop.
Motivation:

ExecutorService.invoke*(...) methods may block by API definition. This can lead to deadlocks if called from inside the EventLoop in SingleThreadEventExecutor as it only has one Thread that does all the work.

Modifications:

Throw a RejectedExectionException if someone tries to call SingleThreadEventExecutor.invoke*(...) while in the EventLoop.

Result:

No more deadlock possible.
2016-07-09 08:08:50 +02:00
Julien Viet
79c8ec4d33 DnsNameResolver search domains support
Motivation:

The current DnsNameResolver does not support search domains resolution. Search domains resolution is supported out of the box by the java.net resolver, making the DnsNameResolver not able to be a drop in replacement for io.netty.resolver.DefaultNameResolver.

Modifications:

The DnsNameResolverContext resolution has been modified to resolve a list of search path first when it is configured so. The resolve method now uses the following algorithm:

if (hostname is absolute (start with dot) || no search domains) {
 searchAsIs
} else {
  if (numDots(name) >= ndots) {
    searchAsIs
  }
  if (searchAsIs wasn't performed or failed) {
    searchWithSearchDomainsSequenciallyUntilOneSucceeds
  }
}

The DnsNameResolverBuilder provides configuration for the search domains and the ndots value. The default search domains value is configured with the OS search domains using the same native configuration the java.net resolver uses.

Result:

The DnsNameResolver performs search domains resolution when they are present.
2016-07-08 22:04:01 +02:00
Scott Mitchell
4baff691b4 DefaultPromise make listeners not volatile
Motivation:
DefaultPromise has a listeners member variable which is volatile to allow for an optimization which makes notification of listeners less expensive when there are no listeners to notify. However this change makes all other operations involving the listeners member variable more costly. This optimization which requires listeners to be volatile can be removed to avoid volatile writes/reads for every access on the listeners member variable.

Modifications:
- DefaultPromise listeners is made non-volatile and the null check optimization is removed

Result:
DefaultPromise.listeners is no longer volatile.
2016-07-07 12:53:03 -07:00
Norman Maurer
29fdb160f3 [#5486] Not operate on serial execution assumption when using EventExecutor in the DefaultChannelPipeline.
Motivation:

In commit f984870ccc I made a change which operated under invalide assumption that tasks executed by an EventExecutor will always be processed in a serial fashion. This is true for SingleThreadEventExecutor sub-classes but not part of the EventExecutor interface contract.

Because of this change implementations of EventExecutor which not strictly execute tasks in a serial fashion may miss events before handlerAdded(...) is called. This is strictly speaking not correct as there is not guarantee in this case that handlerAdded(...) will be called as first task (as there is no ordering guarentee).

Cassandra itself ships such an EventExecutor implementation which has no strict ordering to spread load across multiple threads.

Modifications:

- Add new OrderedEventExecutor interface and let SingleThreadEventExecutor / EventLoop implement / extend it.
- Only expose "restriction" of skipping events until handlerAdded(...) is called for OrderedEventExecutor implementations
- Add ThreadPoolEventExecutor implementation which executes tasks in an unordered fashion. This is used in added unit test but can also be used for protocols which not expose an strict ordering.
- Add unit test.

Result:

Resurrect the possibility to implement an EventExecutor which does not enforce serial execution of events and be able to use it with the DefaultChannelPipeline.
2016-07-07 15:01:56 +02:00
Norman Maurer
6492cb98b2 Revert "DefaultPromise make listeners not volatile"
This reverts commit 4d8132ff24 as I missed something I want to discuss first.
2016-07-07 08:37:41 +02:00
Scott Mitchell
4d8132ff24 DefaultPromise make listeners not volatile
Motivation:
DefaultPromise has a listeners member variable which is volatile to allow for an optimization which makes notification of listeners less expensive when there are no listeners to notify. However this change makes all other operations involving the listeners member variable more costly. This optimization which requires listeners to be volatile can be removed to avoid volatile writes/reads for every access on the listeners member variable.

Modifications:
- DefaultPromise listeners is made non-volatile and the null check optimization is removed

Result:
DefaultPromise.listeners is no longer volatile.
2016-07-07 08:01:25 +02:00
Jason Tedor
d97129b4c0 Log listener Throwables in default promise
Motivation:

The logging statements in i.n.u.c.DefaultPromise do not emit the
caught Throwable when a Throwable is thrown while a listener is being
notified of completed or progressed operations.

Modifications:

This issue arises because the logging message has a single placeholder
but is passing two additional arguments, the second one being the
caught Throwable that is thus quietly not logged. We address this by
modifying the logging statements to ensure the caught Throwable is
logged. In this case, the preferred approach is to use the logger
override that accepts a message and a Throwable parameter since logger
implementations might have special handling for this case.

Result:

Log messages from i.n.u.c.DefaultPromise when a Throwable is thrown
while notifying a listener of completed or progressed operations will
contain the caught Throwable.
2016-07-06 08:52:19 +02:00
buchgr
b0a5d4c266 Fix improper synchronization in DefaultPromise. Fixes #5489
Motivation:

A race detector found that DefaultPromise.listeners is improperly synchronized [1].
Worst case a listener will not be executed when the promise is completed.

Modifications:

Make DefaultPromise.listeners a volatile.

Result:

Hopefully, DefaultPromise is more correct under concurrent execution.

[1] https://github.com/grpc/grpc-java/issues/2015
2016-07-04 19:00:45 +02:00
墨睿
be3e6972a1 Add a test case to check sub strings' AsciiString hash code
Motivation:

AsciiString.hashCode(o) , if "o" is a subString, the hash code is not always same, when netty’s version is 4.1.1.Final and jdk’s version is 1.6.

Modifications:

Use a test to assert hash codes are equal between a new string and any sub string (a part of  a char array),If their values are equal.

Result:

Create a test method to AsciiStringCharacterTest.
2016-07-04 07:19:52 +02:00
Tim Brooks
181c159c24 Remove unneeded calls to hasScheduledTasks() when fetching from scheduled task queue for event executors.
Motivation:

Currently in the single threaded and global event executors when the scheduled task queue is drained, there is a call to hasScheduledTasks(). If there are scheduled tasks then the the code polls the queue for tasks. The poll method duplicates the exact logic of hasScheduledTasks(). This involves two calls to nanoTime when one seems sufficient.

Modifications:

Directly poll the queue for tasks and break if the task returned is null.

Result:

Should be no noticeable impact on functionality. Two calls to nanoTime have been coarsened into a single call.
2016-07-01 17:16:58 +02:00
Norman Maurer
4676a2271c [maven-release-plugin] prepare for next development iteration 2016-07-01 10:33:32 +02:00
Norman Maurer
ad270c02b9 [maven-release-plugin] prepare release netty-4.1.2.Final 2016-07-01 09:07:40 +02:00
Carsten Varming
34d8b514d7 Use reflection to call cleaner on direct byte buffers in JDK9.
Motivation:

Project Jigsaw in JDK9 has moved the direct byte buffer cleaner from
sun.misc.Cleaner to java.lang.ref.Cleaner$Cleanable. This cause the
current platform tests to throw a ClassNotFoundException, disabling the
use of direct byte buffer cleaners.

Modifications:

I use reflection to find the clean method in either sun.misc.Cleaner or
java.lang.ref.Cleaner$Cleanable.

Result:

Netty uses direct byte buffers on JDK9 as it already do on earlier JDKs.
2016-06-30 18:48:36 +02:00
Carsten Varming
8780062a43 Removed HeapByteBuffer address field check.
Motivation:

In JDK9 heap byte buffers have an address field, so we have to remove
the current check as it is invalid in JDK9.

Modifications:

Removed the address field check for heap byte buffers.

Result:
Netty continues to find sun.misc.Unsafe in JDK9 as in previous JDKs.
2016-06-30 18:47:10 +02:00
Carsten Varming
9d933091bf Add version check for JDK9 and beyond.
Motivation:

Netty's platform dependent parts should know about JDK9.

Modifications:

JDK9 introduce Runtime$Version Runtime.version() which has an int major()
method that always return the major Java version. I call that method to
get the Java major version.

Result:

Netty will recognize all future JDK versions.
2016-06-30 18:45:35 +02:00
Scott Mitchell
6af56ffe76 HPACK Encoder headerFields improvements
Motivation:
HPACK Encoder has a data structure which is similar to a previous version of DefaultHeaders. Some of the same improvements can be made.

Motivation:
- Enforce the restriction that the Encoder's headerFields length must be a power of two so we can use masking instead of modulo
- Use AsciiString.hashCode which already has optimizations instead of having yet another hash code algorithm in Encoder

Result:
Fixes https://github.com/netty/netty/issues/5357
2016-06-30 09:00:12 -07:00
Scott Mitchell
a7f7d9c8e0 Remove unsafe char[] access in PlatformDependent
Motivation:
PlatformDependent attempts to use reflection to get the underlying char[] (or byte[]) from String objects. This is fragile as if the String implementation does not utilize the full array, and instead uses a subset of the array, this optimization is invalid. OpenJDK6 and some earlier versions of OpenJDK7 String have the capability to use a subsection of the underlying char[].

Modifications:
- PlatformDependent should not attempt to use the underlying array from String (or other data types) via reflection

Result:
PlatformDependent hash code generation for CharSequence does not depend upon specific JDK implementation details.
2016-06-30 08:58:28 -07:00
Norman Maurer
57672d9854 Use ResourceLeakDetectorFactory in HashedWheelTimer
Motivation:

We recently added the ResourceLeakDetectorFactory but missed to updated HashedWheelTimer to use it.

Modifications:

- Add new abstract method to ResourceLeakDetectorFactory that allows to provide also samplingInterval and maxActive args.
- Deprecate most constructors in ResourceLeakDetector and add doc explaining that people should use ResourceLeakDetectorFactory

Result:

Custom ResourceLeakDetectorFactory will also be used in HashedWheelTimer if configured.
2016-06-29 21:07:17 +02:00
Norman Maurer
bd0a74fca3 Allow to disable ResourceLeak creation when worker thread is deamon in HashedWheelTimer
Motivation:

Sometimes a shared HashedWheelTimer can not easily be stopped in a good place. If the worker thread is daemon this is not a big deal and we should allow to not log a leak.

Modifications:

Add another constructor which allows to disable resource leak detection if worker thread is used.

Result:

Not log resource leak when HashedWheelTimer is not stopped and the  worker thread is a deamon thread.
2016-06-29 20:06:58 +02:00
Scott Mitchell
857ad73676 DefaultPromise make MAX_STACK_DEPTH configurable
Motivation:
Some Netty use cases may want to configure the max allowed stack depth for promise listener notification.

Modifications:
- Add a system property so that this value can be configured.

Result:
DefaultPromise's max stack depth is configurable.
2016-06-29 08:31:27 -07:00
Scott Mitchell
70651cc58d HpackUtil.equals performance improvement
Motivation:
PR #5355 modified interfaces to reduce GC related to the HPACK code. However this came with an anticipated performance regression related to HpackUtil.equals due to AsciiString's increase cost of charAt(..). We should mitigate this performance regression.

Modifications:
- Introduce an equals method in PlatformDependent which doesn't leak timing information and use this in HpcakUtil.equals

Result:
Fixes https://github.com/netty/netty/issues/5436
2016-06-27 14:37:39 -07:00
Guido Medina
f0a5ee068f Update dependencies and plugins to latest possible versions.
Motivation:
It is good to have used dependencies and plugins up-to-date to fix any undiscovered bug fixed by the authors.

Modification:
Scanned dependencies and plugins and carefully updated one by one.

Result:
Dependencies and plugins are up-to-date.
2016-06-27 13:35:35 +02:00
Norman Maurer
731f52fdf7 Allow to inject RejectedExecutionHandler for different EventLoops and EventExecutors
Motiviation:

Sometimes it is useful to allow to specify a custom strategy to handle rejected tasks. For example if someone tries to add tasks from outside the eventloop it may make sense to try to backoff and retries and so give the executor time to recover.

Modification:

Add RejectedEventExecutor interface and implementations and allow to inject it.

Result:

More flexible handling of executor overload.
2016-06-24 17:08:30 +02:00
Norman Maurer
a0d4fb16fc Allow to set max capacity for task queue for EventExecutors and EventLoops
Motivation:

To restrict the memory usage of a system it is sometimes needed to adjust the number of max pending tasks in the tasks queue.

Modifications:

- Add new constructors to modify the number of allowed pending tasks.
- Add system properties to configure the default values.

Result:

More flexible configuration.
2016-06-24 14:00:50 +02:00
Norman Maurer
278c36af0c Merge ThrowableUtils into ThrowableUtil.
Motivation:

We should merge ThrowableUtils into ThrowableUtil as this name is more consistent with the naming of utility classes in netty.

Modifications:

Merge classes.

Result:

More consistent naming
2016-06-23 10:05:41 +02:00
Norman Maurer
e845670043 Set some StackTraceElement on pre-instantiated static exceptions
Motivation:

We use pre-instantiated exceptions in various places for performance reasons. These exceptions don't include a stacktrace which makes it hard to know where the exception was thrown. This is especially true as we use the same exception type (for example ChannelClosedException) in different places. Setting some StackTraceElements will provide more context as to where these exceptions original and make debugging easier.

Modifications:

Set a generated StackTraceElement on these pre-instantiated exceptions which at least contains the origin class and method name. The filename and linenumber are specified as unkown (as stated in the javadocs of StackTraceElement).

Result:

Easier to find the origin of a pre-instantiated exception.
2016-06-20 11:33:05 +02:00
agonigberg
3288cacf8d Pluggable resource leak detector
Allow users of Netty to plug in their own leak detector for the purpose
of instrumentation.

Motivation:

We are rolling out a large Netty deployment and want to be able to
track the amount of leaks we're seeing in production via custom
instrumentation. In order to achieve this today, I had to plug in a
custom `ByteBufAllocator` into the bootstrap and have it initialize a
custom `ResourceLeakDetector`. Due to these classes mostly being marked
`final` or having private or static methods, a lot of the code had to
be copy-pasted and it's quite ugly.

Modifications:

* I've added a static loader method for the `ResourceLeakDetector` in
`AbstractByteBuf` that tries to instantiate the class passed in via the
`-Dio.netty.customResourceLeakDetector`, otherwise falling back to the
default one.
* I've modified `ResourceLeakDetector` to be non-final and to have the
reporting broken out in to methods that can be overridden.

Result:

You can instrument leaks in your application by just adding something
like the following:

```java
public class InstrumentedResourceLeakDetector<T> extends
ResourceLeakDetector<T> {

    @Monitor("InstanceLeakCounter")
    private final AtomicInteger instancesLeakCounter;

    @Monitor("LeakCounter")
    private final AtomicInteger leakCounter;

    public InstrumentedResourceLeakDetector(Class<T> resource) {
        super(resource);
        this.instancesLeakCounter = new AtomicInteger();
        this.leakCounter = new AtomicInteger();
    }

    @Override
    protected void reportTracedLeak(String records) {
        super.reportTracedLeak(records);
        leakCounter.incrementAndGet();
    }

    @Override
    protected void reportUntracedLeak() {
        super.reportUntracedLeak();
        leakCounter.incrementAndGet();
    }

    @Override
    protected void reportInstancesLeak() {
        super.reportInstancesLeak();
        instancesLeakCounter.incrementAndGet();
    }
}
```
2016-06-20 11:14:44 +02:00
Norman Maurer
8ccc795314 Expose DefaultThreadFactory.threadGroup to sub-classes
Motivation:

DefaultThreadFactory allows to override the newThread(...) method and so should have access to all fields that are set via the constructor.

Modifications:

Change threadGroup from private to protected visibility.

Result:

Easier to extend DefaultThreadFactory.
2016-06-17 06:23:53 +02:00
Dmitry Spikhalskiy
428c61673b Logs in invokeExceptionCaught have been made consistent and full
Motivation:

In case of exception in invokeExceptionCaught() only original exception passed to invokeExceptionCaught() will be logged on any log level.
+ AbstractChannelHandlerContext and CombinedChannelDuplexHandler log different exceptions.

Modifications:

Fix inconsistent logging code and add ability to see both stacktraces on DEBUG level.

Result:

Both handlers log now both original exception and thrown from invokeExceptionCaught. To see full stacktrace of exception thrown from invokeExceptionCaught DEBUG log level must be enabled.
2016-06-11 20:11:11 +02:00
Guido Medina
b921f80057 Make JCtools available (provided scope) for tests and optional OSGI, issue #5383 2016-06-11 07:15:48 +02:00
Guido Medina
c3abb9146e Use shaded dependency on JCTools instead of copy and paste
Motivation:
JCTools supports both non-unsafe, unsafe versions of queues and JDK6 which allows us to shade the library in netty-common allowing it to stay "zero dependency".

Modifications:
- Remove copy paste JCTools code and shade the library (dependencies that are shaded should be removed from the <dependencies> section of the generated POM).
- Remove usage of OneTimeTask and remove it all together.

Result:
Less code to maintain and easier to update JCTools and less GC pressure as the queue implementation nt creates so much garbage
2016-06-10 13:19:45 +02:00
Scott Mitchell
56a2f64665 Clarify Future.removeListener[s] javaDocs
Motivation:
The javaDocs for Future.removeListener do not clarify that only the first occurrence of the listener is guaranteed to be removed.

Modifications:
- Clarify the javaDocs for Future.removeListener[s] so it is known that the only the first occurrence of the listener will be removed.

Result:
Fixes https://github.com/netty/netty/issues/5351
2016-06-08 15:48:26 -07:00
Norman Maurer
4dec7f11b7 [maven-release-plugin] prepare for next development iteration 2016-06-07 18:52:34 +02:00
Norman Maurer
cf670fab75 [maven-release-plugin] prepare release netty-4.1.1.Final 2016-06-07 18:52:22 +02:00
Norman Maurer
b461c9d54c Allow to specify a custom EventExecutorChooserFactory. Related to [#1230]
Motivation:

Sometimes it may be benefitially for an user to specify a custom algorithm when choose the next EventExecutor/EventLoop.

Modifications:

Allow to specify a custom EventExecutorChooseFactory that allows to customize algorithm.

Result:

More flexible api.
2016-06-06 11:04:56 +02:00
Norman Maurer
584fbac6ed [#3419] Only use SecureRandom to generate initialSeed if requested
Motivation:

We tried to always use SecureRandom to generate the initialSeed for our ThreadLocalRandom, this can sometimes give warnings under normal usage. We should better not use SecureRandom as default (just as the implementation in jsr166y does) and only try if the user specified -Djava.util.secureRandomSeed=true .

Modifications:

Only try to use SecureRandom when -Djava.util.secureRandomSeed=true is used.

Result:

Less likely to see entropy warnings.
2016-06-06 09:09:29 +02:00
Norman Maurer
3a7dcde320 [#5224] Allow to use Unsafe.reallocateMemory(...) in UnpooledUnsafeNoCleanerDirectByteBuf.
Motivation:

If the user uses unsafe direct buffers with no cleaner we can use Unsafe.reallocateMemory(...) as optimization when we need to expand the buffer.

Modifications:

Use Unsafe.relocateMemory(...) in UnpooledUnsafeNoCleanerDirectByteBuf.

Result:

Less expensive expanding of buffers.
2016-06-04 19:21:41 +02:00
Norman Maurer
3d29bcfc8d Allow to create Unsafe ByteBuf implementations that not use a Cleaner to clean the native memory.
Motivation:

Using the Cleaner to release the native memory has a few drawbacks:

- Cleaner.clean() uses static synchronized internally which means it can be a performance bottleneck
- It put more load on the GC

Modifications:

Add new buffer implementations that can be enabled with a system flag as optimizations. In this case no Cleaner is used at all and the user must ensure everything is always released.

Result:

Less performance impact by direct buffers when need to be allocated and released.
2016-06-03 21:20:10 +02:00
Scott Mitchell
f6ad9df8ac DefaultPromise StackOverflowError protection updates
Modifications:
DefaultPromise provides a ThreadLocal queue to protect against StackOverflowError because of executors which may immediately execute runnables instead of queue them (i.e. ImmediateEventExecutor). However this may be better addressed by fixing these executors to protect against StackOverflowError instead of just fixing for a single use case. Also the most commonly used executors already provide the desired behavior and don't need the additional overhead of a ThreadLocal queue in DefaultPromise.

Modifications:
- Remove ThreadLocal queue from DefaultPromise
- Change ImmediateEventExecutor so it maintains a queue of runnables if reentrant condition occurs

Result:
DefaultPromise StackOverflowError code is simpler, and ImmediateEventExecutor protects against StackOverflowError.
2016-06-02 09:22:47 -07:00
Alex Petrov
bbed330468 Fix the possible reference leak in Recycler
Motivation:

Under very unlikely (however possible) circumstances, Recycler may leak
references. This happens _only_ when the object was already recycled
at least once (which means it's got written to the stack) and then
taken out again, and never returned.

The "never returned" part may be the fault of the user (forgotten
`finally` clause) or the situation when Recycler drops the possibly
youngest item itself.

Modifications:

Nullify the item taken from the stack.

Result:

Reference is cleaned up. If the object is lost, it will be a subject for
GC. The rest of Stack / Recycler functionality remains unaffected.
2016-06-01 06:47:43 +02:00
Norman Maurer
1dfcfc17fa Allow to change link capacity via system property
Motivation:

Sometimes people may want to trade GC with memory overhead. For this it can be useful to allow to change the capacity of the array that is hold in the Link that is used by the Recycler internally.

Modifications:

Introduce a new system property , io.netty.recycler.linkCapacity which allows to change the capcity.

Result:

More flexible configuration of netty.
2016-05-31 14:05:23 +02:00
Milos Fabian
4c186c4c41 Make netty-transport-native-epoll-*-linux-x86_64.jar working in OSGi
Motivation:

Currenlty, netty-transport-native-epoll-*-linux-x86_64.jar is not packed as OSGi bundle
and thus not working in OSGi environment.

Modifications:

In netty-transport-native-epoll's pom.xml added configuration
to attach manifest to the jar with a native library.
In netty-common's pom.xml added configuration instruction (DynamicImport-Package)
to maven bnd plugin to make sure the native code is loaded from
netty-transport-native-epoll bundle.

Result:

The netty-transport-native-epoll-*-linux-x86_64.jar is a bundle (MANIFEST.MF attached)
and the inluced native library can be successfuly loaded in OSGi environment.
Fixing #5119
2016-05-30 22:04:01 +02:00
Norman Maurer
db6b72da19 Add optimized version of setZero(...) / writeZero(...) for Unsafe*ByteBuf implementations
Motivation:

Unsafe offers a method to set memory to a specific value. This can be used to implement an optimized version of setZero(...) and writeZero(...)

Modifications:

Add implementation for all Unsafe*ByteBuf implementations.

Result:

Faster setZero(...) and writeZero(...)
2016-05-30 15:10:15 +02:00
Norman Maurer
86f53083e7 [#5297] Ensure calling NioEventLoop.pendingTasks() and EpollEventLoop.pendingTasks() will not produce livelock
Motivation:

SingleThreadEventExecutor.pendingTasks() will call taskQueue.size() to get the number of pending tasks in the queue. This is not safe when using MpscLinkedQueue as size() is only allowed to be called by a single consumer.

Modifications:

Ensure size() is only called from the EventLoop.

Result:

No more livelock possible when call pendingTasks, no matter from which thread it is done.
2016-05-28 20:04:07 +02:00
Norman Maurer
6ca49d1336 [maven-release-plugin] prepare for next development iteration 2016-05-25 19:16:44 +02:00
Norman Maurer
446b38db52 [maven-release-plugin] prepare release netty-4.1.0.Final 2016-05-25 19:14:15 +02:00
Scott Mitchell
b3c56d5f69 DefaultPromise StackOverflowError protection
Motivation:
f2ed3e6ce8 removed the previous mechanism for StackOverflowError because it didn't work in all cases (i.e. ImmediateExecutor). However if a chain of listeners which complete other promises is formed there is still a possibility of a StackOverflowError.

Modifications:
- Use a ThreadLocal to save any DefaultPromises which could not be notified due to the stack being too large. After the first DefaultPromise on the stack completes notification this ThreadLocal should be used to notify any DefaultPromises which have not yet been notified.

Result:
DefaultPromise has StackOverflowError protection that works with all EventExecutor types.
2016-05-24 14:59:27 -07:00
Scott Mitchell
f5d58e2e1a DefaultPromise concurrency bug
Motivation:
If the executor changes while listeners are added and notification of listeners is being done then listeners can be notified out of order and concurrently. We should ensure that only one executor is used at any given time to notify listeners and ensure the listeners are notified in FIFO order.

Modifications:
- Move the notifyingListeners member variable from DefaultPromise into the synchronized block to prevent concurrent notification of listeners and preserve FIFO notification order

Result:
If the executor is changed for a DefaultPromise the listener notification order should be FIFO.
2016-05-24 11:46:43 -07:00
Norman Maurer
e10dca7601 Mark Recycler.recycle(...) deprecated and update usage.
Motivation:

Recycler.recycle(...) should not be used anymore and be replaced by Handle.recycle().

Modifications:

Mark it as deprecated and update usage.

Result:

Correctly document deprecated api.
2016-05-20 22:11:31 +02:00
Trustin Lee
da46928950 Fix NPE when creating DomainNameMapping via a builder 2016-05-18 19:26:28 +09:00
Trustin Lee
f11a35af32 Replace DomainMappingBuilder with DomainNameMappingBuilder
Motivation:

DomainMappingBuilder should have been named as DomainNameMappingBuilder
because it builds a DomainNameMapping.

Modifications:

- Add DomainNameMappingBuilder that does the same job with
  DomainMappingBuilder
- Deprecate DomainMappingBuilder and delegate its logic to
  DomainNameMappingBuilder
- Remove the references to the deprecated methods and classes related
  with domain name mapping
- Miscellaneous:
  - Fix Javadoc of DomainNameMapping.asMap()
  - Pre-create the unmodifiable map in DomainNameMapping

Result:

- Consistent naming
- Less use of deprecated API
2016-05-18 12:03:14 +02:00
Trustin Lee
ea1f60dbf0 Replace DomainNameMapping.entries() with asMap()
Motivation:

DomainNameMapping.entries() returns Set<Map.Entry<String, V>>, which
doesn't sound very natural.

Modifications:

Replace entries() with asMap() which returns a Map<String, V> instead.

Result:

- Better looking API
- User can do a lookup because it's a Map
2016-05-18 11:18:10 +02:00
Xiaoyan Lin
55dd7d035f Fix a class loader leak in ForkJoinPool
Motivation:

As reported in #4211, when using Netty in Tomcat (or other container based deployment), ForkJoinPool leaks an instance of `Submitter` so that the class loader of `Submitter` won't be GCed. However, since `Submitter` is just a wrapper of `int`, we can replace it with `int[1]`.

Modifications:

Replace `Submitter` with `int[1]`.

Result:

No class loader leak in ForkJoinPool when using in a container.
2016-05-18 10:53:33 +02:00
Xiaoyan Lin
a56ef03f24 Add DomainNameMapping.entries to allow retrieving the domain match lists
Motivation:

See #4200.

Modifications:

Add DomainNameMapping.entries to allow retrieving the domain match lists.

Result:

People can use DomainNameMapping.entries to retrive the match list in DomainNameMapping.
2016-05-17 09:47:45 +02:00
Scott Mitchell
1f8e192e21 Remove EventExecutor.children
Motivation:
EventExecutor.children uses generics in such a way that an entire colleciton must be cast to a specific type of object. This interface is not very flexible and is impossible to implement if the EventExecutor type must be wrapped. The current usage of this method also does not have any clear need within Netty. The Iterator interface allows for EventExecutor to be wrapped and forces the caller to make assumptions about types instead of building the assumptions into the interface.

Motivation:
- Remove EventExecutor.children and undeprecate the iterator() interface

Result:
EventExecutor interface has one less method and is easier to wrap.
2016-05-13 18:17:22 -07:00
Scott Mitchell
88a52f7367 KObjectHashMap PrimitiveIterator NPE
Motivation:
KObjectHashMap.remove(int index) attempts to move back items which may have been displaced because their spot in the hash based array was taken by another item. If this happens the nextIndex reference in PrimitiveIterator will not be updated. At this time the PrimitiveEntry will reference the incorrect index and may result in a NPE.

Modifications:
- If KObjectHashMap.remove(int index) moves entries back then PrimitiveIterator should adjust its nextIndex

Result:
PrimitiveIterator.remove() updates its internal state to reference the new nextIndex and will not NPE.
Fixes https://github.com/netty/netty/issues/5198
2016-05-13 12:44:29 -07:00
Scott Mitchell
c5faa142fb KObjectHashMap probeNext improvement
Motivation:
KObjectHashMap.probeNext(..) usually involves 2 conditional statements and 2 aritmatic operations. This can be improved to have 0 conditional statements.

Modifications:
- Use bit masking to avoid conditional statements

Result:
Improved performance for KObjecthashMap.probeNext(..)
2016-05-13 10:04:26 -07:00
Scott Mitchell
d580245afc DefaultHttp2Connection.close Reentrant Modification
Motivation:
The DefaultHttp2Conneciton.close method accounts for active streams being iterated and attempts to avoid reentrant modifications of the underlying stream map by using iterators to remove from the stream map. However there are a few issues:

- While iterating over the stream map we don't prevent iterations over the active stream collection
- Removing a single stream may actually remove > 1 streams due to closed non-leaf streams being preserved in the priority tree which may result in NPE

Preserving closed non-leaf streams in the priority tree is no longer necessary with our current allocation algorithms, and so this feature (and related complexity) can be removed.

Modifications:
- DefaultHttp2Connection.close should prevent others from iterating over the active streams and reentrant modification scenarios which may result from this
- DefaultHttp2Connection should not keep closed stream in the priority tree
  - Remove all associated code in DefaultHttp2RemoteFlowController which accounts for this case including the ReducedState object
  - This includes fixing writability changes which depended on ReducedState
- Update unit tests

Result:
Fixes https://github.com/netty/netty/issues/5198
2016-05-09 14:16:30 -07:00
Scott Mitchell
f2ed3e6ce8 DefaultPromise LateListener Logic Issues
Motivation:
The LateListener logic is prone to infinite loops and relies on being processed in the EventExecutor's thread for synchronization, but this EventExecutor may not be constant. An infinite loop can occur if the EventExecutor's execute method does not introduce a context switch in LateListener.run. The EventExecutor can be changed by classes which inherit from DefaultPromise. For example the DefaultChannelPromise will return w/e EventLoop the channel is registered to, but this EventLoop can change (re-registration).

Modifications:
- Remove the LateListener concept and instead use a single Object to maintain the listeners while still preserving notification order
- Make the result member variable an atomic variable so it can be outside the synchronized(this) blocks
- Cleanup/simplify existing state management code

Result:
Fixes https://github.com/netty/netty/issues/5185
2016-05-09 10:33:40 -07:00
Daniel Bevenius
ffd2450e11 Add logging configuration to pom.xml
Motivation:
Currently the default log level when running tests is debug. When
running the build on the CI server it might be nice to avoid this debug
level and allow for the level to be configured.

Modifications:
Added a logback-test.xml configuration that has been added to the
common module. This allows for the logLevel to be configured.
The default level will still be debug.

Result:
The log level can now be configured from the command line:
$ mvn test -DlogLevel=error
2016-05-09 15:18:01 +02:00
Norman Maurer
9229ed98e2 [#5088] Add annotation which marks packages/interfaces/classes as unstable
Motivation:

Some codecs should be considered unstable as these are relative new. For this purpose we should introduce an annotation which these codecs should us to be marked as unstable in terms of API.

Modifications:

- Add UnstableApi annotation and use it on codecs that are not stable
- Move http2.hpack to http2.internal.hpack as it is internal.

Result:

Better document unstable APIs.
2016-05-09 15:16:35 +02:00
Norman Maurer
572bdfb494 [maven-release-plugin] prepare for next development iteration 2016-04-10 08:37:18 +02:00
Norman Maurer
c6121a6f49 [maven-release-plugin] prepare release netty-4.1.0.CR7 2016-04-10 08:36:56 +02:00
Norman Maurer
6e919f70f8 [maven-release-plugin] rollback the release of netty-4.1.0.CR7 2016-04-09 22:13:44 +02:00
Norman Maurer
4cdd51509a [maven-release-plugin] prepare release netty-4.1.0.CR7 2016-04-09 22:05:34 +02:00
Xiaoyan Lin
bc6adab48c Make all InternalLoggerFactory implementations be singletons
Motivation:

It's better to make all InternalLoggerFactory implementations be singletons according to the discussions in #5047

Modifications:

Make all InternalLoggerFactory implementations be singletons and hide the construtors.

Result:

All InternalLoggerFactory implementations be singletons.
2016-04-08 09:39:32 +02:00
Kun Zhang
c024dc92d0 Explicit thread group on DefaultThreadFactory.
Motivation:

Fixes #5084. We (gRPC) encountered a bug that was triggered by
grpc/grpc-java@d927180. After that commit, event loop threads are
created per task by NioEventLoopGroup, and inherits the thread group of
the caller, which in our case is an application-provided request-scope
thread. Things go south when the application tries to manipulate (e.g.,
interrupt and join) all threads of the request-scope thread group, which
unexpectedly include the event loop threads.

Modifications:

DefaultThreadFactory will save the current thread group in constructor,
and apply it to all new threads.

Result:

Threads created by DefaultThreadFactory will be in the same thread group
as the thread where the factory is created.
2016-04-06 21:19:43 +02:00
Trustin Lee
4fa5d2cf52 Do now swallow an exception triggered by late listener notification
Related: #3449

Motivation:

When a user shut down an EventExecutor/Loop prematurely, a Promise will
fail to execute its listeners. When it happens, DefaultPromise will log
a message at ERROR level, but there's no way to get notified about it
programmatically.

Modifications:

Do not catch and log the RejectedExecutionException unconditionally,
but only catch and log for non-late listener notifications, so that a
user gets notified on submission failure at least when the listener is
late.

Result:

Remedies #3449 to some extent, although we will need fundamental fix for
that, such as #3566
2016-04-06 11:16:44 +02:00
Xiaoyan Lin
105df33d8d Add Log4J2LoggerFactory and Log4J2Logger
Motivation:

See #3095

Modifications:

Add Log4J2LoggerFactory and Log4J2Logger which is an InternalLogger implementation based on log4j2.

Result:

The user can use log4j2 directly without a special slf4j binding.
2016-04-05 14:01:32 +02:00
Trustin Lee
3b941c2a7c [maven-release-plugin] prepare for next development iteration 2016-04-02 01:25:05 -04:00
Trustin Lee
7368ccc539 [maven-release-plugin] prepare release netty-4.1.0.CR6 2016-04-02 01:24:55 -04:00
Michael Nitschinger
5d76daf33b Allow to customize NIO (channel) select strategies.
Motivation:

Under high throughput/low latency workloads, selector wakeups are
degrading performance when the incoming operations are triggered
from outside of the event loop. This is a common scenario for
"client" applications where the originating input is coming from
application threads rather from the socket attached inside the
event loops.

As a result, it can be desirable to defer the blocking select
so that incoming tasks (write/flush) do not need to wakeup
the selector.

Modifications:

This changeset adds the notion of a generic SelectStrategy which,
based on its contract, allows the implementation to optionally
defer the blocking select based on some custom criteria.

The default implementation resembles the original behaviour, that
is if tasks are in the queue `selectNow()` and move on, and if no
tasks need to be processed go into the blocking select and wait
for wakeup.

The strategy can be customized per `NioEventLoopGroup` in the
constructor.

Result:

High performance client applications are now given the chance to
customize for how long the actual selector blocking should be
deferred by employing a custom select strategy.
2016-03-30 15:01:25 -07:00
Norman Maurer
cee38ed2b6 [maven-release-plugin] prepare for next development iteration 2016-03-29 16:45:13 +02:00
Norman Maurer
9cd9e7daeb [maven-release-plugin] prepare release netty-4.1.0.CR5 2016-03-29 16:44:33 +02:00
Xiaoyan Lin
3ad55eb839 Speed up the slow path of FastThreadLocal
Motivation:

The current slow path of FastThreadLocal is much slower than JDK ThreadLocal. See #4418

Modifications:

- Add FastThreadLocalSlowPathBenchmark for the flow path of FastThreadLocal
- Add final to speed up the slow path of FastThreadLocal

Result:

The slow path of FastThreadLocal is improved.
2016-03-23 11:36:16 +01:00
Norman Maurer
28d03adbfe [maven-release-plugin] prepare for next development iteration 2016-03-21 11:51:50 +01:00
Norman Maurer
4653dc1d05 [maven-release-plugin] prepare release netty-4.1.0.CR4 2016-03-21 11:51:12 +01:00
Norman Maurer
4e1760c91b Allow disable Recycler via -Dio.netty.recycler.maxCapacity=0
Motivation:

It should be possible to disable the Recycler with -Dio.netty.recycler.maxCapacity=0, but because of a typo this is not the case.

Modifications:

Replace <= with < to make it posible to disable the Recycler.

Result:

Correct behaviour when using -Dio.netty.recycler.maxCapacity=0
2016-03-21 08:34:42 +01:00
Norman Maurer
b26652a934 Fix typo in log message during static init of Recycler.
Motivation:

Fix a typo in the log message of the static initializer of Recycler.

Modifications:

Fix typo.

Result:

Correctly log system property io.netty.recycler.maxCapacity.
2016-03-21 08:23:31 +01:00
johnou
26811b53ab Adding support for tcnative fedora flavour in uber jar
Motivation:

We want to allow the use of an uber jar that contains shared dynamic libraries for all platforms (including fedora).

Modifications:

Modified OpenSsl to try and load the fedora library if the OS is Linux and the platform specified library fails before using the default lib.

Result:

True uber support.
2016-03-15 13:56:41 +01:00
Scott Mitchell
45849b2fa8 Deprecate PromiseAggregator
Motivation:
PromiseAggregator's API allows for the aggregate promise to complete before the user is done adding promises. In order to support this use case the API structure would need to change in a breaking manner.

Modifications:
- Deprecate PromiseAggregator and subclasses
- Introduce PromiseCombiner which corrects these issues

Result:
PromiseCombiner corrects the deficiencies in PromiseAggregator.
2016-03-14 10:53:30 -07:00
Norman Maurer
97df3cb039 [#4936] NetUtil can prevent using Netty due to SecurityManager denial
Motivation:

A custom SecurityManager may prevent calling File.exists() and so throw a SecurityException in the static init block of NetUtil.

Modifications:

Correctly catch the exception and so allow to static init NetUtil.

Result:

Allow static init method of NetUtil to work even with custom SecurityManager.
2016-03-14 12:10:16 +01:00
Xiaoyan Lin
e2d4e22243 Add CharsetUtil.encoder/decoder() methods
Motivation:

See #3321

Modifications:

1. Add CharsetUtil.encoder/decoder() methods
2. Deprecate CharsetUtil.getEncoder/getDecoder() methods

Result:

Users can use new CharsetUtil.encoder/decoder() to specify error actions
2016-03-07 10:48:31 +00:00
Dmitry Spikhalskiy
0d3eda38e1 Helper method to get mime-type from Content-Type header of HttpMessage 2016-03-03 15:18:39 +01:00
Xiaoyan Lin
c7aadb5469 Fork SpscLinkedQueue and SpscLinkedAtomicQueue from JCTools
Motivation:

See #3746.

Modifications:

Fork SpscLinkedQueue and SpscLinkedAtomicQueue from JCTools based on 7846450e28

Result:

Add SpscLinkedQueue and SpscLinkedAtomicQueue and apply it in LocalChannel.
2016-02-22 15:55:52 +01:00
Norman Maurer
ca443e42e0 [maven-release-plugin] prepare for next development iteration 2016-02-19 23:00:11 +01:00
Norman Maurer
f39eb9a6b2 [maven-release-plugin] prepare release netty-4.1.0.CR3 2016-02-19 22:59:52 +01:00
Scott Mitchell
050ac709ba PromiseNotifier does not propagate cancel events
Motivation:
If the Future that the PromiseNotifier is listening to is cancelled, it does not propagate the cancel to all the promises it is expected to notify.

Modifications:
- If the future is cancelled then all the promises should be cancelled
- Add a UnaryPromiseNotifier if a collection of promises is not necessary

Result:
PromiseNotifier propagates cancel events to all promises
2016-02-19 10:16:04 -08:00
Moses Nakamura
f0f0b69d90 fixed "sensative" typo to read "sensitive" 2016-02-17 08:18:11 -08:00
Xiaoyan Lin
333f55e9ce Add unescapeCsvFields to parse a CSV line and implement CombinedHttpHeaders.getAll
Motivation:

See #4855

Modifications:

Unfortunately, unescapeCsv cannot be used here because the input could be a CSV line like `"a,b",c`. Hence this patch adds unescapeCsvFields to parse a CSV line and split it into multiple fields and unescaped them. The unit tests should define the behavior of unescapeCsvFields.

Then this patch just uses unescapeCsvFields to implement `CombinedHttpHeaders.getAll`.

Result:

`CombinedHttpHeaders.getAll` will return the unescaped values of a header.
2016-02-15 15:26:15 -08:00
Scott Mitchell
fdc6a5e87f Leak Detector disclosing when records dropped
Motivation:
ResourceLeakDetector enforces a limit as to how large the queue is allowed to grow for stack traces in order to keep memory from growing too large. However it is not always clear when records are dropped, or how many have been dropped. This can make interpreting leak reports more difficult if you assume all information is present when it may not be. Also we should increase the limit (currently 4) when running tests on the CI servers.

Modifications:
- Increase leak detector record limit on CI servers from 4 to 32.
- Track how many records have been discarded and disclose this in the leak report.

Result:
Leak reports clarify how many records were dropped, and how to increase the limit.
2016-02-12 09:57:43 -08:00
Xiaoyan Lin
f43dc7d551 Add unescapeCsv to StringUtil
Motivation:

See #3435

Modifications:

Add unescapeCsv to StringUtil

Result:

StringUtil has the counter part of escapeCsv: unescapeCsv
2016-02-08 14:35:30 -08:00
Fabian Lange
a51e2c8769 Expose Helper to obtain the "best" mac address.
Motivation:

The implementation of obtaining the best possible mac address is very good. There are many sub-par implementations proposed on stackoverflow.
While not strictly a netty concern, it would be nice to offer this util also to netty users.

Modifications:

extract DefaultChannelId#defaultMachineId code obtaining the "best" mac into a new helper called MacAddress, keep the random bytes fallback in DefaultChannelID.

Result:

New helper available.
2016-02-05 09:27:43 +01:00
Norman Maurer
75a2ddd61c [maven-release-plugin] prepare for next development iteration 2016-02-04 16:51:44 +01:00
Norman Maurer
7eb3a60dba [maven-release-plugin] prepare release netty-4.1.0.CR2 2016-02-04 16:37:06 +01:00
Xiaoyan Lin
b7415a3307 Add a reusable ArrayList to InternalThreadLocalMap
Motivation:

See #3411. A reusable ArrayList in InternalThreadLocalMap can avoid allocations in the following pattern:

```
List<...> list = new ArrayList<...>();

add something to list but never use InternalThreadLocalMap

return list.toArray(new ...[list.size()]);

```

Modifications:

Add a reusable ArrayList to InternalThreadLocalMap and update codes to use it.

Result:

Reuse a thread local ArrayList to avoid allocations.
2016-02-01 15:49:28 +01:00
Scott Mitchell
ca305d86fb PlatformDependent String char[] optimization
Motivation:
PlatformDependent0 has an optimization which grabs the char[] from a String. Since this code was introduced http://openjdk.java.net/jeps/254 has been gaining momentum in JDK 9. This JEP changes the internal storage from char[] to byte[], and thus the existing char[] only based optimizations will not work.

Modifications:
- The ASCII encoding char[] String optimizations should also work for byte[].

Result:
ASCII encoding char[] String optimizations don't break if the underlying storage in String is byte[].
2016-01-27 09:55:43 -08:00