Commit Graph

705 Commits

Author SHA1 Message Date
Norman Maurer
57672d9854 Use ResourceLeakDetectorFactory in HashedWheelTimer
Motivation:

We recently added the ResourceLeakDetectorFactory but missed to updated HashedWheelTimer to use it.

Modifications:

- Add new abstract method to ResourceLeakDetectorFactory that allows to provide also samplingInterval and maxActive args.
- Deprecate most constructors in ResourceLeakDetector and add doc explaining that people should use ResourceLeakDetectorFactory

Result:

Custom ResourceLeakDetectorFactory will also be used in HashedWheelTimer if configured.
2016-06-29 21:07:17 +02:00
Norman Maurer
bd0a74fca3 Allow to disable ResourceLeak creation when worker thread is deamon in HashedWheelTimer
Motivation:

Sometimes a shared HashedWheelTimer can not easily be stopped in a good place. If the worker thread is daemon this is not a big deal and we should allow to not log a leak.

Modifications:

Add another constructor which allows to disable resource leak detection if worker thread is used.

Result:

Not log resource leak when HashedWheelTimer is not stopped and the  worker thread is a deamon thread.
2016-06-29 20:06:58 +02:00
Scott Mitchell
857ad73676 DefaultPromise make MAX_STACK_DEPTH configurable
Motivation:
Some Netty use cases may want to configure the max allowed stack depth for promise listener notification.

Modifications:
- Add a system property so that this value can be configured.

Result:
DefaultPromise's max stack depth is configurable.
2016-06-29 08:31:27 -07:00
Scott Mitchell
70651cc58d HpackUtil.equals performance improvement
Motivation:
PR #5355 modified interfaces to reduce GC related to the HPACK code. However this came with an anticipated performance regression related to HpackUtil.equals due to AsciiString's increase cost of charAt(..). We should mitigate this performance regression.

Modifications:
- Introduce an equals method in PlatformDependent which doesn't leak timing information and use this in HpcakUtil.equals

Result:
Fixes https://github.com/netty/netty/issues/5436
2016-06-27 14:37:39 -07:00
Guido Medina
f0a5ee068f Update dependencies and plugins to latest possible versions.
Motivation:
It is good to have used dependencies and plugins up-to-date to fix any undiscovered bug fixed by the authors.

Modification:
Scanned dependencies and plugins and carefully updated one by one.

Result:
Dependencies and plugins are up-to-date.
2016-06-27 13:35:35 +02:00
Norman Maurer
731f52fdf7 Allow to inject RejectedExecutionHandler for different EventLoops and EventExecutors
Motiviation:

Sometimes it is useful to allow to specify a custom strategy to handle rejected tasks. For example if someone tries to add tasks from outside the eventloop it may make sense to try to backoff and retries and so give the executor time to recover.

Modification:

Add RejectedEventExecutor interface and implementations and allow to inject it.

Result:

More flexible handling of executor overload.
2016-06-24 17:08:30 +02:00
Norman Maurer
a0d4fb16fc Allow to set max capacity for task queue for EventExecutors and EventLoops
Motivation:

To restrict the memory usage of a system it is sometimes needed to adjust the number of max pending tasks in the tasks queue.

Modifications:

- Add new constructors to modify the number of allowed pending tasks.
- Add system properties to configure the default values.

Result:

More flexible configuration.
2016-06-24 14:00:50 +02:00
Norman Maurer
278c36af0c Merge ThrowableUtils into ThrowableUtil.
Motivation:

We should merge ThrowableUtils into ThrowableUtil as this name is more consistent with the naming of utility classes in netty.

Modifications:

Merge classes.

Result:

More consistent naming
2016-06-23 10:05:41 +02:00
Norman Maurer
e845670043 Set some StackTraceElement on pre-instantiated static exceptions
Motivation:

We use pre-instantiated exceptions in various places for performance reasons. These exceptions don't include a stacktrace which makes it hard to know where the exception was thrown. This is especially true as we use the same exception type (for example ChannelClosedException) in different places. Setting some StackTraceElements will provide more context as to where these exceptions original and make debugging easier.

Modifications:

Set a generated StackTraceElement on these pre-instantiated exceptions which at least contains the origin class and method name. The filename and linenumber are specified as unkown (as stated in the javadocs of StackTraceElement).

Result:

Easier to find the origin of a pre-instantiated exception.
2016-06-20 11:33:05 +02:00
agonigberg
3288cacf8d Pluggable resource leak detector
Allow users of Netty to plug in their own leak detector for the purpose
of instrumentation.

Motivation:

We are rolling out a large Netty deployment and want to be able to
track the amount of leaks we're seeing in production via custom
instrumentation. In order to achieve this today, I had to plug in a
custom `ByteBufAllocator` into the bootstrap and have it initialize a
custom `ResourceLeakDetector`. Due to these classes mostly being marked
`final` or having private or static methods, a lot of the code had to
be copy-pasted and it's quite ugly.

Modifications:

* I've added a static loader method for the `ResourceLeakDetector` in
`AbstractByteBuf` that tries to instantiate the class passed in via the
`-Dio.netty.customResourceLeakDetector`, otherwise falling back to the
default one.
* I've modified `ResourceLeakDetector` to be non-final and to have the
reporting broken out in to methods that can be overridden.

Result:

You can instrument leaks in your application by just adding something
like the following:

```java
public class InstrumentedResourceLeakDetector<T> extends
ResourceLeakDetector<T> {

    @Monitor("InstanceLeakCounter")
    private final AtomicInteger instancesLeakCounter;

    @Monitor("LeakCounter")
    private final AtomicInteger leakCounter;

    public InstrumentedResourceLeakDetector(Class<T> resource) {
        super(resource);
        this.instancesLeakCounter = new AtomicInteger();
        this.leakCounter = new AtomicInteger();
    }

    @Override
    protected void reportTracedLeak(String records) {
        super.reportTracedLeak(records);
        leakCounter.incrementAndGet();
    }

    @Override
    protected void reportUntracedLeak() {
        super.reportUntracedLeak();
        leakCounter.incrementAndGet();
    }

    @Override
    protected void reportInstancesLeak() {
        super.reportInstancesLeak();
        instancesLeakCounter.incrementAndGet();
    }
}
```
2016-06-20 11:14:44 +02:00
Norman Maurer
8ccc795314 Expose DefaultThreadFactory.threadGroup to sub-classes
Motivation:

DefaultThreadFactory allows to override the newThread(...) method and so should have access to all fields that are set via the constructor.

Modifications:

Change threadGroup from private to protected visibility.

Result:

Easier to extend DefaultThreadFactory.
2016-06-17 06:23:53 +02:00
Dmitry Spikhalskiy
428c61673b Logs in invokeExceptionCaught have been made consistent and full
Motivation:

In case of exception in invokeExceptionCaught() only original exception passed to invokeExceptionCaught() will be logged on any log level.
+ AbstractChannelHandlerContext and CombinedChannelDuplexHandler log different exceptions.

Modifications:

Fix inconsistent logging code and add ability to see both stacktraces on DEBUG level.

Result:

Both handlers log now both original exception and thrown from invokeExceptionCaught. To see full stacktrace of exception thrown from invokeExceptionCaught DEBUG log level must be enabled.
2016-06-11 20:11:11 +02:00
Guido Medina
b921f80057 Make JCtools available (provided scope) for tests and optional OSGI, issue #5383 2016-06-11 07:15:48 +02:00
Guido Medina
c3abb9146e Use shaded dependency on JCTools instead of copy and paste
Motivation:
JCTools supports both non-unsafe, unsafe versions of queues and JDK6 which allows us to shade the library in netty-common allowing it to stay "zero dependency".

Modifications:
- Remove copy paste JCTools code and shade the library (dependencies that are shaded should be removed from the <dependencies> section of the generated POM).
- Remove usage of OneTimeTask and remove it all together.

Result:
Less code to maintain and easier to update JCTools and less GC pressure as the queue implementation nt creates so much garbage
2016-06-10 13:19:45 +02:00
Scott Mitchell
56a2f64665 Clarify Future.removeListener[s] javaDocs
Motivation:
The javaDocs for Future.removeListener do not clarify that only the first occurrence of the listener is guaranteed to be removed.

Modifications:
- Clarify the javaDocs for Future.removeListener[s] so it is known that the only the first occurrence of the listener will be removed.

Result:
Fixes https://github.com/netty/netty/issues/5351
2016-06-08 15:48:26 -07:00
Norman Maurer
4dec7f11b7 [maven-release-plugin] prepare for next development iteration 2016-06-07 18:52:34 +02:00
Norman Maurer
cf670fab75 [maven-release-plugin] prepare release netty-4.1.1.Final 2016-06-07 18:52:22 +02:00
Norman Maurer
b461c9d54c Allow to specify a custom EventExecutorChooserFactory. Related to [#1230]
Motivation:

Sometimes it may be benefitially for an user to specify a custom algorithm when choose the next EventExecutor/EventLoop.

Modifications:

Allow to specify a custom EventExecutorChooseFactory that allows to customize algorithm.

Result:

More flexible api.
2016-06-06 11:04:56 +02:00
Norman Maurer
584fbac6ed [#3419] Only use SecureRandom to generate initialSeed if requested
Motivation:

We tried to always use SecureRandom to generate the initialSeed for our ThreadLocalRandom, this can sometimes give warnings under normal usage. We should better not use SecureRandom as default (just as the implementation in jsr166y does) and only try if the user specified -Djava.util.secureRandomSeed=true .

Modifications:

Only try to use SecureRandom when -Djava.util.secureRandomSeed=true is used.

Result:

Less likely to see entropy warnings.
2016-06-06 09:09:29 +02:00
Norman Maurer
3a7dcde320 [#5224] Allow to use Unsafe.reallocateMemory(...) in UnpooledUnsafeNoCleanerDirectByteBuf.
Motivation:

If the user uses unsafe direct buffers with no cleaner we can use Unsafe.reallocateMemory(...) as optimization when we need to expand the buffer.

Modifications:

Use Unsafe.relocateMemory(...) in UnpooledUnsafeNoCleanerDirectByteBuf.

Result:

Less expensive expanding of buffers.
2016-06-04 19:21:41 +02:00
Norman Maurer
3d29bcfc8d Allow to create Unsafe ByteBuf implementations that not use a Cleaner to clean the native memory.
Motivation:

Using the Cleaner to release the native memory has a few drawbacks:

- Cleaner.clean() uses static synchronized internally which means it can be a performance bottleneck
- It put more load on the GC

Modifications:

Add new buffer implementations that can be enabled with a system flag as optimizations. In this case no Cleaner is used at all and the user must ensure everything is always released.

Result:

Less performance impact by direct buffers when need to be allocated and released.
2016-06-03 21:20:10 +02:00
Scott Mitchell
f6ad9df8ac DefaultPromise StackOverflowError protection updates
Modifications:
DefaultPromise provides a ThreadLocal queue to protect against StackOverflowError because of executors which may immediately execute runnables instead of queue them (i.e. ImmediateEventExecutor). However this may be better addressed by fixing these executors to protect against StackOverflowError instead of just fixing for a single use case. Also the most commonly used executors already provide the desired behavior and don't need the additional overhead of a ThreadLocal queue in DefaultPromise.

Modifications:
- Remove ThreadLocal queue from DefaultPromise
- Change ImmediateEventExecutor so it maintains a queue of runnables if reentrant condition occurs

Result:
DefaultPromise StackOverflowError code is simpler, and ImmediateEventExecutor protects against StackOverflowError.
2016-06-02 09:22:47 -07:00
Alex Petrov
bbed330468 Fix the possible reference leak in Recycler
Motivation:

Under very unlikely (however possible) circumstances, Recycler may leak
references. This happens _only_ when the object was already recycled
at least once (which means it's got written to the stack) and then
taken out again, and never returned.

The "never returned" part may be the fault of the user (forgotten
`finally` clause) or the situation when Recycler drops the possibly
youngest item itself.

Modifications:

Nullify the item taken from the stack.

Result:

Reference is cleaned up. If the object is lost, it will be a subject for
GC. The rest of Stack / Recycler functionality remains unaffected.
2016-06-01 06:47:43 +02:00
Norman Maurer
1dfcfc17fa Allow to change link capacity via system property
Motivation:

Sometimes people may want to trade GC with memory overhead. For this it can be useful to allow to change the capacity of the array that is hold in the Link that is used by the Recycler internally.

Modifications:

Introduce a new system property , io.netty.recycler.linkCapacity which allows to change the capcity.

Result:

More flexible configuration of netty.
2016-05-31 14:05:23 +02:00
Milos Fabian
4c186c4c41 Make netty-transport-native-epoll-*-linux-x86_64.jar working in OSGi
Motivation:

Currenlty, netty-transport-native-epoll-*-linux-x86_64.jar is not packed as OSGi bundle
and thus not working in OSGi environment.

Modifications:

In netty-transport-native-epoll's pom.xml added configuration
to attach manifest to the jar with a native library.
In netty-common's pom.xml added configuration instruction (DynamicImport-Package)
to maven bnd plugin to make sure the native code is loaded from
netty-transport-native-epoll bundle.

Result:

The netty-transport-native-epoll-*-linux-x86_64.jar is a bundle (MANIFEST.MF attached)
and the inluced native library can be successfuly loaded in OSGi environment.
Fixing #5119
2016-05-30 22:04:01 +02:00
Norman Maurer
db6b72da19 Add optimized version of setZero(...) / writeZero(...) for Unsafe*ByteBuf implementations
Motivation:

Unsafe offers a method to set memory to a specific value. This can be used to implement an optimized version of setZero(...) and writeZero(...)

Modifications:

Add implementation for all Unsafe*ByteBuf implementations.

Result:

Faster setZero(...) and writeZero(...)
2016-05-30 15:10:15 +02:00
Norman Maurer
86f53083e7 [#5297] Ensure calling NioEventLoop.pendingTasks() and EpollEventLoop.pendingTasks() will not produce livelock
Motivation:

SingleThreadEventExecutor.pendingTasks() will call taskQueue.size() to get the number of pending tasks in the queue. This is not safe when using MpscLinkedQueue as size() is only allowed to be called by a single consumer.

Modifications:

Ensure size() is only called from the EventLoop.

Result:

No more livelock possible when call pendingTasks, no matter from which thread it is done.
2016-05-28 20:04:07 +02:00
Norman Maurer
6ca49d1336 [maven-release-plugin] prepare for next development iteration 2016-05-25 19:16:44 +02:00
Norman Maurer
446b38db52 [maven-release-plugin] prepare release netty-4.1.0.Final 2016-05-25 19:14:15 +02:00
Scott Mitchell
b3c56d5f69 DefaultPromise StackOverflowError protection
Motivation:
f2ed3e6ce8 removed the previous mechanism for StackOverflowError because it didn't work in all cases (i.e. ImmediateExecutor). However if a chain of listeners which complete other promises is formed there is still a possibility of a StackOverflowError.

Modifications:
- Use a ThreadLocal to save any DefaultPromises which could not be notified due to the stack being too large. After the first DefaultPromise on the stack completes notification this ThreadLocal should be used to notify any DefaultPromises which have not yet been notified.

Result:
DefaultPromise has StackOverflowError protection that works with all EventExecutor types.
2016-05-24 14:59:27 -07:00
Scott Mitchell
f5d58e2e1a DefaultPromise concurrency bug
Motivation:
If the executor changes while listeners are added and notification of listeners is being done then listeners can be notified out of order and concurrently. We should ensure that only one executor is used at any given time to notify listeners and ensure the listeners are notified in FIFO order.

Modifications:
- Move the notifyingListeners member variable from DefaultPromise into the synchronized block to prevent concurrent notification of listeners and preserve FIFO notification order

Result:
If the executor is changed for a DefaultPromise the listener notification order should be FIFO.
2016-05-24 11:46:43 -07:00
Norman Maurer
e10dca7601 Mark Recycler.recycle(...) deprecated and update usage.
Motivation:

Recycler.recycle(...) should not be used anymore and be replaced by Handle.recycle().

Modifications:

Mark it as deprecated and update usage.

Result:

Correctly document deprecated api.
2016-05-20 22:11:31 +02:00
Trustin Lee
da46928950 Fix NPE when creating DomainNameMapping via a builder 2016-05-18 19:26:28 +09:00
Trustin Lee
f11a35af32 Replace DomainMappingBuilder with DomainNameMappingBuilder
Motivation:

DomainMappingBuilder should have been named as DomainNameMappingBuilder
because it builds a DomainNameMapping.

Modifications:

- Add DomainNameMappingBuilder that does the same job with
  DomainMappingBuilder
- Deprecate DomainMappingBuilder and delegate its logic to
  DomainNameMappingBuilder
- Remove the references to the deprecated methods and classes related
  with domain name mapping
- Miscellaneous:
  - Fix Javadoc of DomainNameMapping.asMap()
  - Pre-create the unmodifiable map in DomainNameMapping

Result:

- Consistent naming
- Less use of deprecated API
2016-05-18 12:03:14 +02:00
Trustin Lee
ea1f60dbf0 Replace DomainNameMapping.entries() with asMap()
Motivation:

DomainNameMapping.entries() returns Set<Map.Entry<String, V>>, which
doesn't sound very natural.

Modifications:

Replace entries() with asMap() which returns a Map<String, V> instead.

Result:

- Better looking API
- User can do a lookup because it's a Map
2016-05-18 11:18:10 +02:00
Xiaoyan Lin
55dd7d035f Fix a class loader leak in ForkJoinPool
Motivation:

As reported in #4211, when using Netty in Tomcat (or other container based deployment), ForkJoinPool leaks an instance of `Submitter` so that the class loader of `Submitter` won't be GCed. However, since `Submitter` is just a wrapper of `int`, we can replace it with `int[1]`.

Modifications:

Replace `Submitter` with `int[1]`.

Result:

No class loader leak in ForkJoinPool when using in a container.
2016-05-18 10:53:33 +02:00
Xiaoyan Lin
a56ef03f24 Add DomainNameMapping.entries to allow retrieving the domain match lists
Motivation:

See #4200.

Modifications:

Add DomainNameMapping.entries to allow retrieving the domain match lists.

Result:

People can use DomainNameMapping.entries to retrive the match list in DomainNameMapping.
2016-05-17 09:47:45 +02:00
Scott Mitchell
1f8e192e21 Remove EventExecutor.children
Motivation:
EventExecutor.children uses generics in such a way that an entire colleciton must be cast to a specific type of object. This interface is not very flexible and is impossible to implement if the EventExecutor type must be wrapped. The current usage of this method also does not have any clear need within Netty. The Iterator interface allows for EventExecutor to be wrapped and forces the caller to make assumptions about types instead of building the assumptions into the interface.

Motivation:
- Remove EventExecutor.children and undeprecate the iterator() interface

Result:
EventExecutor interface has one less method and is easier to wrap.
2016-05-13 18:17:22 -07:00
Scott Mitchell
88a52f7367 KObjectHashMap PrimitiveIterator NPE
Motivation:
KObjectHashMap.remove(int index) attempts to move back items which may have been displaced because their spot in the hash based array was taken by another item. If this happens the nextIndex reference in PrimitiveIterator will not be updated. At this time the PrimitiveEntry will reference the incorrect index and may result in a NPE.

Modifications:
- If KObjectHashMap.remove(int index) moves entries back then PrimitiveIterator should adjust its nextIndex

Result:
PrimitiveIterator.remove() updates its internal state to reference the new nextIndex and will not NPE.
Fixes https://github.com/netty/netty/issues/5198
2016-05-13 12:44:29 -07:00
Scott Mitchell
c5faa142fb KObjectHashMap probeNext improvement
Motivation:
KObjectHashMap.probeNext(..) usually involves 2 conditional statements and 2 aritmatic operations. This can be improved to have 0 conditional statements.

Modifications:
- Use bit masking to avoid conditional statements

Result:
Improved performance for KObjecthashMap.probeNext(..)
2016-05-13 10:04:26 -07:00
Scott Mitchell
d580245afc DefaultHttp2Connection.close Reentrant Modification
Motivation:
The DefaultHttp2Conneciton.close method accounts for active streams being iterated and attempts to avoid reentrant modifications of the underlying stream map by using iterators to remove from the stream map. However there are a few issues:

- While iterating over the stream map we don't prevent iterations over the active stream collection
- Removing a single stream may actually remove > 1 streams due to closed non-leaf streams being preserved in the priority tree which may result in NPE

Preserving closed non-leaf streams in the priority tree is no longer necessary with our current allocation algorithms, and so this feature (and related complexity) can be removed.

Modifications:
- DefaultHttp2Connection.close should prevent others from iterating over the active streams and reentrant modification scenarios which may result from this
- DefaultHttp2Connection should not keep closed stream in the priority tree
  - Remove all associated code in DefaultHttp2RemoteFlowController which accounts for this case including the ReducedState object
  - This includes fixing writability changes which depended on ReducedState
- Update unit tests

Result:
Fixes https://github.com/netty/netty/issues/5198
2016-05-09 14:16:30 -07:00
Scott Mitchell
f2ed3e6ce8 DefaultPromise LateListener Logic Issues
Motivation:
The LateListener logic is prone to infinite loops and relies on being processed in the EventExecutor's thread for synchronization, but this EventExecutor may not be constant. An infinite loop can occur if the EventExecutor's execute method does not introduce a context switch in LateListener.run. The EventExecutor can be changed by classes which inherit from DefaultPromise. For example the DefaultChannelPromise will return w/e EventLoop the channel is registered to, but this EventLoop can change (re-registration).

Modifications:
- Remove the LateListener concept and instead use a single Object to maintain the listeners while still preserving notification order
- Make the result member variable an atomic variable so it can be outside the synchronized(this) blocks
- Cleanup/simplify existing state management code

Result:
Fixes https://github.com/netty/netty/issues/5185
2016-05-09 10:33:40 -07:00
Daniel Bevenius
ffd2450e11 Add logging configuration to pom.xml
Motivation:
Currently the default log level when running tests is debug. When
running the build on the CI server it might be nice to avoid this debug
level and allow for the level to be configured.

Modifications:
Added a logback-test.xml configuration that has been added to the
common module. This allows for the logLevel to be configured.
The default level will still be debug.

Result:
The log level can now be configured from the command line:
$ mvn test -DlogLevel=error
2016-05-09 15:18:01 +02:00
Norman Maurer
9229ed98e2 [#5088] Add annotation which marks packages/interfaces/classes as unstable
Motivation:

Some codecs should be considered unstable as these are relative new. For this purpose we should introduce an annotation which these codecs should us to be marked as unstable in terms of API.

Modifications:

- Add UnstableApi annotation and use it on codecs that are not stable
- Move http2.hpack to http2.internal.hpack as it is internal.

Result:

Better document unstable APIs.
2016-05-09 15:16:35 +02:00
Norman Maurer
572bdfb494 [maven-release-plugin] prepare for next development iteration 2016-04-10 08:37:18 +02:00
Norman Maurer
c6121a6f49 [maven-release-plugin] prepare release netty-4.1.0.CR7 2016-04-10 08:36:56 +02:00
Norman Maurer
6e919f70f8 [maven-release-plugin] rollback the release of netty-4.1.0.CR7 2016-04-09 22:13:44 +02:00
Norman Maurer
4cdd51509a [maven-release-plugin] prepare release netty-4.1.0.CR7 2016-04-09 22:05:34 +02:00
Xiaoyan Lin
bc6adab48c Make all InternalLoggerFactory implementations be singletons
Motivation:

It's better to make all InternalLoggerFactory implementations be singletons according to the discussions in #5047

Modifications:

Make all InternalLoggerFactory implementations be singletons and hide the construtors.

Result:

All InternalLoggerFactory implementations be singletons.
2016-04-08 09:39:32 +02:00
Kun Zhang
c024dc92d0 Explicit thread group on DefaultThreadFactory.
Motivation:

Fixes #5084. We (gRPC) encountered a bug that was triggered by
grpc/grpc-java@d927180. After that commit, event loop threads are
created per task by NioEventLoopGroup, and inherits the thread group of
the caller, which in our case is an application-provided request-scope
thread. Things go south when the application tries to manipulate (e.g.,
interrupt and join) all threads of the request-scope thread group, which
unexpectedly include the event loop threads.

Modifications:

DefaultThreadFactory will save the current thread group in constructor,
and apply it to all new threads.

Result:

Threads created by DefaultThreadFactory will be in the same thread group
as the thread where the factory is created.
2016-04-06 21:19:43 +02:00