Commit Graph

875 Commits

Author SHA1 Message Date
Norman Maurer
751cbf0f26 Fix compile error introduced by cherry-pick 6c4d6ae332 2019-03-05 09:26:44 +01:00
Norman Maurer
6c4d6ae332 Correctly monkey-patch id also in whe os / arch is used within library name. (#8913)
Motivation:

2bb9f64e16 introduced a change which made it possible to use different shaded versions of netty-tcnative on the classpath. This only partly worked as we did not correctly handled the case when os / arch is part of the library name (which is the case when netty-tcnative-boringssl-static is used with the uber jar).

Modifications:

- If patching the ID failed we retry again with the os / arch stripped
- Add unit tests to verify that patching ID now works with and without os / arch as suffix.

Result:

Using multiple shaded version of netty-tcnative-boringssl-static on MacOS works.
2019-03-05 09:11:50 +01:00
Norman Maurer
795f9600ba Adjust tests to be able to build / test when using IBM J9 / OpenJ9 (#8900)
Motivation:

We should run a CI job using J9 to ensure netty also works when using different JVMs.

Modifications:

- Adjust PooledByteBufAllocatorTest to be able to complete faster when using a JVM which takes longer when joining Threads (this seems to be the case with J9).

Result:

Be able to run CI against J9.
2019-03-01 06:48:35 +01:00
Norman Maurer
f58d074caf Tighten up contract of PromiseCombiner and so make it more safe to use (#8886)
Motivation:

PromiseCombiner is not thread-safe and even assumes all added Futures are using the same EventExecutor. This is kind of fragile as we do not enforce this. We need to enforce this contract to ensure it's safe to use and easy to spot concurrency problems.

Modifications:

- Add new contructor to PromiseCombiner that takes an EventExecutor and deprecate the old non-arg constructor.
- Check if methods are called from within the EventExecutor thread and if not fail
- Correctly dispatch on the right EventExecutor if the Future uses a different EventExecutor to eliminate concurrency issues.

Result:

More safe use of PromiseCombiner + enforce correct usage / contract.
2019-02-28 20:39:37 +01:00
Norman Maurer
94c33936db Add test for Iterator.remove() on KObjectHashMap.values().iterator() (#8891)
Motivation:

https://github.com/netty/netty/pull/8866 added support for calling Iterator.remove() but did not add a testcase.

Modifications:

Add testcase to ensure removal works.

Result:

Better test-coverage.
2019-02-27 12:06:46 +01:00
Michael André Pearce
a40655ad0c Support removal using values iterator. (#8866)
Motivation:

As ActiveMQ project using netty, we want to make use of this class, unfortunately the iterator on values(), seems to not support remove method, even so the delegated iterator does. Currently we have to clone and modify this class locally albeit a one line change is needed, it would be ideal if netty could allow remove, then removing the need to maintain a clone.  

Modifications:

* remove throws UnsupportedOperationException, and instead call remove method on delegated iterator

Result:

Be able to call Iterator.remove() for the values.
2019-02-26 21:29:07 +01:00
Eric Anderson
23190bd787 Log the shaded form of native workdir system property (#8867)
Motivation:

When users' /tmp is noexec, NativeLibraryLoader logs a message informing
them how to fix the problem by setting a system property. However, if
Netty has been shaded that message will tell them to set the un-shaded
system property name, which won't work.

Modifications:

Change the code to let shading tools rename the native.workdir property
name reference within user-visible log messages.

Notably, debug logs were _not_ changed, as there's many debug statements
including a variety of property names. Fixing them would be a much more
invasive change and have limited benefit.

Result:

The users will see the correctly-named system property to set if they
are using a noexec /tmp.
2019-02-14 15:19:08 -08:00
Norman Maurer
f6f7564602 Don't update state of PromiseCombiner when finish(null) is called (#8843)
Motivation:

When we fail a call to PromiseCombiner.finish(...) because of a null argument we must not update the internal state before throwing.

Modifications:

- First do the null check and only after we validated that the argument is not null update the internal state
- Add test case.

Modifications:

Do not mess up internal state of PromiseCombiner when finish(...) is called with a null argument.

Result:

After your change, what will change.
2019-02-04 19:15:57 +01:00
Dmitriy Dumanskiy
116f72db8d Legacy properties removed (#8839)
Motivation:

We can remove some properties for which we introduced replacements.

Modifications:

io.netty.buffer.bytebuf.checkAccessible, io.netty.leakDetectionLevel, org.jboss.netty.tryUnsafe properties removed

Result:

Code cleanup
2019-02-04 13:56:15 +01:00
田欧
e8efcd82a8 migrate java8: use requireNonNull (#8840)
Motivation:

We can just use Objects.requireNonNull(...) as a replacement for ObjectUtil.checkNotNull(....)

Modifications:

- Use Objects.requireNonNull(...)

Result:

Less code to maintain.
2019-02-04 10:32:25 +01:00
Dmitriy Dumanskiy
2e433889b2 Improve DateFormatter parsing performance (#8821)
Motivation:

Just was looking through code and found 1 interesting place DateFormatter.tryParseMonth that was not very effective, so I decided to optimize it a bit.

Modification:

Changed DateFormatter.tryParseMonth method. Instead of invocation regionMatch() for every month - compare chars one by one.

Result:

DateFormatter.parseHttpDate method performance improved from ~3% to ~15%.

Benchmark                                                                (DATE_STRING)   Mode  Cnt        Score       Error  Units
DateFormatter2Benchmark.parseHttpHeaderDateFormatter     Sun, 27 Jan 2016 19:18:46 GMT  thrpt    6  4142781.221 ± 82155.002  ops/s
DateFormatter2Benchmark.parseHttpHeaderDateFormatter     Sun, 27 Dec 2016 19:18:46 GMT  thrpt    6  3781810.558 ± 38679.061  ops/s
DateFormatter2Benchmark.parseHttpHeaderDateFormatterNew  Sun, 27 Jan 2016 19:18:46 GMT  thrpt    6  4372569.705 ± 30257.537  ops/s
DateFormatter2Benchmark.parseHttpHeaderDateFormatterNew  Sun, 27 Dec 2016 19:18:46 GMT  thrpt    6  4339785.100 ± 57542.660  ops/s
2019-02-04 10:04:35 +01:00
Nick Hill
3ac9363ef9 Fix varargs parameter logging in LocationAwareSlf4JLogger (#8834)
Motivation

As pointed out by @91he in
https://github.com/netty/netty/pull/8595#issuecomment-459181794, there
is a remaining bug in LocationAwareSlf4JLogger following the updates
done in #8595. The logging methods which take a varargs message
parameter array should format using MessageFormatter.arrayFormat rather
than MessageFormatter.format.

Modifications

Change varargs param methods in LocationAwareSlf4JLogger to use
MessageFormatter.arrayFormat and extend unit test to cover these cases.

Results

Correct log output when logging messages with > 2 parameters when using
LocationAwareSlf4JLogger.
2019-02-02 07:08:58 +01:00
Norman Maurer
abefbd7071 Do not schedule notify task if there are no listeners attached to the promise. (#8797)
Motivation:

If there are no listeners attached to the promise when full-filling it we do not need to schedule a task to notify.

Modifications:

- Don't schedule a task if there is nothing to notify.
- Add unit tests.

Result:

Fixes https://github.com/netty/netty/issues/8795.
2019-01-31 08:57:11 +01:00
Norman Maurer
d74682f123 Fix AppendableCharSequence.subSequence(...) where start == end. (#8798)
Motivation:

To conform to the CharSequence interface we need to return an empty CharSequence when start == end index and a subSequence is requested.

Modifications:

- Correctly handle the case where start == end
- Add unit test

Result:

Fix https://github.com/netty/netty/issues/8796.
2019-01-30 09:46:13 +01:00
田欧
6222101924 migrate java8: use lambda and method reference (#8781)
Motivation:

We can use lambdas now as we use Java8.

Modification:

use lambda function for all package, #8751 only migrate transport package.

Result:

Code cleanup.
2019-01-29 14:06:05 +01:00
田欧
e941cbe27a remove unused import statement (#8792)
Motivation:
The code contained some unused import statements.

Modification:
Remove unused import statements.

Result:
Code cleanup
2019-01-28 16:50:15 +01:00
田欧
934a07fbe2 migrate java8 (#8779)
Motivation:

We can omit argument types when using Java8.

Modification:

Omit arguments where possible.

Result:

Cleaner code.
2019-01-28 05:55:30 +01:00
kezhenxu94
7b6336f1fd Java 8 Migration: remove uneccessary if statement (#8755)
Motivation:

As netty 4.x supported Java 6 we had various if statements to check for java versions < 8. We can remove these now.

Modification:

Remove unnecessary if statements that check for java versions < 8.

Result:

Cleanup code.
2019-01-25 08:57:11 +01:00
Norman Maurer
310f31b392
Update to new checkstyle plugin (#8777)
Motivation:

We need to update to a new checkstyle plugin to allow the usage of lambdas.

Modifications:

- Update to new plugin version.
- Fix checkstyle problems.

Result:

Be able to use checkstyle plugin which supports new Java syntax.
2019-01-24 16:24:19 +01:00
Dmitriy Dumanskiy
9e9ee2ecd0 Java 8 migration. Join similar catch blocks (#8767)
Motivation:

We can now use multi-catch blocks.

Modification:

Join similar catch blocks + small cleanup.

Result:

Cleaner code.
2019-01-23 13:03:07 +01:00
Norman Maurer
3d6e6136a9
Decouple EventLoop details from the IO handling for each transport to… (#8680)
* Decouble EventLoop details from the IO handling for each transport to allow easy re-use of code and customization

Motiviation:

As today extending EventLoop implementations to add custom logic / metrics / instrumentations is only possible in a very limited way if at all. This is due the fact that most implementations are final or even package-private. That said even if these would be public there are the ability to do something useful with these is very limited as the IO processing and task processing are very tightly coupled. All of the mentioned things are a big pain point in netty 4.x and need improvement.

Modifications:

This changeset decoubled the IO processing logic from the task processing logic for the main transport (NIO, Epoll, KQueue) by introducing the concept of an IoHandler. The IoHandler itself is responsible to wait for IO readiness and process these IO events. The execution of the IoHandler itself is done by the SingleThreadEventLoop as part of its EventLoop processing. This allows to use the same EventLoopGroup (MultiThreadEventLoupGroup) for all the mentioned transports by just specify a different IoHandlerFactory during construction.

Beside this core API change this changeset also allows to easily extend SingleThreadEventExecutor / SingleThreadEventLoop to add custom logic to it which then can be reused by all the transports. The ideas are very similar to what is provided by ScheduledThreadPoolExecutor (that is part of the JDK). This allows for example things like:

  * Adding instrumentation / metrics:
    * how many Channels are registered on an SingleThreadEventLoop
    * how many Channels were handled during the IO processing in an EventLoop run
    * how many task were handled during the last EventLoop / EventExecutor run
    * how many outstanding tasks we have
    ...
    ...
  * Implementing custom strategies for choosing the next EventExecutor / EventLoop to use based on these metrics.
  * Use different Promise / Future / ScheduledFuture implementations
  * decorate Runnable / Callables when submitted to the EventExecutor / EventLoop

As a lot of functionalities are folded into the MultiThreadEventLoopGroup and SingleThreadEventLoopGroup this changeset also removes:

  * AbstractEventLoop
  * AbstractEventLoopGroup
  * EventExecutorChooser
  * EventExecutorChooserFactory
  * DefaultEventLoopGroup
  * DefaultEventExecutor
  * DefaultEventExecutorGroup

Result:

Fixes https://github.com/netty/netty/issues/8514 .
2019-01-23 08:32:05 +01:00
Dmitriy Dumanskiy
7b92ff2500 Java 8 migration. Remove ThreadLocalProvider and inline java.util.concurrent.ThreadLocalRandom.current() where necessary. (#8762)
Motivation:

Custom Netty ThreadLocalRandom and ThreadLocalRandomProvider classes are no longer needed and can be removed.

Modification:

Remove own ThreadLocalRandom

Result:

Less code to maintain
2019-01-22 20:14:28 +01:00
Dmitriy Dumanskiy
32d96a7f79 Java 8 migration. Similar catch blocks joined (#8759)
Motivation:

Avoid IDE warnings, easier to read.

Modification:

Same catch blocks joined.

Result:

Cleanup
2019-01-22 18:00:10 +01:00
Dmitriy Dumanskiy
42376c052a Java 8 migration. Inline PlatformDependent.newConcurrentHashMap() (#8760)
Motivation:

PlatformDependent.newConcurrentHashMap() is no longer needed so it could be easily removed and new ConcurrentHashMap<>() inlined instead of invoking PlatformDependent.newConcurrentHashMap().

Modification:

Use ConcurrentHashMap provided by the JDK directly.

Result:

Less code to maintain.
2019-01-22 17:18:50 +01:00
Dmitriy Dumanskiy
afc03da93b Java 8 migration. Use reference to StandardCharsets in CharsetUtil. (#8757)
Motivation:

No need to initialize charsets from the string. We can take already allocated charset from StandardCharsets class.

Modification:

Replace Charset.forName("US-ASCII") with StandardCharsets.US_ASCII.
Removed Charset[] values() method and internal static variable as it was used only in tests.

Result:

Reuse what the JDK provides
2019-01-22 16:50:36 +01:00
Oleksii Kachaiev
b3500f50a6 Java 8 migration. Replace Collection.newSetFromMap with ConcurrentHashMap.newKeySet (#8756)
Motivation:

Simplify code with newer Java API. It also reduce levels of indirection, but I don't
think we can gain any visible performance improvements out of this.

Modifications:

Replace Collection.newSetFromMap(new ConcurrentHashMap()) with ConcurrentHashMap.newKeySet().

Result:

Use Java8 APIs.
2019-01-22 16:41:15 +01:00
田欧
9d62deeb6f Java 8 migration: Use diamond operator (#8749)
Motivation:

We can use the diamond operator these days.

Modification:

Use diamond operator whenever possible.

Result:

More modern code and less boiler-plate.
2019-01-22 16:07:26 +01:00
Dmitriy Dumanskiy
c34340fff8 Java 8 migration. Auto close for try catch blocks (#8752)
Motivation:

Since Java 7 we can automatically close resources in try () construction.

Modification:

Changed all try catches in the code with autoclose try (resource)

Result:

Less boiler-plate
2019-01-22 15:57:30 +01:00
Dmitriy Dumanskiy
b2a9a8e9fe Java 8 migration. Removed custom LongCounter and LongAdderCounter classes and replaced with direct reference to LongAdder (#8750)
Motivation:

After switching to Java 8 we no longer need LongCounter and LongAdderCounter. We can directly replace them with LongAdder.

Modification:

Removed LongCounter and LongAdderCounter classes.

Result:

Less code to maintain
2019-01-22 13:53:28 +01:00
Dmitriy Dumanskiy
82b0db5013 Java 8 migration. Removed custom MathUtil.compare methods (#8748)
Motivation:

Netty uses own Integer.compare and Long.compare methods. Since Java 7 we can use Java implementation instead.

Modification:

Remove own implementation

Result:

Less code to maintain
2019-01-22 12:37:09 +01:00
Dmitriy Dumanskiy
91e7e3d8c9 Not used IntegerHolder class removed (#8747)
Motivation:

IntergerHolder was left from some old days and not used anymore.

Modification:

Remove IntergerHolder from the codebase.

Result:

Removed unused code
2019-01-22 11:25:08 +01:00
Dmitriy Dumanskiy
45c0ea543f Java 8 migration. Replace netty ConcurrentSet with Java KeySet. (#8745)
Motivation:

The concurrent set is present in Java 8 and above so we can use it instead of own implementation.

Modification:

io.netty.utik.internal.ConcurrentSet replaced with ConcurrentHashMap.newKeySet().

Result:

Less code to maintain.
2019-01-22 10:40:40 +01:00
kezhenxu94
2df8bca8da fix typo (#8741)
Motivation:

Correct typo

Modification:

Correct typo

Result:

JavaDoc and method name are more readable
2019-01-22 08:51:54 +01:00
kezhenxu94
ae1340fd72 extract duplicate code into method (#8720)
Motivation:

Clean up code to increase readability.

Modification:

Extract duplicate code blocks into method.

Result:

Less code duplication
2019-01-16 10:56:36 +01:00
Derek Lewis
f08e80ffd6 Remove unnecessary loop variable from AsciiString. (#8711)
Motivation:

Incrementing two variables in sync is not necessary when only one will do.

Modifications:

- Remove `j` from `for` loop and replace with `i`.
- Add more unit testing scenarios to cover changed code.

Results:

Unnecessary variable removed.
2019-01-15 08:33:48 +01:00
kashike
c0aa1ea5c7 Fix minor spelling issues in javadocs (#8701)
Motivation:

Javadocs contained some spelling errors, we should fix these.

Modification:

Fix spelling

Result:

Javadoc cleanup.
2019-01-14 07:25:13 +01:00
Norman Maurer
71430d8bbd Call FastThreadLocal.removeAll() before notify termination future of … (#8666)
Motivation:

We should try removing all FastThreadLocals for the Thread before we notify the termination. future. The user may block on the future and once it unblocks the JVM may terminate and start unloading classes.

Modifications:

Remove all FastThreadLocals for the Thread before notify termination future.

Result:

Fixes https://github.com/netty/netty/issues/6596.
2018-12-21 11:11:05 +01:00
Norman Maurer
7f22743d92
Revert "Ability to run a task at the end of an eventloop iteration." (#8637)
Motivation:

executeAfterEventLoopIteration is an Unstable API and isnt used in Netty. We should remove it to reduce complexity.

Changes:

This reverts commit 77770374fb.

Result:

Simplify implementation / cleanup.
2018-12-12 10:37:07 +01:00
Feri73
563793688f Adding support for whitespace in resource path in tests (#8606)
Motivation:

In windows if the project is in a path that contains whitespace,
resources cannot be accessed and tests fail.

Modifications:

Adds ResourcesUtil.java in netty-common. Tests use ResourcesUtil.java to access a resource.

Result:

Being able to build netty in a path containing whitespace
2018-12-12 10:29:19 +01:00
Nick Hill
fc0a4e15ab Harden ref-counting concurrency semantics (#8583)
Motivation

#8563 highlighted race conditions introduced by the prior optimistic
update optimization in 83a19d5650. These
were known at the time but considered acceptable given the perf
benefit in high contention scenarios.

This PR proposes a modified approach which provides roughly half the
gains but stronger concurrency semantics. Race conditions still exist
but their scope is narrowed to much less likely cases (releases
coinciding with retain overflow), and even in those
cases certain guarantees are still assured. Once release() returns true,
all subsequent release/retains are guaranteed to throw, and in
particular deallocate will be called at most once.

Modifications

- Use even numbers internally (including -ve) for live refcounts
- "Final" release changes to odd number (equivalent to refcount 0)
- Retain still uses faster getAndAdd, release uses CAS loop
- First CAS attempt uses non-volatile read
- Thread.yield() after a failed CAS provides a net gain

Result

More (though not completely) robust concurrency semantics for ref
counting; increased latency under high contention, but still roughly
twice as fast as the original logic. Bench results to follow
2018-11-29 08:32:50 +01:00
Norman Maurer
cb1090653f LocationAwareSlf4jLogger does not correctly format log message. (#8595)
Motivation:

We did miss to use MessageFormatter inside LocationAwareSlf4jLogger and so {} was not correctly replaced in log messages when using slf4j.
This regression was introduced by afe0767e9c.

Modifications:

- Make use of MessageFormatter
- Add unit test.

Result:

Fixes https://github.com/netty/netty/issues/8483.
2018-11-27 11:45:11 +01:00
Norman Maurer
2e0922d773 Use addAndGet(...) as a replacement for compareAndSet(...) when tracking the direct memory usage. (#8596)
Motivation:

We can change from using compareAndSet to addAndGet, which emits a different CPU instruction on x86 (CMPXCHG to XADD) when count direct memory usage. This instruction is cheaper in general and so produce less overhead on the "happy path". If we detect too much memory usage we just rollback the change before throwing the Error.

Modifications:

Replace compareAndSet(...) with addAndGet(...)

Result:

Less overhead when tracking direct memory.
2018-11-27 08:38:01 +01:00
Jake Luciani
63dc1f5aaa Allow adjusting of lead detection sampling interval. (#8568)
Motivation:

We should allow adjustment of the leak detecting sampling interval when in SAMPLE mode.

Modifications:

Added new int property io.netty.leakDetection.samplingInterval

Result:

Be able to consume changes made by the user.
2018-11-16 17:22:03 +01:00
Norman Maurer
845a65b31c
Nio|Epoll|KqueueEventLoop task execution might throw UnsupportedOperationException on shutdown. (#8476)
Motivation:

There is a racy UnsupportedOperationException instead because the task removal is delegated to MpscChunkedArrayQueue that does not support removal. This happens with SingleThreadEventExecutor that overrides the newTaskQueue to return an MPSC queue instead of the LinkedBlockingQueue returned by the base class such as NioEventLoop, EpollEventLoop and KQueueEventLoop.

Modifications:

- Catch the UnsupportedOperationException
- Add unit test.

Result:

Fix #8475
2018-11-15 07:19:28 +01:00
时无两丶
28f9136824 Replace ConcurrentHashMap at allLeaks with a thread-safe set (#8467)
Motivation:
allLeaks is to store the DefaultResourceLeak. When we actually use it, the key is DefaultResourceLeak, and the value is actually a meaningless value.
We only care about the keys of allLeaks and don't care about the values. So Set is more in line with this scenario.
Using Set as a container is more consistent with the definition of a container than Map.

Modification:

Replace allLeaks with set. Create a thread-safe set using 'Collections.newSetFromMap(new ConcurrentHashMap<DefaultResourceLeak<?>, Boolean>()).'
2018-11-06 11:21:56 +01:00
Norman Maurer
bde2865ef8
Make it clear that HashedWheelTimer only support millis. (#8322)
Motivation:

HWT does not support anything smaller then 1ms so we should make it clear that this is the case.

Modifications:

Log a warning if < 1ms is used.

Result:

Less suprising behaviour.
2018-11-02 08:10:18 +01:00
Norman Maurer
d533befa96
PlatformDependent.maxDirectMemory() must respect io.netty.maxDirectMemory (#8452)
Motivation:

In netty we use our own max direct memory limit that can be adjusted by io.netty.maxDirectMemory but we do not take this in acount when maxDirectMemory() is used. That will lead to non optimal configuration of PooledByteBufAllocator in some cases.

This came up on stackoverflow:
https://stackoverflow.com/questions/53097133/why-is-default-num-direct-arena-derived-from-platformdependent-maxdirectmemory

Modifications:

Correctly respect io.netty.maxDirectMemory and so configure PooledByteBufAllocator correctly by default.

Result:

Correct value for max direct memory.
2018-11-02 07:19:43 +01:00
Nick Hill
d7fa7be67f Exploit PlatformDependent.allocateUninitializedArray() in more places (#8393)
Motivation:

There are currently many more places where this could be used which were
possibly not considered when the method was added.

If https://github.com/netty/netty/pull/8388 is included in its current
form, a number of these places could additionally make use of the same
BYTE_ARRAYS threadlocal.

There's also a couple of adjacent places where an optimistically-pooled
heap buffer is used for temp byte storage which could use the
threadlocal too in preference to allocating a temp heap bytebuf wrapper.
For example
https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/ByteBufUtil.java#L1417.

Modifications:

Replace new byte[] with PlatformDependent.allocateUninitializedArray()
where appropriate; make use of ByteBufUtil.getBytes() in some places
which currently perform the equivalent logic, including avoiding copy of
backing array if possible (although would be rare).

Result:

Further potential speed-up with java9+ and appropriate compile flags.
Many of these places could be on latency-sensitive code paths.
2018-10-27 10:43:28 -05:00
almson
1cc692dd7d Fix incorrect reachability assumption in ResourceLeakDetector (#8410)
Motivation:

trackedObject != null gives no guarantee that trackedObject remains reachable. This may cause problems related to premature finalization: false leak detector warnings.
 
Modifications:

Add private method reachabilityFence0 that works on JDK 8 and can be factored out into PlatformDependent. Later, it can be swapped for the real Reference.reachabilityFence.
 
Result:

No false leak detector warnings in future versions of JDK.
2018-10-24 22:15:13 +02:00
almson
fc35e20e2c Include correct duped value in DefaultResourceLeak.toString() (#8413)
Motivation:

DefaultResourceLeak.toString() did include the wrong value for duplicated records.

Modifications:

Include the correct value.

Result:

Correct toString() implementation.
2018-10-22 15:01:38 +02:00