Commit Graph

1163 Commits

Author SHA1 Message Date
feijermu
e648aa259d
Add a null check to NetUtil. sysctlGetInt(...) (#10027)
Motivation:

BufferedReader.readLine() may return null and cause a NPE.

Modification:

Simply add a null check.

Result:

If BufferedReader.readLine() returns null, the sysctlGetInt will just return null rather than cause NPE.
2020-02-14 09:08:27 +01:00
Norman Maurer
713e0aa68a
java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/os-release" "read") (#10018)
Motivation:

Modifications:

- Wrap the code and execute with an AccessController
- Ignore SecurityException (by just logging it)
- Add some more debug logging

Result:

Fixes https://github.com/netty/netty/issues/10017
2020-02-13 11:48:40 +01:00
violetagg
fcf55fcf71
When BlockHound is installed, do not report GlobalEventExecutor/SingleThreadEventExecutor#takeTask as blocking call. (#10020)
Motivation:

GlobalEventExecutor/SingleThreadEventExecutor#taskQueue is BlockingQueue.

Modifications:

Add allowBlockingCallsInside configuration for GlobalEventExecutor/SingleThreadEventExecutor#takeTask.

Result:

Fixes #9984
When BlockHound is installed, GlobalEventExecutor/SingleThreadEventExecutor#takeTask is not reported as a blocking call.
2020-02-11 20:24:41 +01:00
Ruwei
6e5f229589
fix bug: scheduled tasks may not be executed (#9980)
Motivation:

If there was always a task in the taskQueue of GlobalEvenExecutor, scheduled tasks in the
scheduledTaskQueue will never be executed.

Related to  #1614

Modifications:

fix bug in GlobalEventExecutor#takeTask

Result:

fix bug
2020-01-31 10:57:38 +01:00
Johno Crawford
0671b18e24
SSL / BlockHound works out of the box with the default SSL provider (#9969)
Motivation:

JDK is the default SSL provider and internally uses blocking IO operations.

Modifications:

Add allowBlockingCallsInside configuration for SslHandler runAllDelegate function.

Result:

When BlockHound is installed, SSL works out of the box with the default SSL provider.


Co-authored-by: violetagg <milesg78@gmail.com>
2020-01-30 11:35:16 +01:00
Norman Maurer
eb50e5148a
Initialize ThreadLocalRandom at runtime to improve GraalVM support (#9977)
Motivation:

We need to initialize ThreadLocalRandom at runtime as it uses System.nanoTime() in a static block to init the seed.

Modifications:

Add io.netty.util.internal.ThreadLocalRandom to properties file

Result:

Better support for GraalVM
2020-01-29 15:00:12 +01:00
Iván López
066a180a43 Initialize some classes at runtime to improve GraalVM support (#9963)
Motivation:

Deploying a Micronaut application as GraalVM native image to AWS Lambda with custom runtime fails when using Micronaut Http Client.

This PR initializes at runtime some classes needed to fix the issue. There is more information in our original issue in Micronaut https://github.com/micronaut-projects/micronaut-core/issues/2335#issuecomment-570151944

At this moment I've added those classes into Micronaut (b383d3ab14) as a workaround but this should be included in Netty so it's available for everyone.

Modification:

Mark 3 classes to be initialized at runtime for GraalVM.

Result:

Mark 3 classes to be initialized at runtime for GraalVM.
2020-01-24 06:40:39 -08:00
root
9b1ea10a12 [maven-release-plugin] prepare for next development iteration 2020-01-13 09:13:53 +00:00
root
136db8680a [maven-release-plugin] prepare release netty-4.1.45.Final 2020-01-13 09:13:30 +00:00
时无两丶
b82258b72f Introduce needReport for ResourceLeakDetector. (#9910)
Motivation:

We can extend `ResourceLeakDetector` through `ResourceLeakDetectorFactory`, and then report the leaked information by covering `reportTracedLeak` and `reportUntracedLeak`. However, the behavior of `reportTracedLeak` and `reportUntracedLeak` is controlled by `logger.isErrorEnabled()`, which is not reasonable. In the case of extending `ResourceLeakDetector`, we sometimes need `needReport` to always return true instead of relying on `logger.isErrorEnabled ()`.

Modification:

introduce `needReport` method and let it be `protected`

Result:

We can control the report leak behavior.
2020-01-10 05:21:24 +01:00
Francesco Nigro
bc026ef8ba Faster decodeHexNibble (#9896)
Motivation:

decodeHexNibble can be a lot faster using a lookup table

Modifications:

decodeHexNibble is made faster by using a lookup table

Result:

decodeHexNibble is faster
2019-12-23 21:15:56 +01:00
Ikhun Um
7f241f1f3c Fix typos in javadocs (#9900)
Motivation:

Javadocs should have no typos.

Modifications:

Fix two typos

Result:

Less typos.
2019-12-23 08:34:16 +01:00
Norman Maurer
68cfab472e
Ignore inline comments when parsing nameservers (#9894)
Motivation:

The resolv.conf file may contain inline comments which should be ignored

Modifications:

- Detect if we have a comment after the ipaddress and if so skip it
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/9889
2019-12-18 21:07:23 +01:00
root
79d4e74019 [maven-release-plugin] prepare for next development iteration 2019-12-18 08:32:54 +00:00
root
5ddf45a2d5 [maven-release-plugin] prepare release netty-4.1.44.Final 2019-12-18 08:31:43 +00:00
Robert Mihaly
ed16da2e84 Ensure scheduled tasks are executed before shutdown (#9858)
Motivation:

In #9603 the executor hung on shutdown because of an abandoned task
on another executor the first was waiting for.

Modifications:

This commit modifies the executor shutdown sequence to include
switching to SHUTDOWN state and then running all remaining tasks.
This ensures that no more tasks are scheduled after SHUTDOWN and
the last pass of running remaining tasks will take it all.
Any tasks scheduled after SHUTDOWN will be rejected.

This change preserves the functionality of graceful shutdown with
quiet period and only adds one more pass of task execution after
the default shutdown process has finished and the executor is
ready for termination.

Result:

After this change tasks that succeed to be added to the executor will
be always executed. Tasks which come late will be rejected instead of
abandoned.
2019-12-11 10:48:38 +01:00
时无两丶
0cde4d9cb4 Uniform null pointer check. (#9840)
Motivation:
Uniform null pointer check.

Modifications:

Use ObjectUtil.checkNonNull(...)

Result:
Less code, same result.
2019-12-09 09:47:35 +01:00
Norman Maurer
385a17b103
Include JCTools sources for shaded classes in the sources jar (#9838)
Motivation:

We should include the shaded sources for JCTools in our sources jar to make it easier to debug.

Modifications:

- Adjust plugin configuration to execute plugins in correct order
- Update source plugin
- Add configuration for shade plugin to generate source jar content

Result:

Fixes https://github.com/netty/netty/issues/6640.
2019-12-05 09:10:32 +01:00
时无两丶
df121e5e55 Replace map with set. (#9833)
Motivation:
Replace Map with Set. `reportedLeaks` has better semantics as a Set, and if it is a Map, it seems that the value of this Map has no meaning to us.

Modifications:

Use Set.

Result:
Cleaner code
2019-12-04 13:53:11 +01:00
Nick Hill
e208e96f12 Clean up NioEventLoop (#9799)
Motivation

The event loop implementations had become somewhat tangled over time and
work was done recently to streamline EpollEventLoop. NioEventLoop would
benefit from the same treatment and it is more straighforward now that
we can follow the same structure as was done for epoll.

Modifications

Untangle NioEventLoop logic and mirror what's now done in EpollEventLoop
w.r.t. the volatile selector wake-up guard and scheduled task deadline
handling.

Some common refinements to EpollEventLoop have also been included - to
use constants for the "special" deadline/wakeup volatile values and to
avoid some unnecessary calls to System.nanoTime() on task-only
iterations.

Result

Hopefully cleaner, more efficient and less fragile NIO transport
implementation.
2019-11-26 08:25:59 +01:00
Norman Maurer
804c33ef46
Cleanup Recycler to better encapsulate stuff (#9739)
Motivation:

We can move some methods etc to make encapsulation better in Recycler

Modifications:

Move / rename methods to make usage more clear

Result:

Code cleanup
2019-11-04 17:43:53 +01:00
Nick Hill
feb804dca8 Avoid extra Runnable allocs when scheduling tasks outside event loop (#9744)
Motivation

Currently when future tasks are scheduled via EventExecutors from a
different thread, at least two allocations are performed - the
ScheduledFutureTask wrapping the to-be-run task, and a Runnable wrapping
the action to add to the scheduled task priority queue. The latter can
be avoided by incorporating this logic into the former.

Modification

- When scheduling or cancelling a future task from outside the event
loop, enqueue the task itself rather than wrapping in a Runnable
- Have ScheduledFutureTask#run first verify the task's deadline has
passed and if not add or remove it from the scheduledTaskQueue depending
on its cancellation state
- Add new outside-event-loop benchmarks to ScheduleFutureTaskBenchmark

Result

Fewer allocations when scheduling/cancelling future tasks
2019-11-04 11:57:53 +01:00
Nick Hill
17bffce90e Use interval instead of mask comparison for Recycler ratio (#9748)
Motivation

The recycling ratio is currently implemented by comparing with a masked
count. The mask operation is not free and also not necessary.

Modification

Change the count(s) to just iterate over the corresponding interval,
which requires only a comparison and no mask.

Also make "first time recycle" behaviour consistent and revert change to
RecyclerTest made in #9727.

Result

Less recycling overhead
2019-11-04 11:24:20 +01:00
Norman Maurer
71860e5b94
Enforce ratioMask also for WeakOrderQueue (#9727)
Motivation:

At the moment we only enfore ratioMask for the Stack which means that we only guard against recycle burts when recycled from the same Thread. We should also enforce the ratioMask in the WeakOrderQueue so we also guard against the bursts when recycle from other threads.

Modifications:

- Keep counter in WeakOrderQueue to enforce ratioMask as well
- Adjust unit test

Result:

Better guard against recycle bursts which could pollute the heap unnecessary.
2019-11-01 07:10:42 +01:00
Nick Hill
fb0d34a2f9 Avoid synthetic methods in Recycler (#9736)
Motivation

Currently the visibility of the various Recycler inner classes and their
fields isn't optimal. Some private members are accessed by other classes
resulting in synthetic methods, and other non-private classes/members
are only accessed privately and so can be made private.

Modifications

- Increase/reduce visibility of various fields/methods/classes within
Recycler
- Have WeakOrderQueue extend WeakReference<Thread> to eliminate the
owner field
- Change local DefaultHandle var to DefaultHandle<?> to avoid raw type
compiler warning

Result

Tidier code, fewer implicit methods on hot paths (reducing inlining
depths)
2019-10-31 11:22:05 +01:00
Nick Hill
a18c57ea87 Externalize lazy execution semantic for EventExecutors (#9587)
Motivation

This is already done internally for various reasons but it would make
sense i.m.o. as a top level concept: submitting a task to be run on the
event loop which doesn't need to run immediately but must still be
executed in FIFO order relative all other submitted tasks (be those
"lazy" or otherwise).

It's nice to separate this abstract "relaxed" semantic from concrete
implementations - the simplest is to just delegate to existing execute,
but for the main EL impls translates to whether a wakeup is required
after enqueuing.

Having a "global" abstraction also allows for simplification of our
internal use - for example encapsulating more of the common scheduled
future logic within AbstractScheduledEventExecutor.

Modifications

- Introduce public LazyRunnable interface and
AbstractEventExecutor#lazyExecute method (would be nice for this to be
added to EventExecutor interface in netty 5)
- Tweak existing SingleThreadEventExecutor mechanics to support these
- Replace internal use of NonWakeupRunnable (such as for pre-flush
channel writes)
- Uplift scheduling-related hooks into AbstractScheduledEventExecutor,
eliminating intermediate executeScheduledRunnable method

Result

Simpler code, cleaner and more useful/flexible abstractions - cleaner in
that they fully communicate the intent in a more general way, without
implying/exposing/restricting implementation details
2019-10-31 10:01:53 +01:00
Norman Maurer
1d57241565
Remove usage of finalizer in Recycler (#9726)
Motivation:

We currently use a finalizer to ensure we correctly return the reserved back to the Stack but this is not really needed as we can ensure we return it when needed before dropping the WeakOrderQueue

Modifications:

Use explicit method call to ensure we return the reserved space back before dropping the object

Result:

Less finalizer usage and so less work for the GC
2019-10-31 09:58:23 +01:00
Norman Maurer
7c85c9ea0b
Correctly update size of the Stack before doing any validation in Recycler (#9731)
Motivation:

We null out the element in the array after we decrement the current size of the Stack but not directly write back the updated size to the stored field. This is problematic as we do some validation before we write it back and so may never do so if the validation fails. This then later can lead to have null objects returned where not expected

Modifications:

Update size directly after null out object

Result:

No more unexpected null value possible
2019-10-31 09:52:07 +01:00
Bennett Lynch
9976ab7fe8 Prefer Log4J2 over Log4J1 for default InternalLoggerFactory (#9734)
##Motivation

The InternalLoggerFactory attempts to instantiate different logger
implementations to discover what is available on the class path,
accepting the first implementation that does not throw an exception.

Currently, the default ordering will attempt to instantiate a Log4j1
logger before Log4j2. For environments where both Log4j1 and Log4j2 are
available, this will result in using the older version. It seems that it
would be more intuitive to prefer the newer version, when possible.

##Modifications

Change the default ordering to attempt to use the Log4J2LoggerFactory
before the Log4JLoggerFactory.

##Result

For environments where both Log4j1 and Log4j2 are available on the class
path (but Slf4J is not available), Netty will now use Log4j2 instead of
Log4j1.
2019-10-30 19:35:28 +01:00
Norman Maurer
ff2a792923
Don't pollute FastThreadLocal for Threads with WeakHashMap if maxDelayedQueues == 0 (#9722)
Motivation:

If maxDelayedQueues == 0 we should never put any WeakHashMap into the FastThreadLocal for a Thread.

Modifications:

Check if maxDelayedQueues == 0 and if so return directly. This will ensure we never call FastThreadLocal.initialValue() in this case

Result:

Less overhead / memory usage when maxDelayedQueues == 0
2019-10-28 18:35:45 +01:00
Norman Maurer
6c061abc49
Hide Recycler implemention to allow experimenting with different implementions of an Object pool (#9715)
Motivation:

At the moment we directly extend the Recycler base class in our code which makes it hard to experiment with different Object pool implementation. It would be nice to be able to switch from one to another by using a system property in the future. This would also allow to more easily test things like https://github.com/netty/netty/pull/8052.

Modifications:

- Introduce ObjectPool class with static method that we now use internally to obtain an ObjectPool implementation.
- Wrap the Recycler into an ObjectPool and return it for now

Result:

Preparation for different ObjectPool implementations
2019-10-26 00:34:30 -07:00
Norman Maurer
efce8e5363
Guard against busy spinning in HashedWheelTimer when using windows and a tickDuration of 1 (#9714)
Motivation:

We do not correct guard against the gact that when applying our workaround for windows we may end up with a 0 sleep period. In this case we should just sleep for 1 ms.

Modifications:

Guard agains the case when our calculation will produce 0 as sleep time on windows

Result:

Fixes https://github.com/netty/netty/issues/9710.
2019-10-25 11:14:06 -07:00
Sergei Egorov
f4b536edcb Add BlockHound integration that detects blocking calls in event loops (#9687)
Motivation:

Netty is an asynchronous framework.
If somebody uses a blocking call inside Netty's event loops,
it may lead to a severe performance degradation.
BlockHound is a tool that helps detecting such calls.

Modifications:

This change adds a BlockHound's SPI integration that marks
threads created by Netty (`FastThreadLocalThread`s) as non-blocking.
It also marks some of Netty's internal methods as whitelisted
as they are required to run the event loops.

Result:

When BlockHound is installed, any blocking call inside event loops
is intercepted and reported (by default an error will be thrown).
2019-10-25 06:14:14 -07:00
root
844b82b986 [maven-release-plugin] prepare for next development iteration 2019-10-24 12:57:00 +00:00
root
d066f163d7 [maven-release-plugin] prepare release netty-4.1.43.Final 2019-10-24 12:56:30 +00:00
Nick Hill
19b4adf79c Avoid wrapping scheduled Runnables in Callable adapter (#9666)
Motivation

Currently when future tasks are scheduled via schedule(Runnable, ...)
methods, the supplied Runnable is wrapped in a newly allocated Callable
adapter prior to being wrapped in a ScheduledFutureTask.

This can be avoided which saves an object allocation per scheduled task.

Modifications

Change the Callable task field of ScheduledFutureTask to be of type
Object so that it can hold/run Runnables directly in addition to
Callables.

An "adapter" is still used in the case a Runnable is scheduled with an
explicit constant non-null completion value, assumed to be rare.

Result

Less garbage
2019-10-17 07:01:52 -07:00
Norman Maurer
6c05d16967
Use @SuppressJava6Requirement for animal sniffer plugin to ensure we always guard correctly (#9655)
Motivation:

We can use the `@SuppressJava6Requirement` annotation to be more precise about when we use Java6+ APIs. This helps us to ensure we always protect these places.

Modifications:

Make use of `@SuppressJava6Requirement` explicit

Result:

Fixes https://github.com/netty/netty/issues/2509.
2019-10-14 15:54:49 +02:00
Nick Hill
170e4deee6 Fix event loop shutdown timing fragility (#9616)
Motivation

The current event loop shutdown logic is quite fragile and in the
epoll/NIO cases relies on the default 1 second wait/select timeout that
applies when there are no scheduled tasks. Without this default timeout
the shutdown would hang indefinitely.

The timeout only takes effect in this case because queued scheduled
tasks are first cancelled in
SingleThreadEventExecutor#confirmShutdown(), but I _think_ even this
isn't robust, since the main task queue is subsequently serviced which
could result in some new scheduled task being queued with much later
deadline.

It also means shutdowns are unnecessarily delayed by up to 1 second.

Modifications

- Add/extend unit tests to expose the issue
- Adjust SingleThreadEventExecutor shutdown and confirmShutdown methods
to explicitly add no-op tasks to the taskQueue so that the subsequent
event loop iteration doesn't enter blocking wait (as looks like was
originally intended)

Results

Faster and more robust shutdown of event loops, allows removal of the
default wait timeout
2019-10-07 11:06:01 +04:00
Nick Hill
85a663fa52 Null out completed tasks to help with garbage collection (#9613)
Motivation

When ScheduledFutureTasks complete, there's no need to retain a ref to
the wrapped task. Clearing it could help in particular with the case
where many scheduled tasks have been cancelled but their queue removal
delayed (since it is done lazily).

Modifications

This comprises just the PromiseTask changes from #9580. Upon completion,
replace the task reference with a static sentinel depending on the type
of completion (so that it will be reflected by toString).

Result

More expedient collection of cancelled task objects
2019-09-27 09:54:42 +02:00
时无两丶
3ef00eaa06 Double check size to avoid ArrayIndexOutOfBoundsException (#9609)
Motivation:

Recycler$Stack.pop will occurs `ArrayIndexOutOfBoundsException` in some race cases, we should double check `size` even after `scavenge` called.

Modifications:

Double check `size` after `scavenge`

Result:

avoid ArrayIndexOutOfBoundsException in `pop`
2019-09-26 21:53:35 +02:00
root
92941cdcac [maven-release-plugin] prepare for next development iteration 2019-09-25 06:15:31 +00:00
root
bd907c3b3a [maven-release-plugin] prepare release netty-4.1.42.Final 2019-09-25 06:14:31 +00:00
Nick Hill
2791f0fefa Avoid use of global AtomicLong for ScheduledFutureTask ids (#9599)
Motivation

Currently a static AtomicLong is used to allocate a unique id whenever a
task is scheduled to any event loop. This could be a source of
contention if delayed tasks are scheduled at a high frequency and can be
easily avoided by having a non-volatile id counter per queue.

Modifications

- Replace static AtomicLong ScheduledFutureTask#nextTaskId with a long
field in AbstractScheduledExecutorService
- Set ScheduledFutureTask#id based on this when adding the task to the
queue (in event loop) instead of at construction time
- Add simple benchmark

Result

Less contention / cache-miss possibility when scheduling future tasks

Before:

Benchmark      (num)   Mode  Cnt    Score    Error  Units
scheduleLots  100000  thrpt   20  346.008 ± 21.931  ops/s

Benchmark      (num)   Mode  Cnt    Score    Error  Units
scheduleLots  100000  thrpt   20  654.824 ± 22.064  ops/s
2019-09-25 07:34:25 +02:00
wyzhang
338e1a991c Fix a bug introduced by 79706357c7 which can cause thread to spin in an infinite loop. (#9579)
Motivation:
peek() is implemented in a similar way to poll() for the mpsc queue, thus it is more like a consumer call.
It is possible that we could have multiple thread call peek() and possibly one thread calls poll() at at the same time.
This lead to multiple consumer scenario, which violates the multiple producer single consumer condition and could lead to spin in an infinite loop in peek()

Modification:
Use isEmpty() instead of peek() to check if task queue is empty

Result:
Dont violate the mpsc semantics.
2019-09-19 11:59:51 +02:00
Norman Maurer
2fe2a15593
No need to explicit use the AccessController when SystemPropertyUtil is used (#9577)
Motivation:

SystemPropertyUtil already uses the AccessController internally so not need to wrap its usage with AccessController as well.

Modifications:

Remove explicit AccessController usage when SystemPropertyUtil is used.

Result:

Code cleanup
2019-09-19 08:41:27 +02:00
root
01d805bb76 [maven-release-plugin] prepare for next development iteration 2019-09-12 16:09:55 +00:00
root
7cf69022d4 [maven-release-plugin] prepare release netty-4.1.41.Final 2019-09-12 16:09:00 +00:00
root
aef47bec7f [maven-release-plugin] prepare for next development iteration 2019-09-12 05:38:11 +00:00
root
267e5da481 [maven-release-plugin] prepare release netty-4.1.40.Final 2019-09-12 05:37:30 +00:00
Andrey Mizurov
bcb0d02248 Fix HttpContentEncoder does not handle multiple Accept-Encoding (#9557)
Motivation:
At the current moment HttpContentEncoder handle only first value of multiple accept-encoding headers.

Modification:

Join multiple accept-encoding headers to one separated by comma.

Result:

Fixes #9553
2019-09-11 08:46:06 +02:00