Commit Graph

936 Commits

Author SHA1 Message Date
Scott Mitchell
1cc4607f07 AppendableCharSequence not to depend upon IndexOutOfBoundsException for resize
Motivation:
AppendableCharSequence depends upon IndexOutOfBoundsException to trigger a resize operation under the assumption that the resize operation will be rare if the initial size guess is good. However if the initial size guess is not good then the performance will be more unpredictable and likely suffer.

Modifications:
- Check the position in AppendableCharSequence#append to determine if a resize is necessary

Result:
More predictable performance in AppendableCharSequence#append.
2017-06-12 12:42:20 -07:00
Norman Maurer
fd67a2354d [maven-release-plugin] prepare for next development iteration 2017-06-08 21:06:24 +02:00
Norman Maurer
3acd5c68ea [maven-release-plugin] prepare release netty-4.1.12.Final 2017-06-08 21:06:01 +02:00
Norman Maurer
f208b147a6 Use FQCN to prevent classloader issues on java6
Motivation:

We need to use FQCN to prevent classloader issues for classes that are > Java6. This is a cleanup of ed5fcbb773.

Modifications:

Just remove the imports and use FQCN.

Result:

No classloader issues with java6
2017-06-08 12:04:17 +02:00
Michael K. Werle
ed5fcbb773 Add explicit message when noexec prevents library loading.
Motivation:

Docker's `--tmpfs` flag mounts the temp volume with `noexec` by default,
resulting in an UnsatisfiedLinkError.  While this is good security
practice, it is a surprising failure from a seemingly innocuous flag.

Modifications:

Add a best-effort attempt in `NativeLibraryLoader` to detect when temp
files beng loaded cannot be executed even when execution permissions
are set, often because the `noexec` flag is set on the volume.

Requires numerous additional exclusions to the Animal Sniffer config
for Java7 POSIX permissions manipulation.

Result:

Fixes [#6678].
2017-06-07 09:20:05 -07:00
Norman Maurer
201d9b6536 Share code that is needed to support shaded native libraries.
Motivation:

For our native libraries in netty we support shading, to have this work on runtime the user needs to set a system property. This code should shared.

Modifications:

Move logic to NativeLbiraryLoader and so share for all native libs.

Result:

Less code duplication and also will work for netty-tcnative out of the box once it support shading
2017-05-19 19:33:21 +02:00
Nikolay Fedorovskikh
d768c5e628 MessageFormatter improvements
Motivation:

`FormattingTuple.getArgArray()` is never used.
In the `MessageFormatter` it is possible to make
some improvements, e.g. replace `StringBuffer`
with `StringBuilder`, avoid redundant allocations, etc.

Modifications:

- Remove `argArray` field from the `FormattingTuple`.
- In `MessageFormatter`:
  - replace `StringBuffer` with `StringBuilder`,
  - replace `HashMap` with `HashSet` and make it lazy initialized.
  - avoid redundant allocations (`substring()`, etc.)
  - use appropriate StringBuilder's methods for the some `Number` values.
- Porting unit tests from `slf4j`.

Result:

Less GC load on logging with internal `MessageFormatter`.
2017-05-19 09:47:11 -07:00
Nikolay Fedorovskikh
e4531918a3 Optimizations in NetUtil
Motivation:

IPv4/6 validation methods use allocations, which can be avoided.
IPv4 parse method use StringTokenizer.

Modifications:

Rewriting IPv4/6 validation methods to avoid allocations.
Rewriting IPv4 parse method without use StringTokenizer.

Result:

IPv4/6 validation and IPv4 parsing faster up to 2-10x.
2017-05-18 16:42:22 -07:00
Norman Maurer
0ee49e6d66 Eliminate noisy logging when using sun.misc.Unsafe and running on pre Java9
Motivation:

We should only try to load jdk.internal.misc.Unsafe if we run on Java9+ to eliminate noise in the log.

Modifications:

- Move javaVersion() and related methods to PlatformDependent0 to be able to use these in the static initializer without creating a cycle.
- Only try to load jdk.internal.misc.Unsafe when running in Java9+

Result:

Less noise in the log when running pre java9.
2017-05-16 08:29:06 +02:00
Jason Tedor
d88cd23bfc Trim thread local string builder if large
Motivation:

A previous change allocated a new thread local string builder if it
was getting too large. This is a good change, these string builders
can accidentally get too large and then never shrunk and that is sort
of a memory leak. However, the change allocates an entirely new string
builder which is more allocations than necessary. Instead, we can trim
the string builder if its too large, this only allocates an extra
backing array instead of a whole new object.

Modifications:

If the string builder is above a threshold, we trim the string builder
and then ensure its capacity is reasonable to we do not allocate too
much as we start using the string builder.

Result:

The thread local string builder do not serve as a memory yet we do not
allocate too many new objects.
2017-05-12 08:20:04 +02:00
Norman Maurer
0db2901f4d [maven-release-plugin] prepare for next development iteration 2017-05-11 16:00:55 +02:00
Norman Maurer
f7a19d330c [maven-release-plugin] prepare release netty-4.1.11.Final 2017-05-11 16:00:16 +02:00
Nikolay Fedorovskikh
5643cc6a10 IPv6 validation fixes
Motivation:

`NetUtil`'s methods `isValidIpV6Address` and `getIPv6ByName` incorrectly validate some IPv6 addresses.

Modifications:

- `getIPv6ByName`: add checks for single colon at the start or end.
- `isValidIpV6Address`: fix checks for the count of colons and use `endOffset` instead of `ipAddress.length()` for the cases with the brackets or '%'.

Result:

More correct implementation of `NetUtil#isValidIpV6Address` and `NetUtil#getIPv6ByName`.
2017-05-11 08:10:25 -07:00
jiachun.fjc
cd80b6c2d8 Use simple volatile read for SingleThreadEventExecutor#state instead of UNSAFE(AtomicIntegerFieldUpdater#get), CAS operation still to use AtomicIntegerFieldUpdater
Motivation:

AtomicIntegerFieldUpdater#get is unnecessary, I think use simple volatile read is cleaner

Modifications:

Replace code STATE_UPDATER.get(this) to state in SingleThreadEventExecutor

Result:

Cleaner code
2017-05-08 19:36:19 +02:00
jiachun.fjc
963cd22a05 InternalThreadLocalMap#stringBuilder: ensure memory overhead
Motivation:

InternalThreadLocalMap#stringBuilder: ensure memory overhead

Modification:

If the capacity of StringBuilder is greater than 65536 then release it on the next time you get StringBuilder and re-create a StringBuilder.

Result:

Possible less memory usage.
2017-05-05 09:28:51 -07:00
Jason Tedor
02a2738cd2 Do not try to use cleaner if no unsafe
Motivation:

If unsafe is unavailable, we can not use the cleaner anyway. If we try
to set it up, we get an annoying log message about unsafe being
unavailable (when debug logging is enabled). We know this will fail, so
we should not even bother and avoid the log message.

Modifications:

This commit adds a guard against setting up the cleaner if it is not
going to be available because unsafe is unavailable.

Result:

We do not try to set up the cleaner if unsafe is unavailable, and we do
not get an annoying log message.
2017-05-03 13:36:00 -07:00
Jason Tedor
9a0fd3a7b8 Do not log on explicit no unsafe again
Motivation:

Users should not see a scary log message when Netty is initialized if
Netty configuration explicitly disables unsafe. The log message that
produces this warning was previously guarded but the guard was
lost.

Modifications:

This commit brings back the guard against the scary log message if
unsafe is explicitly disabled.

Result:

No log message is produced when unsafe is unavailable because Netty was
told to not look for it.
2017-05-03 13:29:58 -07:00
Scott Mitchell
3cc4052963 New native transport for kqueue
Motivation:
We currently don't have a native transport which supports kqueue https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2. This can be useful for BSD systems such as MacOS to take advantage of native features, and provide feature parity with the Linux native transport.

Modifications:
- Make a new transport-native-unix-common module with all the java classes and JNI code for generic unix items. This module will build a static library for each unix platform, and included in the dynamic libraries used for JNI (e.g. transport-native-epoll, and eventually kqueue).
- Make a new transport-native-unix-common-tests module where the tests for the transport-native-unix-common module will live. This is so each unix platform can inherit from these test and ensure they pass.
- Add a new transport-native-kqueue module which uses JNI to directly interact with kqueue

Result:
JNI support for kqueue.
Fixes https://github.com/netty/netty/issues/2448
Fixes https://github.com/netty/netty/issues/4231
2017-05-03 09:53:22 -07:00
jiachun.fjc
e58095c4f2 Simplify code
Motivation:

Code can be simplified

Modification:

Refactor code to remove extra branching

Result:

Cleaner code.
2017-05-02 15:16:19 -07:00
Vladimir Kostyukov
ed37cf20ef Introduce HashedWheelTimer.pendingTimeouts()
Motivation:

Fixes #6681.

Modification:

For the sake of better timer observability, expose the number of pending timeouts through the new HashedWheelTimer.pendingTimeouts method .

Result:

It's now ridiculously easy to observe Netty timer's very basic and yet important metric, the number of pending tasks/timeouts.
2017-05-01 20:10:22 -07:00
Norman Maurer
6915ec3bb9 [maven-release-plugin] prepare for next development iteration 2017-04-29 14:10:00 +02:00
Norman Maurer
f30f242fee [maven-release-plugin] prepare release netty-4.1.10.Final 2017-04-29 14:09:32 +02:00
Scott Mitchell
ea1cb20c90 Netutil IPv6 bugs
Motivation:
NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName allowed an invalid form of mapped IPv4 addresses which lead to accepting invalid IPv6 addresses as valid.

Modifications:
- NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName should only allow 7 colons for an IPv4 address if they are the first 2 characters.

Result:
More correct implementation of NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName
2017-04-28 07:29:36 +02:00
Scott Mitchell
9cb858fcf6 NetUtil IPv6 bugs related to IPv4 and compression
Motivation:
NetUtil#getByName and NetUtil#isValidIpV6Address do not strictly enforce the format of IPv4 addresses that are allowed to be embedded in IPv6 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5. This may lead to invalid addresses being parsed, or invalid addresses being considered valid. Compression of a single IPv6 word was also not handled correctly if there are 7 : characters.

Modifications:
- NetUtil#isValidIpV6Address should enforce the IPv4-Compatible and IPv4-Mapped are the only valid formats for including IPv4 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5
- NetUtil#getByName should more stritcly parse IPv6 addresses which contain IPv4 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5
- NetUtil should allow compression even if the number of : characters is 7.
- NetUtil#createByteArrayFromIpAddressString should use the same IP string to byte[] translation which is used in NetUtil#getByName

Result:
NetUtil#getByName and NetUtil#isValidIpV6Address respect the IPv6 RFC which defines the valid formats for embedding IPv4 addresses.
2017-04-25 15:10:38 -07:00
Jason Tedor
98beb777f8 Enable configuring available processors
Motivation:

In cases when an application is running in a container or is otherwise
constrained to the number of processors that it is using, the JVM
invocation Runtime#availableProcessors will not return the constrained
value but rather the number of processors available to the virtual
machine. Netty uses this number in sizing various resources.
Additionally, some applications will constrain the number of threads
that they are using independenly of the number of processors available
on the system. Thus, applications should have a way to globally
configure the number of processors.

Modifications:

Rather than invoking Runtime#availableProcessors, Netty should rely on a
method that enables configuration when the JVM is started or by the
application. This commit exposes a new class NettyRuntime for enabling
such configuraiton. This value can only be set once. Its default value
is Runtime#availableProcessors so that there is no visible change to
existing applications, but enables configuring either a system property
or configuring during application startup (e.g., based on settings used
to configure the application).

Additionally, we introduce the usage of forbidden-apis to prevent future
uses of Runtime#availableProcessors from creeping. Future work should
enable the bundled signatures and clean up uses of deprecated and
other forbidden methods.

Result:

Netty can be configured to not use the underlying number of processors,
but rather the constrained number of processors.
2017-04-23 10:31:17 +02:00
Nikolay Fedorovskikh
0692bf1b6a fix the typos 2017-04-20 04:56:09 +02:00
Norman Maurer
e482d933f7 Add 'io.netty.tryAllocateUninitializedArray' system property which allows to allocate byte[] without memset in Java9+
Motivation:

Java9 added a new method to Unsafe which allows to allocate a byte[] without memset it. This can have a massive impact in allocation times when the byte[] is big. This change allows to enable this when using Java9 with the io.netty.tryAllocateUninitializedArray property when running Java9+. Please note that you will need to open up the jdk.internal.misc package via '--add-opens java.base/jdk.internal.misc=ALL-UNNAMED' as well.

Modifications:

Allow to allocate byte[] without memset on Java9+

Result:

Better performance when allocate big heap buffers and using java9.
2017-04-19 11:45:39 +02:00
Norman Maurer
1b0b8f80cd AbstractScheduledEventExecutor.schedule(...) must accept delay <= 0.
Motivation:

As the javadoc of ScheduledExecutorService state:

Zero and negative delays (but not periods) are also allowed in schedule methods,and are treated as requests for immediate execution.

Modifications:

- Correctly handle delay <= 0.
- Add unit tests.

Result:

Fixes [#6627].
2017-04-19 11:35:50 +02:00
Lukasz Strzalkowski
7bd0905969 Introduce ReferenceCounted.refCnt()
Motivation:

When debugging netty memory leaks, it's sometimes helpful to
print the object's reference count.

Modifications:

Add `refCnt` methods to set of already exitsting helpers for ref coutned
objects.

Result:

Users will have utility to print object's ref count without much of a
boilerplate.
2017-04-17 19:43:44 +02:00
Norman Maurer
7b6119a0a4 Allow to free direct buffers on java9 again
Motivation:

Java9 adds a new method to Unsafe which allows to free direct ByteBuffer via the cleaner without the need to use an commandline arguments.

Modifications:

- Add Cleaner interface
- Add CleanerJava9 which will be used when using Java9+ and take care of release direct ByteBuffer
- Let Cleaner0 implement Cleaner

Result:

Be able to free direct ByteBuffer on Java9+ again without any commandline arguments.
2017-04-17 19:40:52 +02:00
Norman Maurer
493a8135f8 Ensure test introduced in 5c1c14286d also works on Java9 2017-03-29 22:43:00 +02:00
Norman Maurer
5c1c14286d Allow negative memoryAddress when calling PlatformDependent0.newDirectBuffer(...)
Motivation:

When UNSAFE.allocateMemory is returning an address whose high bit is set we currently throw an IllegalArgumentException. This is not correct as it may return a negative number on at least sparc.

Modifications:

- Allow to pass in negative memoryAddress
- Add unit tests

Result:

Correctly validate the memoryAddress and so also work on sparc as expected. Fixes [#6574].
2017-03-29 22:33:34 +02:00
David Dossot
9c1a191696 Trim optional white space in CombinedHttpHeaders values
Motivation:

The updated HTTP/1.x RFC allows for header values to be CSV and separated by OWS [1]. CombinedHttpHeaders should remove this OWS on insertion.

[1] https://tools.ietf.org/html/rfc7230#section-7

Modification:

CombinedHttpHeaders doesn't account for the OWS and returns it back to the user as part of the value.

Result:

Fixes #6452
2017-03-19 08:17:29 -07:00
Norman Maurer
9e6e1a3e7b Use SystemPropertyUtil to access system properties
Motivation:

We should use SystemPropertyUtil to access system properties and so always handle SecurityExceptions.

Modifications:

Use SystemPropertyUtil everywhere.

Result:

Better and consist handling of SecurityException.
2017-03-19 08:09:29 -07:00
Norman Maurer
2b8c8e0805 [maven-release-plugin] prepare for next development iteration 2017-03-10 07:46:17 +01:00
Norman Maurer
1db58ea980 [maven-release-plugin] prepare release netty-4.1.9.Final 2017-03-10 07:45:28 +01:00
Norman Maurer
e12f504ac1 Remove deprecated usage of Mockito methods
Motivation:

We used some deprecated Mockito methods.

Modifications:

- Replace deprecated method usage
- Some cleanup

Result:

No more usage of deprecated Mockito methods. Fixes [#6482].
2017-03-09 20:59:54 +01:00
Norman Maurer
9ade81ab5b Use system property to detect if root is running the program
Motivation:

We forked a new process to detect if the program is run by root. We should better just use user.name system property

Modifications:

- Change PlatformDependent.isRoot0() to read the user.name system property to detect if root runs the program and rename it to maybeSuperUser0().
- Rename PlatformDependent.isRoot() to maybeSuperUser() and let it init directly in the static block

Result:

Less heavy way to detect if the program is run by root.
2017-03-09 11:16:10 +01:00
Norman Maurer
c6a3cae269 UnorderedThreadPoolEventExecutor consumes 100% CPU when idle
Motivation:

When UnorderedThreadPoolEventExecutor.execute / submit etc is called it will consume up to 100 % CPU even after the task was executed.

Modifications:

Add a special wrapper which we will be used in execute(...) to wrap the submitted Runnable. This is needed as  ScheduledThreadPoolExecutor.execute(...) will delegate to submit(...) which will then use decorateTask(...). The problem with this is that decorateTask(...) needs to ensure we only do our own decoration if we not call from execute(...) as otherwise we may end up creating an endless loop because DefaultPromise will call  EventExecutor.execute(...) when notify the listeners of the promise.

Result:

Fixes [#6507].
2017-03-09 11:12:42 +01:00
Nikolay Fedorovskikh
2993760e92 Fix misordered 'assertEquals' arguments in tests
Motivation:

Wrong argument order in some 'assertEquals' applying.

Modifications:

Flip compared arguments.

Result:

Correct `assertEquals` usage.
2017-03-08 22:48:37 -08:00
Scott Mitchell
8b21cd9e35 PlatformDependent0 should enforce array index scale for byte[] explicitly
Motivation:
PlatformDependent0 makes assumptions that the array index scale for byte[] is always 1. If this is not the case the results from methods which make this assumption will be undefined.

Modifications:
- PlatformDependent0 should check if unsafe.arrayIndexScale(byte[].class) is not 1, and if so not use unsafe

Result:
Assumptions made by optimizations in PlatformDependent0 which use byte[] are explicitly enforced.
2017-03-08 10:02:37 -08:00
Norman Maurer
1e5d33f8d5 Remove unused code
Motivation:

Cleanup PlatformDependent* and remove unused code.

Modifications:

Code cleanup

Result:

Removed unused code
2017-03-07 21:33:41 +01:00
Norman Maurer
1392bc351f Correctly build socketaddress string, followup of 8b2badf44f 2017-03-01 20:05:20 +01:00
Norman Maurer
0514b0c61b Only add port to HOST header value if needed
Motivation:

We only need to add the port to the HOST header value if its not a standard port.

Modifications:

- Only add port if needed.
- Fix parsing of ipv6 address which is enclosed by [].

Result:

Fixes [#6426].
2017-03-01 19:08:19 +01:00
Nikolay Fedorovskikh
943f4ec7ff Make methods 'static' where it missed
Motivation:

Calling a static method is faster then dynamic

Modifications:

Add 'static' keyword for methods where it missed

Result:

A bit faster method calls
2017-02-23 11:01:57 +01:00
Norman Maurer
7d08b4fc35 Remove optional dependency on javassist
Motivation:

We shipped a javassist based implementation for typematching and logged a confusing debug message about missing javassist. We never were able to prove it really gives any perf improvements so we should just remove it.

Modifications:

- Remove javassist dependency and impl
- Fix possible classloader deadlock as reported by intellij

Result:

Less code to maintain and less confusing log message.
2017-02-23 07:54:42 +01:00
Nikolay Fedorovskikh
0623c6c533 Fix javadoc issues
Motivation:

Invalid javadoc in project

Modifications:

Fix it

Result:

More correct javadoc
2017-02-22 07:31:07 +01:00
Nikolay Fedorovskikh
634a8afa53 Fix some warnings at generics usage
Motivation:

Existing warnings from java compiler

Modifications:

Add/fix type parameters

Result:

Less warnings
2017-02-22 07:29:59 +01:00
Norman Maurer
67be7c5b9f Log why it was not possible to use ByteBuffer.cleaner
Motivation:

We should log why we can not use ByteBuffer.cleaner and so maybe allow the user to fix it.

Modifications:

- Use Unsafe to access the field
- Log the exception when we can not use ByteBuffer.cleaner

Result:

Easier to debug why using cleaner is not possible.
2017-02-17 07:34:34 +01:00
Norman Maurer
fbf0e5f4dd Prefer JDK ThreadLocalRandom implementation over ours.
Motivation:

We have our own ThreadLocalRandom implementation to support older JDKs . That said we should prefer the JDK provided when running on JDK >= 7

Modification:

Using ThreadLocalRandom implementation of the JDK when possible.

Result:

Make use of JDK implementations when possible.
2017-02-16 15:44:00 -08:00
Scott Mitchell
4d7d478a3d Update JCTools to 2.0.1 2017-02-16 15:09:35 -08:00
Norman Maurer
6ac5f35077 Use Unsafe to read ByteBuffer.address field to make it work on Java9 as well.
Motivation:

Java9 does not allow changing access level via reflection by default. This lead to the situation that netty disabled Unsafe completely as ByteBuffer.address could not be read.

Modification:

Use Unsafe to read the address field as this works on all Java versions.

Result:

Again be able to use Unsafe optimisations when using Netty with Java9
2017-02-16 20:40:59 +01:00
Norman Maurer
1843b31885 Guard against having NetworkInterface.getNetworkInterfaces() return null
Motivation:

NetworkInterface.getNetworkInterfaces() may return null if no network interfaces are found. We should guard against it.

Modifications:

Check for null return value.

Result:

Fixes [#6384]
2017-02-16 07:59:31 +01:00
Jason Tedor
d59b4840c1 Cleanup ChannelId handling of basic methods
Motiviation:

Simplify implementation of compareTo/equals/hashCode for ChannelIds.

Modifications:

We simplfy the hashCode implementation for DefaultChannelId by not
making it random, but making it based on the underlying data. We fix the
compareTo implementation for DefaultChannelId by using lexicographic
comparison of the underlying data array. We fix the compareTo
implementation for CustomChannelId to avoid the possibility of overflow.

Result:

Cleaner code that is easier to maintain.
2017-02-15 11:53:36 -08:00
Norman Maurer
84188395be Remove backports of JDK8 classes
Motivation:

Java8 is out now for some time and JDK7 is no longer supported officially. We should remove all our backports and just use what the JDK provides us. This also will allow us to use intrinsics that are offered by the JDK implementations.

Modifications:

Remove all backports of jdk8 classes.

Result:

Use what the JDK offers us. This also fixes [#5458]
2017-02-15 20:44:23 +01:00
Dmitriy Dumanskiy
506f0d8f8c Cleanup : String.length() == 0 replaced with String.isEmpty, removed unnecessary assert, class cast 2017-02-14 15:36:42 +01:00
Norman Maurer
90bc605477 Initialization of PlatformDependent0 fails on Java 9
Motivation:

Initialization of PlatformDependent0 fails on Java 9 in static initializer when calling setAccessible(true).

Modifications:

Add RefelectionUtil which can be used to safely try if setAccessible(true) can be used or not and if not fail back to non reflection.

Result:

Fixed [#6345]
2017-02-14 10:15:27 +01:00
fenik17
0cf3f54a8d Adding 'final' keyword for private fields where possible
Motivation

Missing 'final' keyword for fields

Modifications

Add 'final' for fields where possible

Result

More safe and consistent code
2017-02-14 08:29:15 +01:00
Scott Mitchell
a1b5b5dcca EpollRecvByteAllocatorHandle doesn't inform delegate of more data
Motivation:
EpollRecvByteAllocatorHandle intends to override the meaning of "maybe more data to read" which is a concept also used in all existing implementations of RecvByteBufAllocator$Handle but the interface doesn't support overriding. Because the interfaces lack the ability to propagate this computation EpollRecvByteAllocatorHandle attempts to implement a heuristic on top of the delegate which may lead to reading when we shouldn't or not reading data.

Modifications:
- Create a new interface ExtendedRecvByteBufAllocator and ExtendedHandle which allows the "maybe more data to read" between interfaces
- Deprecate RecvByteBufAllocator and change all existing implementations to extend ExtendedRecvByteBufAllocator
- transport-native-epoll should require ExtendedRecvByteBufAllocator so the "maybe more data to read" can be propagated to the ExtendedHandle

Result:
Fixes https://github.com/netty/netty/issues/6303.
2017-02-13 17:42:24 -08:00
Scott Mitchell
54c9ecf682 DnsNameResolver should respect /etc/resolv.conf and /etc/resolver
Motivation:
The JDK uses gethostbyname for blocking hostname resoltuion. gethostbyname can be configured on Unix systems according to [1][2]. This may impact the name server that is used to resolve particular domains or just override the default fall-back resolver. DnsNameResolver currently ignores these configuration files which means the default resolution behavior is different than the JDK. This may lead to unexpected resolution failures which succeed when using the JDK's resolver.

Modifications:
- Add an interface which can override what DnsServerAddressStream to use for a given hostname
- Provide a Unix specific implementation of this interface and implement [1][2]. Some elements may be ignored sortlist, timeout, etc...

Result:
DnsNameResolver behaves more like the JDK resolver by default.

[1] https://linux.die.net/man/5/resolver
[2] https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man5/resolver.5.html
2017-02-13 11:54:09 -08:00
md-5
fe522fb18e Update Groovy to 2.4.8 for Java 9-ea+154 support.
Motivation:

Update of Groovy is needed to compile on recent java9 releases.

Modification:

Update to Groovy 2.4.8

Result:

This change allows Netty to be successfully compiled on more recent Java 9 previews.
2017-02-11 14:09:15 +01:00
Dmitriy Dumanskiy
c95517f759 Cleanup : removed unnecessary 'continue', explicit array creation, unwrapping 2017-02-10 12:25:01 +01:00
Scott Mitchell
14b902fced Deprecate and ignore ResourceLeakDetector's maxActive parameter
Motivation:
ResourceLeakDetector supports a parameter called maxActive. This parameter is used in attempt to limit the amount of objects which are being tracked for leaks at any given time, and generates an error log message if this limit is exceeded. This assumes that there is a relationship between leak sample rate and object lifetime for objects which are already being tracked. This relationship may appear to work in cases were there are a single leak record per object and those leak records live for the lifetime of the application but in general this relationship doesn't exist. The original motivation was to provide a limit for cases such as HashedWheelTimer to limit the number of instances which exist at any given time. This limit is not enforced in all circumstances in HashedWheelTimer (e.g. if the thread is a daemon) and can be implemented outside ResourceLeakDetector.

Modifications:
- Deprecate all methods which interact with maxActive in ResourceLeakDetectorFactory and ResourceLeakDetector
- Remove all logic related to maxActive in ResourceLeakDetector
- HashedWheelTimer implements its own logic to impose a limit and warn users if too many instances exists at any given time.

Result:
Fixes https://github.com/netty/netty/issues/6225.
2017-02-08 19:59:58 -08:00
Norman Maurer
a7c0ff665c Only use Mockito for mocking.
Motivation:

We used various mocking frameworks. We should only use one...

Modifications:

Make usage of mocking framework consistent by only using Mockito.

Result:

Less dependencies and more consistent mocking usage.
2017-02-07 08:47:22 +01:00
Jason Tedor
42c0359820 Do not prefer empty MAC address
Motivation:

When comparing MAC addresses searching for the best MAC address, if
locally-administered address (e.g., from a Docker container) is compared
against an empty MAC address, the empty MAC address will be marked as
preferred. In cases this is the only available MAC address, this leaves
Netty using a random machine ID instead of using a perfectly valid
machine ID from the locally-adminstered address.

Modifications:

This commit modifies the MAC address logic so that the empty MAC address
is not preferred over a locally-administered address. This commit also
simplifies the comparison logic here.

Result:

Empty MAC addresses will not be preferred over locally-administered
addresses thus permitting the default machine ID to be the
locally-adminstered MAC address if it is the only available MAC address.
2017-02-06 12:14:27 -08:00
Scott Mitchell
3482651e0c HTTP/2 Non Active Stream RFC Corrections
Motivation:
codec-http2 couples the dependency tree state with the remainder of the stream state (Http2Stream). This makes implementing constraints where stream state and dependency tree state diverge in the RFC challenging. For example the RFC recommends retaining dependency tree state after a stream transitions to closed [1]. Dependency tree state can be exchanged on streams in IDLE. In practice clients may use stream IDs for the purpose of establishing QoS classes and therefore retaining this dependency tree state can be important to client perceived performance. It is difficult to limit the total amount of state we retain when stream state and dependency tree state is combined.

Modifications:
- Remove dependency tree, priority, and weight related items from public facing Http2Connection and Http2Stream APIs. This information is optional to track and depends on the flow controller implementation.
- Move all dependency tree, priority, and weight related code from DefaultHttp2Connection to WeightedFairQueueByteDistributor. This is currently the only place which cares about priority. We can pull out the dependency tree related code in the future if it is generally useful to expose for other implementations.
- DefaultHttp2Connection should explicitly limit the number of reserved streams now that IDLE streams are no longer created.

Result:
More compliant with the HTTP/2 RFC.
Fixes https://github.com/netty/netty/issues/6206.

[1] https://tools.ietf.org/html/rfc7540#section-5.3.4
2017-02-01 10:34:27 -08:00
Norman Maurer
735d6dd636 [maven-release-plugin] prepare for next development iteration 2017-01-30 15:14:02 +01:00
Norman Maurer
76e22e63f3 [maven-release-plugin] prepare release netty-4.1.8.Final 2017-01-30 15:12:36 +01:00
Jon Chambers
94cb389c04 Restore add(Promise) and addAll(Promise...) methods to PromiseCombiner.
Motivation:

A testing goof in 7c630fe introduced a binary incompatibility when the old Promise-specific `add` and `addAll` methods in PromiseCombiner were generalized to accept `Futures`.

Modification:

- Restore (but mark as `@Deprecated`) old PromiseCombiner methods.
- Fixed a couple minor documentation typos because sure why not.

Result:

`PromiseCombiner` is binary-compatible with previous versions of Netty.
2017-01-30 09:23:11 +01:00
Norman Maurer
a416b79d86 DnsNameResolver.resolve*(...) never notifies the Future when empty hostname is used.
Motivation:

When an empty hostname is used in DnsNameResolver.resolve*(...) it will never notify the future / promise. The root cause is that we not correctly guard against errors of IDN.toASCII(...) which will throw an IllegalArgumentException when it can not parse its input. That said we should also handle an empty hostname the same way as the JDK does and just use "localhost" when this happens.

Modifications:

- If the try to resolve an empty hostname we use localhost
- Correctly guard against errors raised by IDN.toASCII(...) so we will always noify the future / promise
- Add unit test.

Result:

DnsNameResolver.resolve*(...) will always notify the future.
2017-01-24 21:21:49 +01:00
Tim Brooks
3344cd21ac Wrap operations requiring SocketPermission with doPrivileged blocks
Motivation:

Currently Netty does not wrap socket connect, bind, or accept
operations in doPrivileged blocks. Nor does it wrap cases where a dns
lookup might happen.

This prevents an application utilizing the SecurityManager from
isolating SocketPermissions to Netty.

Modifications:

I have introduced a class (SocketUtils) that wraps operations
requiring SocketPermissions in doPrivileged blocks.

Result:

A user of Netty can grant SocketPermissions explicitly to the Netty
jar, without granting it to the rest of their application.
2017-01-19 21:12:52 +01:00
Norman Maurer
71efb0b39e Do not replace System.err during Slf4JLoggerFactory construction
Motivation:

Replacing System.err during Slf4JLoggerFactory construction is problematic as another class may optain the System.err reference before we set it back to the original value.

Modifications:

Remove code that temporary replaced System.err.

Result:

Fixes [#6212].
2017-01-19 19:28:01 +01:00
Dmitriy Dumanskiy
0e654f77e2 Level initialization cleanup. 2017-01-19 10:11:38 -08:00
Dmitriy Dumanskiy
141554f5d1 Removed unnecessary pattern matching during number paring and unnecessary toLowerCase() invocation.
Motivation:

Pattern matching not necessary for number parsing.

Modification:

Removed pattern matching for number parsing and removed unnecessary toLowerCase() operation.

Result:

No static variable with pattern, removed unnecessary matching operation and toLowerCase() operation.
2017-01-19 10:11:55 +01:00
Norman Maurer
4f6ef69582 Follow-up cleanup of 0c4826586f 2017-01-19 08:01:14 +01:00
Norman Maurer
0c4826586f Cleanup PlatformDependent* code
Motivation:

PlatformDependent* contains some methods that are not used and some other things that can be cleaned-up.

Modifications:

- Remove unused methods
- cleanup

Result:

Code cleanup.
2017-01-19 07:55:18 +01:00
Norman Maurer
7f01da8d0f [maven-release-plugin] prepare for next development iteration 2017-01-12 11:36:51 +01:00
Norman Maurer
7a21eb1178 [maven-release-plugin] prepare release netty-4.1.7.Final 2017-01-12 11:35:58 +01:00
Norman Maurer
eb5dc4bced Correctly handle IPV6 in HttpProxyHandler
Motivation:

The HttpProxyHandler is expected to be capable of issuing a valid CONNECT request for a tunneled connection to an IPv6 host.

Modifications:

- Correctly format the IPV6 address.
- Add unit tests

Result:

HttpProxyHandler works with IPV6 as well. Fixes [#6152].
2017-01-12 07:51:37 +01:00
Scott Mitchell
ec3d077e0d DefaultHttp2Connection modifying child map while iterating
Motivation:
When DefaultHttp2Connection removes a stream it iterates over all children and adds them as children to the parent of the stream being removed. This process may remove elements from the child map while iterating without using the iterator's remove() method. This is generally unsafe and may result in an undefined iteration.

Modifications:
- We should use the Iterator's remove() method while iterating over the child map

Result:
Fixes https://github.com/netty/netty/issues/6163
2017-01-11 11:07:18 -08:00
Norman Maurer
3c5e677964 Add assert to ensure we not create an endless loop and fix unsafe publication
Motivation:

[#6153] reports an endless loop that existed in the Recycler, while this was fixed adding a few asserts to ensure this remains fixed is a good thing. Beside this we also should ensure this can not escape the constructor to avoid unsafe publication.

Modifications:

- Add asserts
- Fix unsafe publication

Result:

More correct code.
2017-01-11 12:14:27 +01:00
Norman Maurer
2368f238ad Fix typo in inner-class name
Motivation:

There is a typo in the inner-class name.

Modifications:

Fix typo.

Result:

One typo less. Fixes [#6185].
2017-01-10 13:49:52 +01:00
Shixiong Zhu
2457f386d8 Set prev to null when setting cursor to head in scavengeSome.
Motivation:

`scavengeSome()` has a corner case: when setting `cursor` to `head`, `this.prev` may point to the tail of the  `WeakOrderQueue` linked list. Then it's possible that the following while loop will link the tail to the head, and cause endless loop.

I made a reproducer in 36522e7b72 . The unit test will just run forever. Unfortunately, I cannot change it to a unit test because it needs to add some codes to `scavengeSome` to control the execution flow.

Modification:

Set `prev` to null when setting `cursor` to `head` in `scavengeSome`

Result:

Fixes #6153.
2017-01-07 20:48:46 +01:00
Max Zhuravkov
a8950dfc4c InternalThreadLocalMap.arrayList should create a reusable ArrayList only if arrayList field is NULL.
Motivation:

InternalThreadLocalMap.arrayList returns a new ArrayList every time it's called that defeats the purpose of having a reusable ArrayList.

Modification:

Modified InternalThreadLocalMap.arrayList to create an ArrayList only if arrayList field is NULL.

Result:

InternalThreadLocalMap.arrayList now creates a reusable ArrayList only if arrayList field is NULL.
2017-01-03 12:54:50 -08:00
Norman Maurer
31da0ddbac Revert "Disallow setting logger factory twice"
This reverts commit 3c92f2b64a which needs more thoughts and so will go into the next release.
2016-12-21 15:14:53 +01:00
Norman Maurer
28c39a3183 Ensure we use a MPMC queue in ThreadDeathWatcher as it may be used from multiple threads at the same time.
Motivation:

We used a MPSC queue in ThreadDeathWatcher and checked if it empty via isEmpty() from multiple threads if very unlucky. Depending on the implementation this is not safe and may even produce things like live-locks.

Modifications:

Change to use a MPMC queue.

Result:

No more risk to run into issues when multiple threads call watch(...) / unwatch(...) concurrently.
2016-12-21 07:31:20 +01:00
Scott Mitchell
3d11334151 Fix DefaultChannelId MAC address parsing bug
Motivation:
DefaultChannelId provides a regular expression which validates if a user provided MAC address is valid. This regular expression may allow invalid MAC addresses and also not allow valid MAC addresses.

Modifications:
- Introduce a MacAddressUtil#parseMac method which can parse and validate the MAC address at the same time. The regular expression check before hand is additional overhead if we have to parse the MAC address.

Result:
Fixes https://github.com/netty/netty/issues/6132.
2016-12-20 17:06:27 -08:00
Jon Chambers
7c630feefc Document and generalize PromiseCombiner
Motivation:

`PromiseCombiner` is really handy, but it's not obvious how to use it from its existing documentation/method signatures.

Modification:

- Added javadoc comments to explain the theory of operation of `PromiseCombiner`.
- Generalized `PromiseCombiner` to work with `Futures` so it's clearer that the things for which it's listening won't be modified.

Result:

`PromiseCombiner` is easier to understand.
2016-12-20 11:15:03 +01:00
Scott Mitchell
06e7627b5f Read Only Http2Headers
Motivation:
A read only implementation of Http2Headers can allow for a more efficient usage of memory and more performant combined construction and iteration during serialization.

Modifications:
- Add a new ReadOnlyHttp2Headers class

Result:
ReadOnlyHttp2Headers exists and can be used for performance reasons when appropriate.

```
Benchmark                                            (headerCount)  Mode  Cnt    Score   Error  Units
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders               1  avgt   20   96.156 ± 1.902  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders               5  avgt   20  157.925 ± 3.847  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders              10  avgt   20  236.257 ± 2.663  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultClientHeaders              20  avgt   20  392.861 ± 3.932  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders               1  avgt   20   48.759 ± 0.466  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders               5  avgt   20  113.122 ± 0.948  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders              10  avgt   20  192.698 ± 1.936  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultServerHeaders              20  avgt   20  348.974 ± 3.111  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                    1  avgt   20   35.694 ± 0.271  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                    5  avgt   20   98.993 ± 2.933  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                   10  avgt   20  171.035 ± 5.068  ns/op
ReadOnlyHttp2HeadersBenchmark.defaultTrailers                   20  avgt   20  330.621 ± 3.381  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders              1  avgt   20   40.573 ± 0.474  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders              5  avgt   20   56.516 ± 0.660  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders             10  avgt   20   76.890 ± 0.776  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyClientHeaders             20  avgt   20  117.531 ± 1.393  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders              1  avgt   20   29.206 ± 0.264  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders              5  avgt   20   44.587 ± 0.312  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders             10  avgt   20   64.458 ± 1.169  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyServerHeaders             20  avgt   20  107.179 ± 0.881  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                   1  avgt   20   21.563 ± 0.202  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                   5  avgt   20   41.019 ± 0.440  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                  10  avgt   20   64.053 ± 0.785  ns/op
ReadOnlyHttp2HeadersBenchmark.readOnlyTrailers                  20  avgt   20  113.737 ± 4.433  ns/op
```
2016-12-18 09:32:24 -08:00
Norman Maurer
fe2b55cea1 Allow to disable deleting of the native library file after it is loaded.
Motivation:

When profiling it is sometimes needed to still have the native library file avaible. We should allow to disable the explicit deletion and just delete it when the JVM stops.

This is related to #6110

Modifications:

Add io.netty.native.deleteLibAfterLoading system property which allows to disable the explicit delete after laoding

Result:

Possible to profile native libraries better.
2016-12-16 08:28:01 +00:00
Norman Maurer
89e93968ac Remove usage of own Atomic*FieldUpdater in favor of JDKs
Motivation:

In later Java8 versions our Atomic*FieldUpdater are slower then the JDK implementations so we should not use ours anymore. Even worse the JDK implementations provide for example an optimized version of addAndGet(...) using intrinsics which makes it a lot faster for this use-case.

Modifications:

- Remove methods that return our own Atomic*FieldUpdaters.
- Use the JDK implementations everywhere.

Result:

Faster code.
2016-12-15 08:09:06 +00:00
Norman Maurer
f6ac8b5d32 [#6114] Increase test timeout for test introduced in c2f4daa739
Motivation:

c2f4daa739 added a unit test but used a too small test timeout.

Modifications:

Increase timeout.

Result:

Test should have enough time to complete on the CI.
2016-12-08 13:17:42 +01:00
Jason Tedor
3c92f2b64a Disallow setting logger factory twice
Motivation:

InternalLoggerFactory either sets a default logger factory
implementation based on the logging implementations on the classpath, or
applications can set a logger factory explicitly. If applications wait
too long to set the logger factory, Netty will have already set a logger
factory leading to some objects using one logging implementation and
other objets using another logging implementation. This can happen too
if the application tries to set the logger factory twice, which is
likely a bug in the application. Yet, the Javadocs for
InternalLoggerFactory warn against this saying that
InternalLoggerFactory#setLoggerFactory "should be called as early as
possible and shouldn't be called more than once". Instead, Netty should
guard against this.

Modications:

We replace the logger factory field with an atomic reference on which we
can do CAS operations to safely guard against it being set twice. We
also add an internal holder class that captures the static interface of
InternalLoggerFactory that can aid in testing.

Result:

The logging factory can not be set twice, and applications that want to
set the logging factory must do it before any Netty classes are
initialized (or the default logger factory will be set).
2016-12-08 10:38:01 +01:00
Norman Maurer
c2f4daa739 Fix false-positives when using ResourceLeakDetector.
Motivation:

We need to ensure the tracked object can not be GC'ed before ResourceLeak.close() is called as otherwise we may get false-positives reported by the ResourceLeakDetector. This can happen as the JIT / GC may be able to figure out that we do not need the tracked object anymore and so already enqueue it for collection before we actually get a chance to close the enclosing ResourceLeak.

Modifications:

- Add ResourceLeakTracker and deprecate the old ResourceLeak
- Fix some javadocs to correctly release buffers.
- Add a unit test for ResourceLeakDetector that shows that ResourceLeakTracker has not the problems.

Result:

No more false-positives reported by ResourceLeakDetector when ResourceLeakDetector.track(...) is used.
2016-12-04 09:01:39 +01:00
Norman Maurer
f332a00e1a Support compiling netty with Java9
Motivation:

Java9 will be released soon so we should ensure we can compile netty with Java9 and run all our tests. This will help to make sure Netty will be usable with Java9.

Modification:

- Add some workarounds to be able to compile with Java9, note that the full profile is not supported with Java9 atm.
- Remove some usage of internal APIs to be able to compile on java9
- Not support Alpn / Npn and so not run the tests when using Java9 for now. We will do a follow up PR to add support.

Result:

Its possible to build netty and run its testsuite with Java9.
2016-12-03 20:12:56 +01:00
Norman Maurer
fd0ae840b6 Small performance improvements in ResourceLeakDetector
Motivation:

42fba015ce changed the implemention of ResourceLeakDetector to improve performance. While this was done a branch was missed that can be removed. Beside this  using a Boolean as value for the ConcurrentMap is sub-optimal as when calling remove(key, value) an uncessary instanceof check and cast is needed on each removal.

Modifications:

- Remove branch which is not needed anymore
- Replace usage of Boolean as value type of the ConcurrentMap and use our own special type which only compute hash-code one time and use a == operation for equals(...) to reduce overhead present when using Boolean.

Result:

Faster and cleaner ResourceLeakDetector.
2016-12-01 21:28:46 +01:00
Jason Tedor
7dac4fdd25 Do not try to be unsafe when told not to be unsafe
Motivation:

Netty has a flag (io.netty.noUnsafe) for specifying to Netty to not be
unsafe. Yet, when initializing PlatformDependent0, Netty still tries to
be unsafe. For application that specify to Netty to not be unsafe and
run under a security manager, this can lead to an obnoxious (debug
level) stack trace. Since Netty was told not to be unsafe, Netty should
not try to be unsafe.

Modifications:

The initialization logic in PlatformDependent0 should take into account
that Netty was told not to be unsafe. This means that we need to
initialize PlatformDependent#IS_EXPLICIT_NO_UNSAFE as soon as possible,
before the static initializer for PlatformDependent0 has a chance to
run. Thus the following modifications are made:
 - initialize PlatformDependent#IS_EXPLICIT_NO_UNSAFE before any other
   code in PlatformDependent causes PlatformDependent0 to initialize
 - expose the value of PlatformDependent#IS_EXPLICIT_NO_UNSAFE for
   reading in PlatformDependent0
 - take the value of PlatformDependent#IS_EXPLICIT_NO_UNSAFE into
   account in PlatformDependent0

Result:

Netty does not try to be unsafe when told not to be unsafe.
2016-11-29 08:21:20 +01:00
Jason Tedor
b9959c869b Do not eagerly initialize the logger factory
Motivation:

For applications that set their own logger factory, they want that
logger factory to be the one logger factory. Yet, Netty eagerly
initializes this and then triggers initialization of other classes
before the application has had a chance to set its preferred logger
factory.

Modifications:

With this commit there are two key changes:
 - Netty does not attempt to eagerly initialize the default logger
   factory, only doing so if the application layer above Netty has not
   already set a logger factory
 - do not eagerly initialize unrelated classes from the logger factory;
   while the motivation behind this was to initialize ThreadLocalRandom
   as soon as possible in case it has to block reading from /dev/random,
   this can be worked around for applications where it is problematic by
   setting securerandom.source=file:/dev/urandom in their Java system
   security policy (no, it is not less secure; do not even get me
   started on myths about /dev/random)

Result:

Netty uses the logger factory that the application prefers, and does not
initialize unrelated classes.
2016-11-29 08:16:16 +01:00
Scott Mitchell
a043cf4a98 Catch exceptions from PlatformDependent#getSystemClassLoader
Motivation:
PlatformDependent#getSystemClassLoader may throw a wide variety of exceptions based upon the environment. We should handle all exceptions and continue initializing the slow path if an exception occurs.

Modifications:
- Catch Throwable in cases where PlatformDependent#getSystemClassLoader is used

Result:
Fixes https://github.com/netty/netty/issues/6038
2016-11-19 09:23:25 -08:00
Norman Maurer
0bc30a123e Eliminate usage of releaseLater(...) to reduce memory usage during tests
Motiviation:

We used ReferenceCountUtil.releaseLater(...) in our tests which simplifies a bit the releasing of ReferenceCounted objects. The problem with this is that while it simplifies stuff it increase memory usage a lot as memory may not be freed up in a timely manner.

Modifications:

- Deprecate releaseLater(...)
- Remove usage of releaseLater(...) in tests.

Result:

Less memory needed to build netty while running the tests.
2016-11-18 09:34:11 +01:00