Commit Graph

8418 Commits

Author SHA1 Message Date
Norman Maurer
0ee49e6d66 Eliminate noisy logging when using sun.misc.Unsafe and running on pre Java9
Motivation:

We should only try to load jdk.internal.misc.Unsafe if we run on Java9+ to eliminate noise in the log.

Modifications:

- Move javaVersion() and related methods to PlatformDependent0 to be able to use these in the static initializer without creating a cycle.
- Only try to load jdk.internal.misc.Unsafe when running in Java9+

Result:

Less noise in the log when running pre java9.
2017-05-16 08:29:06 +02:00
Norman Maurer
827c409656 Add uber-staging and uber-snapshot profile that can be used to generate uber all jars.
Motivation:

As we now include native code for multiple platforms we need to generate an uber all jar before release it from the staging repository. For this the uber-staging profile can be used. To create a snapshot uber jar the uber-snapshot profile can be used.

Modifications:

- Add uber-staging and uber-snapshot profile
- Correct comment in pom.xml file to show usage.

Result:

Easier to create snapshot and release uber jars.
2017-05-16 08:28:23 +02:00
Jason Tedor
d88cd23bfc Trim thread local string builder if large
Motivation:

A previous change allocated a new thread local string builder if it
was getting too large. This is a good change, these string builders
can accidentally get too large and then never shrunk and that is sort
of a memory leak. However, the change allocates an entirely new string
builder which is more allocations than necessary. Instead, we can trim
the string builder if its too large, this only allocates an extra
backing array instead of a whole new object.

Modifications:

If the string builder is above a threshold, we trim the string builder
and then ensure its capacity is reasonable to we do not allocate too
much as we start using the string builder.

Result:

The thread local string builder do not serve as a memory yet we do not
allocate too many new objects.
2017-05-12 08:20:04 +02:00
Norman Maurer
0db2901f4d [maven-release-plugin] prepare for next development iteration 2017-05-11 16:00:55 +02:00
Norman Maurer
f7a19d330c [maven-release-plugin] prepare release netty-4.1.11.Final 2017-05-11 16:00:16 +02:00
Nikolay Fedorovskikh
5643cc6a10 IPv6 validation fixes
Motivation:

`NetUtil`'s methods `isValidIpV6Address` and `getIPv6ByName` incorrectly validate some IPv6 addresses.

Modifications:

- `getIPv6ByName`: add checks for single colon at the start or end.
- `isValidIpV6Address`: fix checks for the count of colons and use `endOffset` instead of `ipAddress.length()` for the cases with the brackets or '%'.

Result:

More correct implementation of `NetUtil#isValidIpV6Address` and `NetUtil#getIPv6ByName`.
2017-05-11 08:10:25 -07:00
Norman Maurer
9e62c79574 Correctly include all modules during build
Motivation:

To ensure the release plugin works correctly we need to ensure all modules are included during build.

Modification:

- Include all modules
- Skip compilation and tests for native code when not supported but still include the module and build the jar

Result:

Build and release works again
2017-05-11 16:57:14 +02:00
Nitesh Kant
a093b89bfe Allow HTTP decoding post CONNECT in HttpClientCode
__Motivation__

`HttpClientCodec` skips HTTP decoding on the connection after a successful HTTP CONNECT response is received.
 This behavior follows the spec for a client but pragmatically, if one creates a client to use a proxy transparently, the codec becomes useless after HTTP CONNECT.
 Ideally, one should be able to configure whether HTTP CONNECT should result in pass-through or not. This will enable client writers to continue using HTTP decoding even after HTTP CONNECT.

 __Modification__

 Added overloaded constructors to accept `parseHttpPostConnect`. If this parameter is `true` then the codec continues decoding even after a successful HTTP CONNECT.

 Also fixed a bug in the codec that was incrementing request count post HTTP CONNECT but not decrementing it on response. Now, the request count is only incremented if the codec is not `done`.

 __Result__

 Easier usage by HTTP client writers who wants to connect to a proxy but still decode HTTP for their users for subsequent requests.
2017-05-11 16:48:16 +02:00
Norman Maurer
dd837fe803 Remove some dead-code and cleanup 2017-05-10 20:48:47 +02:00
Scott Mitchell
ce2ce9d7a4 ByteToMessageDecoder#handlerRemoved may release cumulation buffer prematurely
Motivation:
ByteToMessageDecoder#handlerRemoved will immediately release the cumulation buffer, but it is possible that a child class may still be using this buffer, and therefore use a dereferenced buffer.

Modifications:
- ByteToMessageDecoder#handlerRemoved and ByteToMessageDecoder#decode should coordinate to avoid the case where a child class is using the cumulation buffer but ByteToMessageDecoder releases that buffer.

Result:
Child classes of ByteToMessageDecoder are less likely to reference a released buffer.
2017-05-10 11:16:26 -07:00
Norman Maurer
c053c5144d Correctly detect if Ocsp is supported
Motivation:

We only used the openssl version to detect if Ocsp is supported or not which is not good enough as even the version is correct it may be compiled without support for OCSP (like for example on ubuntu).

Modifications:

Try to enable OCSP while static init OpenSsl and based on if this works return true or false when calling OpenSsl.isOcspSupported().

Result:

Correctly detect if OSCP is supported.
2017-05-10 08:42:28 +02:00
Nikolay Fedorovskikh
94e9448ae3 Simplify JUnit assertions
Motivation:

Some JUnit assert calls can be replaced by simpler.

Modifications:

Replacement with a more suitable methods.

Result:

More informative JUnit reports.
2017-05-09 20:19:10 +02:00
Norman Maurer
ec935c5a7b Correctly delete SelfSignedCertificate once done with it.
Motivation:

In OpenSsl init code we create a SelfSignedCertificate which we not explicitly delete. This can lead to have the deletion delayed.

Modifications:

Delete the SelfSignedCertificate once done with it.

Result:

Fixes [#6716]
2017-05-09 11:28:20 +02:00
Scott Mitchell
63f5cdb0d5 ByteBuf#ensureWritable(int, boolean) should not throw
Motivation:
The javadocs for ByteBuf#ensureWritable(int, boolean) indicate that it should not throw, and instead the return code should indicate the result of the operation. Due to a bug in AbstractByteBuf it is possible for a resize to be attempted on a buffer that may exceed maxCapacity() and therefore throw.

Modifications:
- If there is not enough space in the buffer, and force is false, then a resize should not be attempted

Result:
AbstractByteBuf#ensureWritable(int, boolean) enforces the javadoc constraints and does not throw.
2017-05-09 00:12:25 -07:00
Scott Mitchell
141089998f OpenSslEngine wrap may generate bad data if multiple src buffers
Motivation:
SSL_write requires a fixed amount of bytes for overhead related to the encryption process for each call. OpenSslEngine#wrap(..) will attempt to encrypt multiple input buffers until MAX_PLAINTEXT_LENGTH are consumed, but the size estimation provided by calculateOutNetBufSize may not leave enough room for each call to SSL_write. If SSL_write is not able to completely write results to the destination buffer it will keep state and attempt to write it later. Netty doesn't account for SSL_write keeping state and assumes all writes will complete synchronously (by attempting to allocate enough space to account for the overhead) and feeds the same data to SSL_write again later which results in corrupted data being generated.

Modifications:
- OpenSslEngine#wrap should only produce a single TLS packet according to the SSLEngine API specificaiton [1].
[1] https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLEngine.html#wrap-java.nio.ByteBuffer:A-int-int-java.nio.ByteBuffer-
- OpenSslEngine#wrap should only consider a single buffer when determining if there is enough space to write, because only a single buffer will ever be consumed.

Result:
OpenSslEngine#wrap will no longer produce corrupted data due to incorrect accounting of space required in the destination buffers.
2017-05-08 14:43:36 -07:00
jiachun.fjc
cd80b6c2d8 Use simple volatile read for SingleThreadEventExecutor#state instead of UNSAFE(AtomicIntegerFieldUpdater#get), CAS operation still to use AtomicIntegerFieldUpdater
Motivation:

AtomicIntegerFieldUpdater#get is unnecessary, I think use simple volatile read is cleaner

Modifications:

Replace code STATE_UPDATER.get(this) to state in SingleThreadEventExecutor

Result:

Cleaner code
2017-05-08 19:36:19 +02:00
Moses Nakamura
cf26227c6c Supply a builder for Http2Codec
Motivation:

DefaultHttp2FrameWriter has constructors that it would be a hassle to
expose as configuration parameters on Http2Codec. We should instead
make a builder for Http2Codec.

Modifications:

Get rid of the public constructors on Http2Codec and instead make sure
you can always use the builder where you would have used the constructor
before.

Result:

Http2Codec can be configured more flexibly, and the SensitivityDetector
can be configured.
2017-05-05 09:32:46 -07:00
jiachun.fjc
963cd22a05 InternalThreadLocalMap#stringBuilder: ensure memory overhead
Motivation:

InternalThreadLocalMap#stringBuilder: ensure memory overhead

Modification:

If the capacity of StringBuilder is greater than 65536 then release it on the next time you get StringBuilder and re-create a StringBuilder.

Result:

Possible less memory usage.
2017-05-05 09:28:51 -07:00
Norman Maurer
f65885fc54 Make DnsNameResolverTest pass on Java7
Motivation:

IDN.toUnicode(...) removes trailing dots when used in Java7 while it not does on java8.

Modifications:

Check if we should test with the trailing dot removed or not.

Result:

Test pass on Java7 as well.
2017-05-05 09:27:08 -07:00
Norman Maurer
aab89b058e Ensure Netty is usable on Java7
Motivation:

When adding SNIMatcher support we missed to use static delegating methods and so may try to load classes that not exists in Java7. Which will lead to errors.

Modifications:

- Correctly only try to load classes when running on java8+
- Ensure Java8+ related tests only run when using java8+

Result:

Fixes [#6700]
2017-05-04 14:10:53 -07:00
Dmitriy Dumanskiy
174f4ea005 HttpServerKeepAliveHandler doesn't correctly handle VoidChannelPromise
Motivation:

HttpServerKeepAliveHandler throws unexpected error when I do ctx.writeAndFlush(msg, ctx.voidPromise()); where msg is with header "Connection:close".

Modification:

HttpServerKeepAliveHandler does promise.unvoid() before adding close listener.

Result:

No error for VoidChannelPromise with HttpServerKeepAliveHandler. Fixes [#6698].
2017-05-04 14:08:18 -07:00
Dmitry Spikhalskiy
464ae9fb7a Expose CharSequence version of HttpUtil#getMimeType and HttpUtil#getCharset
Motivation:

It would be more flexible to make getCharset and getMimeType code usable not only for HttpMessage entity but just for any CharSequence. This will improve usability in general purpose code and will help to avoid multiple fetching of ContentType header from a message. It could be done in an external code once and CharSequence method versions could be applied.

Modification:
Expose HttpUtil#getMimeType, HttpUtil#getCharsetAsString, HttpUtil#getCharset versions which works with CharSequence. New methods are reused in the old ones which work with HttpMessage entity.

Result:

More flexible methods set with a good code reusing.
2017-05-03 14:14:39 -07:00
Jason Tedor
02a2738cd2 Do not try to use cleaner if no unsafe
Motivation:

If unsafe is unavailable, we can not use the cleaner anyway. If we try
to set it up, we get an annoying log message about unsafe being
unavailable (when debug logging is enabled). We know this will fail, so
we should not even bother and avoid the log message.

Modifications:

This commit adds a guard against setting up the cleaner if it is not
going to be available because unsafe is unavailable.

Result:

We do not try to set up the cleaner if unsafe is unavailable, and we do
not get an annoying log message.
2017-05-03 13:36:00 -07:00
Jason Tedor
9a0fd3a7b8 Do not log on explicit no unsafe again
Motivation:

Users should not see a scary log message when Netty is initialized if
Netty configuration explicitly disables unsafe. The log message that
produces this warning was previously guarded but the guard was
lost.

Modifications:

This commit brings back the guard against the scary log message if
unsafe is explicitly disabled.

Result:

No log message is produced when unsafe is unavailable because Netty was
told to not look for it.
2017-05-03 13:29:58 -07:00
Michael K. Werle
e70fbe316d Fire exceptionCaught before exception-caused close for WebSockets.
Motivation:

WebSocket decoding throws exceptions on failure that should cause the
pipline to close.  These are currently ignored in the
`WebSocketProtocolHandler` and `WebSocketServerProtocolHandler`.  In
particular, this means that messages exceding the max message size will
cause the channel to close with no reported failure.

Modifications:

Re-fire the event just before closing the socket to allow it to be
handled appropriately.

Result:

Closes [#3063].
2017-05-03 13:27:11 -07:00
Norman Maurer
a3e496a521 Not try to compresses HttpMessage if IDENTITY header value is set.
Motivation:

If Content-Encoding: IDENTITY is used we should not try to compress the http message but just let it pass-through.

Modifications:

Remove "!"

Result:

Fixes [#6689]
2017-05-03 10:55:13 -07:00
Scott Mitchell
3cc4052963 New native transport for kqueue
Motivation:
We currently don't have a native transport which supports kqueue https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2. This can be useful for BSD systems such as MacOS to take advantage of native features, and provide feature parity with the Linux native transport.

Modifications:
- Make a new transport-native-unix-common module with all the java classes and JNI code for generic unix items. This module will build a static library for each unix platform, and included in the dynamic libraries used for JNI (e.g. transport-native-epoll, and eventually kqueue).
- Make a new transport-native-unix-common-tests module where the tests for the transport-native-unix-common module will live. This is so each unix platform can inherit from these test and ensure they pass.
- Add a new transport-native-kqueue module which uses JNI to directly interact with kqueue

Result:
JNI support for kqueue.
Fixes https://github.com/netty/netty/issues/2448
Fixes https://github.com/netty/netty/issues/4231
2017-05-03 09:53:22 -07:00
Guanpeng Xu
2d38a4453c Remove the comment which is a bit misleading
This fixes #6652.

Rationale

The invocation of initChannel of ChannelInitializer has been moved to as
early as during handlerAdded is invoked in 26aa34853, whereas it was
only invoked during channelRegistered is invoked before that. So the
comment does not describe how handlers are added in normal circumstances
anymore.

However, the code is kept as-is since there might be unusual cases, and
adding ServerBootstrapAcceptor via the event loop is always safe to
enforce the correct order.
2017-05-02 15:36:20 -07:00
jiachun.fjc
e58095c4f2 Simplify code
Motivation:

Code can be simplified

Modification:

Refactor code to remove extra branching

Result:

Cleaner code.
2017-05-02 15:16:19 -07:00
Vladimir Kostyukov
ed37cf20ef Introduce HashedWheelTimer.pendingTimeouts()
Motivation:

Fixes #6681.

Modification:

For the sake of better timer observability, expose the number of pending timeouts through the new HashedWheelTimer.pendingTimeouts method .

Result:

It's now ridiculously easy to observe Netty timer's very basic and yet important metric, the number of pending tasks/timeouts.
2017-05-01 20:10:22 -07:00
Norman Maurer
6915ec3bb9 [maven-release-plugin] prepare for next development iteration 2017-04-29 14:10:00 +02:00
Norman Maurer
f30f242fee [maven-release-plugin] prepare release netty-4.1.10.Final 2017-04-29 14:09:32 +02:00
Norman Maurer
3ba4da8a79 Mark conscrypt as optional
Motivation:

Conscrypt is not needed when using the handler module, so it should be marked as optional

Modifications:

Mark conscrypt as optional

Result:

Be able to use handler module without conscrypt
2017-04-28 17:36:31 +02:00
kennylbj
92efa3c916 Add UptimeServer and adjust UptimeClient's code style.
Motivation:

Uptime example is lack of server.
UptimeClient's code style is a little bit different from others, which make reader feel confused.
We don't need to create a new Bootstrap instance each time client reconnect to server.

Modification:

Add UptimeServer and UptimeServerHandler which simply accept all connection and discard all message.
Change UptimeClient's code style.
Share a single Bootstrap instance.

Result:

Uptime server support.
Consistent code style.
Single Bootstrap for all reconnection.
2017-04-28 07:41:07 +02:00
Scott Mitchell
ea1cb20c90 Netutil IPv6 bugs
Motivation:
NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName allowed an invalid form of mapped IPv4 addresses which lead to accepting invalid IPv6 addresses as valid.

Modifications:
- NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName should only allow 7 colons for an IPv4 address if they are the first 2 characters.

Result:
More correct implementation of NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName
2017-04-28 07:29:36 +02:00
Daniel Schobel
b1cb059540 Motivation:
It is generally useful to have origin http servers respond to
"expect: continue-100" as soon as possible but applications without a
HttpObjectAggregator in their pipelines must use boiler plate to do so.

Modifications:

Introduce the HttpServerExpectContinueHandler handler to make it easier.

Result:

Less boiler plate for http application authors.
2017-04-27 16:20:29 -07:00
Aron Wieck
ffd6911586 Use constant string instead of user provided file name for DiskFileUpload temp file names.
Motivation:

DiskFileUpload creates temporary files for storing user uploads containing the user provided file name as part of the temporary file name. While most security problems are prevented by using "new File(userFileName).getName()" a small risk for bugs or security issues remains.

Modifications:

Use a constant string as file name and rely on the callers use of File.createTemp to ensure unique disk file names.

Result:

A slight security improvement at the cost of a little more obfuscated temp file names.
2017-04-27 16:02:41 -07:00
Scott Mitchell
5a2d04684e DNS Resolver visibility into individual queries
Motivation:
A single DNS query may follow many different paths through resolver-dns. The query may fail for various reasons related to the DNS protocol, general IO errors, it may be cancelled due to the query count being exceeded, or other reasons. A query may also result in other queries as we follow the DNS protocol (e.g. redirects, CNAME, etc...). It is currently impossible to collect information about the life cycle of an individual query though resolver-dns. This information may be valuable when considering which DNS servers are preferred over others.

Modifications:
- Introduce an interface which can provide visibility into all the potential outcomes of an individual DNS query

Result:
resolver-dns provides visibility into individual DNS queries which can be used to avoid poorly performing DNS servers.
2017-04-27 15:17:20 -07:00
Johno Crawford
0123190214 Netty BOM inherit from parent and its own dependency management
Motivation:

A BOM should only expose dependency management for the artifacts that the project produces (so in this case, netty artifacts). It should not have an opinion about third party dependencies (otherwise including that BOM in a project potentially overrides decisions that were made about them).

Modifications:

Stop inheriting from netty-parent and explicitly set the versions for the artifacts.

Result:

Valid BOM.
2017-04-27 19:35:58 +02:00
Andrew McCall
231e6a5b7d Calls to discardSomeReadBytes() causes the JsonDecoder to get corrupted
Modification:

Added a lastReaderIndex value and if the current readerIndex has been reset, resets the idx and the decoder.

Result:

Fixes #6156.
2017-04-27 19:34:15 +02:00
Norman Maurer
b662afeece Correctly release all buffers in UnpooledTest
Motivation:

We not correctly released all buffers in the UnpooledTest and so showed "bad" way of handling buffers to people that inspect our code to understand when a buffer needs to be released.

Modifications:

Explicit release all buffers.

Result:

Cleaner and more correct code.
2017-04-27 19:29:45 +02:00
Norman Maurer
8b13d9656a Notify connect promise of ProxyHandler after codecs are removed
Motivation:

We need to notify the promise after the codecs are removed to allow writing from the listener and ensure we not try to do any encoding anymore. If we not do we will end up with corrupt data.

Modifications:

Notify promise after codecs are removed.

Result:

Fixes [#6671].
2017-04-27 14:54:56 +02:00
Roger Kapsi
57d3393527 Ability to extend SniHandler and configure it with arbitrary runtime data
Motivation

SniHandler is "hardcoded" to use hostname -> SslContext mappings but there are use-cases where it's desireable and necessary to return more information than a SslContext. The only option so far has been to use a delegation pattern

Modifications

Extract parts of the existing SniHandler into an abstract base class and extend SniHandler from it. Users can do the same by extending the new abstract base class and implement custom behavior that is possibly very different from the common/default SniHandler.

Touches

- f97866dbc6
- b604a22395

Result

Fixes #6603
2017-04-26 19:05:16 -07:00
Scott Mitchell
9cb858fcf6 NetUtil IPv6 bugs related to IPv4 and compression
Motivation:
NetUtil#getByName and NetUtil#isValidIpV6Address do not strictly enforce the format of IPv4 addresses that are allowed to be embedded in IPv6 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5. This may lead to invalid addresses being parsed, or invalid addresses being considered valid. Compression of a single IPv6 word was also not handled correctly if there are 7 : characters.

Modifications:
- NetUtil#isValidIpV6Address should enforce the IPv4-Compatible and IPv4-Mapped are the only valid formats for including IPv4 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5
- NetUtil#getByName should more stritcly parse IPv6 addresses which contain IPv4 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5
- NetUtil should allow compression even if the number of : characters is 7.
- NetUtil#createByteArrayFromIpAddressString should use the same IP string to byte[] translation which is used in NetUtil#getByName

Result:
NetUtil#getByName and NetUtil#isValidIpV6Address respect the IPv6 RFC which defines the valid formats for embedding IPv4 addresses.
2017-04-25 15:10:38 -07:00
Dmitriy Dumanskiy
a45f9d7939 Improvement : allocate less object during multipart form parsing. 2017-04-25 14:06:10 +02:00
Scott Mitchell
d21f2adb98 HTTP/2 StreamByteDistributor improve parameter validation
Motivation:
Each StreamByteDistributor may allow for priority in different ways, but there are certain characteristics which are invalid regardless of the distribution algorithm. We should validate these invalid characteristics at the flow controller level.

Modifications:
- Disallow negative stream IDs from being used. These streams may be accepted by the WeightedFairQueueByteDistributor and cause state for other valid streams to be evicted.
- Improve unit tests to verify limits are enforced.

Result:
Boundary conditions related to the priority parameters are validated more strictly.
2017-04-24 17:17:18 -07:00
Norman Maurer
3ac6d07168 Skip forbidden API check when running examples
Motivation:

We should skip the forbidden API check when run the examples as otherwise it may fail.

Modifications:

Skip the API check in run-example.sh

Result:

Be able to run the examples in all cases.
2017-04-24 09:52:06 +02:00
Norman Maurer
7f3b75a509 Revert "SslHandler avoid calling wrap/unwrap when unnecessary"
This reverts commit 6353c229fd to "fix" [#6578].
2017-04-23 21:08:03 +02:00
Norman Maurer
1c63cc8067 Skip forbidden API check when building with java9 for now
Motivation:

There needs to be some work be done to allow using forbidden API check plugin when using java9.

Modifications:

Skip forbidden API check when using java9

Result:

Builds again with java9
2017-04-23 20:17:55 +02:00
Jason Tedor
98beb777f8 Enable configuring available processors
Motivation:

In cases when an application is running in a container or is otherwise
constrained to the number of processors that it is using, the JVM
invocation Runtime#availableProcessors will not return the constrained
value but rather the number of processors available to the virtual
machine. Netty uses this number in sizing various resources.
Additionally, some applications will constrain the number of threads
that they are using independenly of the number of processors available
on the system. Thus, applications should have a way to globally
configure the number of processors.

Modifications:

Rather than invoking Runtime#availableProcessors, Netty should rely on a
method that enables configuration when the JVM is started or by the
application. This commit exposes a new class NettyRuntime for enabling
such configuraiton. This value can only be set once. Its default value
is Runtime#availableProcessors so that there is no visible change to
existing applications, but enables configuring either a system property
or configuring during application startup (e.g., based on settings used
to configure the application).

Additionally, we introduce the usage of forbidden-apis to prevent future
uses of Runtime#availableProcessors from creeping. Future work should
enable the bundled signatures and clean up uses of deprecated and
other forbidden methods.

Result:

Netty can be configured to not use the underlying number of processors,
but rather the constrained number of processors.
2017-04-23 10:31:17 +02:00