Commit Graph

8134 Commits

Author SHA1 Message Date
Scott Mitchell
ea1cb20c90 Netutil IPv6 bugs
Motivation:
NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName allowed an invalid form of mapped IPv4 addresses which lead to accepting invalid IPv6 addresses as valid.

Modifications:
- NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName should only allow 7 colons for an IPv4 address if they are the first 2 characters.

Result:
More correct implementation of NetUtil#isValidIpV6Address and NetUtil#getIPv6ByName
2017-04-28 07:29:36 +02:00
Daniel Schobel
b1cb059540 Motivation:
It is generally useful to have origin http servers respond to
"expect: continue-100" as soon as possible but applications without a
HttpObjectAggregator in their pipelines must use boiler plate to do so.

Modifications:

Introduce the HttpServerExpectContinueHandler handler to make it easier.

Result:

Less boiler plate for http application authors.
2017-04-27 16:20:29 -07:00
Aron Wieck
ffd6911586 Use constant string instead of user provided file name for DiskFileUpload temp file names.
Motivation:

DiskFileUpload creates temporary files for storing user uploads containing the user provided file name as part of the temporary file name. While most security problems are prevented by using "new File(userFileName).getName()" a small risk for bugs or security issues remains.

Modifications:

Use a constant string as file name and rely on the callers use of File.createTemp to ensure unique disk file names.

Result:

A slight security improvement at the cost of a little more obfuscated temp file names.
2017-04-27 16:02:41 -07:00
Scott Mitchell
5a2d04684e DNS Resolver visibility into individual queries
Motivation:
A single DNS query may follow many different paths through resolver-dns. The query may fail for various reasons related to the DNS protocol, general IO errors, it may be cancelled due to the query count being exceeded, or other reasons. A query may also result in other queries as we follow the DNS protocol (e.g. redirects, CNAME, etc...). It is currently impossible to collect information about the life cycle of an individual query though resolver-dns. This information may be valuable when considering which DNS servers are preferred over others.

Modifications:
- Introduce an interface which can provide visibility into all the potential outcomes of an individual DNS query

Result:
resolver-dns provides visibility into individual DNS queries which can be used to avoid poorly performing DNS servers.
2017-04-27 15:17:20 -07:00
Johno Crawford
0123190214 Netty BOM inherit from parent and its own dependency management
Motivation:

A BOM should only expose dependency management for the artifacts that the project produces (so in this case, netty artifacts). It should not have an opinion about third party dependencies (otherwise including that BOM in a project potentially overrides decisions that were made about them).

Modifications:

Stop inheriting from netty-parent and explicitly set the versions for the artifacts.

Result:

Valid BOM.
2017-04-27 19:35:58 +02:00
Andrew McCall
231e6a5b7d Calls to discardSomeReadBytes() causes the JsonDecoder to get corrupted
Modification:

Added a lastReaderIndex value and if the current readerIndex has been reset, resets the idx and the decoder.

Result:

Fixes #6156.
2017-04-27 19:34:15 +02:00
Norman Maurer
b662afeece Correctly release all buffers in UnpooledTest
Motivation:

We not correctly released all buffers in the UnpooledTest and so showed "bad" way of handling buffers to people that inspect our code to understand when a buffer needs to be released.

Modifications:

Explicit release all buffers.

Result:

Cleaner and more correct code.
2017-04-27 19:29:45 +02:00
Norman Maurer
8b13d9656a Notify connect promise of ProxyHandler after codecs are removed
Motivation:

We need to notify the promise after the codecs are removed to allow writing from the listener and ensure we not try to do any encoding anymore. If we not do we will end up with corrupt data.

Modifications:

Notify promise after codecs are removed.

Result:

Fixes [#6671].
2017-04-27 14:54:56 +02:00
Roger Kapsi
57d3393527 Ability to extend SniHandler and configure it with arbitrary runtime data
Motivation

SniHandler is "hardcoded" to use hostname -> SslContext mappings but there are use-cases where it's desireable and necessary to return more information than a SslContext. The only option so far has been to use a delegation pattern

Modifications

Extract parts of the existing SniHandler into an abstract base class and extend SniHandler from it. Users can do the same by extending the new abstract base class and implement custom behavior that is possibly very different from the common/default SniHandler.

Touches

- f97866dbc6
- b604a22395

Result

Fixes #6603
2017-04-26 19:05:16 -07:00
Scott Mitchell
9cb858fcf6 NetUtil IPv6 bugs related to IPv4 and compression
Motivation:
NetUtil#getByName and NetUtil#isValidIpV6Address do not strictly enforce the format of IPv4 addresses that are allowed to be embedded in IPv6 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5. This may lead to invalid addresses being parsed, or invalid addresses being considered valid. Compression of a single IPv6 word was also not handled correctly if there are 7 : characters.

Modifications:
- NetUtil#isValidIpV6Address should enforce the IPv4-Compatible and IPv4-Mapped are the only valid formats for including IPv4 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5
- NetUtil#getByName should more stritcly parse IPv6 addresses which contain IPv4 addresses as specified in https://tools.ietf.org/html/rfc4291#section-2.5.5
- NetUtil should allow compression even if the number of : characters is 7.
- NetUtil#createByteArrayFromIpAddressString should use the same IP string to byte[] translation which is used in NetUtil#getByName

Result:
NetUtil#getByName and NetUtil#isValidIpV6Address respect the IPv6 RFC which defines the valid formats for embedding IPv4 addresses.
2017-04-25 15:10:38 -07:00
Dmitriy Dumanskiy
a45f9d7939 Improvement : allocate less object during multipart form parsing. 2017-04-25 14:06:10 +02:00
Scott Mitchell
d21f2adb98 HTTP/2 StreamByteDistributor improve parameter validation
Motivation:
Each StreamByteDistributor may allow for priority in different ways, but there are certain characteristics which are invalid regardless of the distribution algorithm. We should validate these invalid characteristics at the flow controller level.

Modifications:
- Disallow negative stream IDs from being used. These streams may be accepted by the WeightedFairQueueByteDistributor and cause state for other valid streams to be evicted.
- Improve unit tests to verify limits are enforced.

Result:
Boundary conditions related to the priority parameters are validated more strictly.
2017-04-24 17:17:18 -07:00
Norman Maurer
3ac6d07168 Skip forbidden API check when running examples
Motivation:

We should skip the forbidden API check when run the examples as otherwise it may fail.

Modifications:

Skip the API check in run-example.sh

Result:

Be able to run the examples in all cases.
2017-04-24 09:52:06 +02:00
Norman Maurer
7f3b75a509 Revert "SslHandler avoid calling wrap/unwrap when unnecessary"
This reverts commit 6353c229fd to "fix" [#6578].
2017-04-23 21:08:03 +02:00
Norman Maurer
1c63cc8067 Skip forbidden API check when building with java9 for now
Motivation:

There needs to be some work be done to allow using forbidden API check plugin when using java9.

Modifications:

Skip forbidden API check when using java9

Result:

Builds again with java9
2017-04-23 20:17:55 +02:00
Jason Tedor
98beb777f8 Enable configuring available processors
Motivation:

In cases when an application is running in a container or is otherwise
constrained to the number of processors that it is using, the JVM
invocation Runtime#availableProcessors will not return the constrained
value but rather the number of processors available to the virtual
machine. Netty uses this number in sizing various resources.
Additionally, some applications will constrain the number of threads
that they are using independenly of the number of processors available
on the system. Thus, applications should have a way to globally
configure the number of processors.

Modifications:

Rather than invoking Runtime#availableProcessors, Netty should rely on a
method that enables configuration when the JVM is started or by the
application. This commit exposes a new class NettyRuntime for enabling
such configuraiton. This value can only be set once. Its default value
is Runtime#availableProcessors so that there is no visible change to
existing applications, but enables configuring either a system property
or configuring during application startup (e.g., based on settings used
to configure the application).

Additionally, we introduce the usage of forbidden-apis to prevent future
uses of Runtime#availableProcessors from creeping. Future work should
enable the bundled signatures and clean up uses of deprecated and
other forbidden methods.

Result:

Netty can be configured to not use the underlying number of processors,
but rather the constrained number of processors.
2017-04-23 10:31:17 +02:00
Dmitriy Dumanskiy
e78ccd6d52 #6657 do not throw ClassCastException when rule subnet version doesn't match remote IP version 2017-04-23 08:30:02 +02:00
Eric Anderson
799350c369 Fix HTTP/2 dependency tree corruption
Motivation:

Chrome was randomly getting stuck loading the tiles examples.
Investigation showed that the Netty flow controller thought it had
nothing to send for the connection even though some streams has queued
data and window available.

Modifications:

Fixed an accounting error where an implicitly created parent was not
being added to the dependency tree, thus it and all of its children were
orphaned from the connection's tree and would never have data written.

Result:

Fixes #6621
2017-04-22 08:26:47 -07:00
Eric Anderson
c6ad9338b3 Avoid infinite loop in HTTP/2 distributor toString()
Motivation:

Although effectively unused, the toString() of
WeightedFairQueueByteDistributor.State is useful for debugging. It
accidentally had an infinite loop, as it would recurse infinitely
between a parent and its child, which makes it less useful for
debugging.

Modifications:

Prune the infinite loop by using the parent's streamId instead of the
parent's toString().

Result:

Faster, less stack-overflowing toString()
2017-04-22 08:16:41 -07:00
Norman Maurer
7214740c06 Add support for SNIMatcher when using SslProvider.OPENSSL* and Java8+
Motivation:

Java8 adds support for SNIMatcher to reject SNI when the hostname not matches what is expected. We not supported doing this when using SslProvider.OPENSSL*.

Modifications:

- Add support for SNIMatcher when using SslProvider.OPENSSL*
- Add unit tests

Result:

SNIMatcher now support with our own SSLEngine as well.
2017-04-21 10:23:47 +02:00
Norman Maurer
34ff9cf5f2 Fix possible overflow when calculate in the size of the out buffer in Base64
Motivation:

We not correctly guarded against overflow and so call Base64.encode(...) with a big buffer may lead to an overflow when calculate the size of the out buffer.

Modifications:

Correctly guard against overflow.

Result:

Fixes [#6620].
2017-04-21 08:11:17 +02:00
Norman Maurer
c663a94359 Fix buffer leak in local transport when a close triggers the close of a remote peer and there are still messages in the inbound buffer.
Motivation:

We need to release all the buffers that may be put into our inbound queue since we closed the Channel to ensure we not leak any memory. This is fine as it basically gives the same guarantees as TCP which  means even if the promise was notified before its not really guaranteed that the "remote peer" will see the buffer at all.

Modifications:

Ensure we release all buffers in the inbound buffer if a doClose() is called.

Result:

No more leaks.
2017-04-21 07:42:29 +02:00
Norman Maurer
4dd6c14ba2 Only use test SslProviders that are supported in SslHandlerTest.testCompositeBufSizeEstimationGuaranteesSynchronousWrite().
Motivation:

We need to ensure we only try to to test with the SslProviders that are supported when running the SslHandlerTest.testCompositeBufSizeEstimationGuaranteesSynchronousWrite test.

Modifications:

Skip SslProvider.OPENSSL* if not supported.

Result:

No more test-failures if openssl is not installed on the system.
2017-04-20 19:29:52 +02:00
Norman Maurer
bf0beb772c Fix IllegalArgumentException when release a wrapped ByteBuffer on Java9
Motivation:

Unsafe.invokeCleaner(...) checks if the passed in ByteBuffer is a slice or duplicate and if so throws an IllegalArgumentException on Java9. We need to ensure we never try to free a ByteBuffer that was provided by the user directly as we not know if its a slice / duplicate or not.

Modifications:

Never try to free a ByteBuffer that was passed into UnpooledUnsafeDirectByteBuf constructor by an user (via Unpooled.wrappedBuffer(....)).

Result:

Build passes again on Java9
2017-04-20 19:19:11 +02:00
Nikolay Fedorovskikh
970d310ec9 Regulation of the InternetProtocolFamily usage
Motivation:

1. The use of InternetProtocolFamily is not consistent:
   the DnsNameResolverContext and DnsNameResolver contains switches
   instead of appropriate methods usage.
2. The InternetProtocolFamily class contains redundant switches in the
   constructor.

Modifications:

1. Replacing switches to the use of an appropriate methods.
2. Simplifying the InternetProtocolFamily constructor.

Result:

Code is cleaner and simpler.
2017-04-20 05:22:24 +02:00
Nikolay Fedorovskikh
0692bf1b6a fix the typos 2017-04-20 04:56:09 +02:00
Norman Maurer
38483e8790 Correctly manage buffer life-cycle in http2 multiplex example
Motivation:

We not correctly managed the life-cycle of the buffer / frames in our http2 multiplex example which lead to a memory leak.

Modifications:

- Correctly release frame if not echo'ed back the remote peer.
- Not retain content before echo back to remote peer.

Result:

No more leak in the example, fixes [#6636].
2017-04-19 14:19:50 +02:00
Norman Maurer
a0fcb72e5d Use jndi-dns to obtain default name servers
Motivation:

Using reflection to obtain the default name servers may fail in Java9 and also in previous Java versions if a SecurityManager is present.

Modifications:

Try using jndi-dns to obtain default name servers and only try using reflection if this fails.

Result:

Be able to detect default name servers in all cases. Fixes [#6347].
2017-04-19 12:24:06 +02:00
Norman Maurer
e482d933f7 Add 'io.netty.tryAllocateUninitializedArray' system property which allows to allocate byte[] without memset in Java9+
Motivation:

Java9 added a new method to Unsafe which allows to allocate a byte[] without memset it. This can have a massive impact in allocation times when the byte[] is big. This change allows to enable this when using Java9 with the io.netty.tryAllocateUninitializedArray property when running Java9+. Please note that you will need to open up the jdk.internal.misc package via '--add-opens java.base/jdk.internal.misc=ALL-UNNAMED' as well.

Modifications:

Allow to allocate byte[] without memset on Java9+

Result:

Better performance when allocate big heap buffers and using java9.
2017-04-19 11:45:39 +02:00
Norman Maurer
fb113dce3a HttpPostRequestEncoder may return a slice which is not retained.
Motivation:

We miss to retain a slice before return it to the user and so an reference count error may accour later on.

Modifications:

Use readRetainedSlice(...) and so ensure we retain the buffer before hand it of to the user.

Result:

Fixes [#6626].
2017-04-19 11:40:38 +02:00
Norman Maurer
1b0b8f80cd AbstractScheduledEventExecutor.schedule(...) must accept delay <= 0.
Motivation:

As the javadoc of ScheduledExecutorService state:

Zero and negative delays (but not periods) are also allowed in schedule methods,and are treated as requests for immediate execution.

Modifications:

- Correctly handle delay <= 0.
- Add unit tests.

Result:

Fixes [#6627].
2017-04-19 11:35:50 +02:00
Norman Maurer
119383873d VoidChannelPromise not notified when exception is thrown.
Motivation:

When a VoidChannelPromise is used by the user we need to ensure we propergate the exception through the ChannelPipeline otherwise the exception will just be swallowed and so the user has no idea whats going on.

Modifications:

- Always call tryFailure / trySuccess even when we use the VoidChannelPromise
- Add unit test

Result:

Fixes [#6622].
2017-04-19 11:25:59 +02:00
Norman Maurer
38b054c65c Correctly handle read-only ByteBuf in ByteToMessageDecoder
Motivation:

If a read-only ByteBuf is passed to the ByteToMessageDecoder.channelRead(...) method we need to make a copy of it once we try to merge buffers for cumulation. This usually is not the case but can for example happen if the local transport is used. This was the cause of the leak report we sometimes saw during the codec-http2 tests, as we are using the local transport and write a read-only buffer. This buffer will then be passed to the peer channel and fired through the pipeline and so end up as the cumulation buffer in the ByteToMessageDecoder. Once the next fragement is received we tried to merge these and failed with a ReadOnlyBufferException which then produced a leak.

Modifications:

Ensure we copy the buffer if its read-only.

Result:

No more exceptions and so leak when a read-only buffer is passed to ByteToMessageDecoder.channelRead(...)
2017-04-19 07:26:26 +02:00
Norman Maurer
4436d697a6 Update to netty-tcnative 2.0.1.Final 2017-04-18 13:45:10 +02:00
Norman Maurer
1dfd852dff Revert "Add support for SNIMatcher when using SslProvider.OPENSSL* and Java8+"
This reverts commit cc5d1d0a7e.
2017-04-18 13:43:03 +02:00
Norman Maurer
cc5d1d0a7e Add support for SNIMatcher when using SslProvider.OPENSSL* and Java8+
Motivation:

Java8 adds support for SNIMatcher to reject SNI when the hostname not matches what is expected. We not supported doing this when using SslProvider.OPENSSL*.

Modifications:

- Add support for SNIMatcher when using SslProvider.OPENSSL*
- Add unit tests

Result:

SNIMatcher now support with our own SSLEngine as well.
2017-04-18 08:16:33 +02:00
Lukasz Strzalkowski
7bd0905969 Introduce ReferenceCounted.refCnt()
Motivation:

When debugging netty memory leaks, it's sometimes helpful to
print the object's reference count.

Modifications:

Add `refCnt` methods to set of already exitsting helpers for ref coutned
objects.

Result:

Users will have utility to print object's ref count without much of a
boilerplate.
2017-04-17 19:43:44 +02:00
Norman Maurer
7b6119a0a4 Allow to free direct buffers on java9 again
Motivation:

Java9 adds a new method to Unsafe which allows to free direct ByteBuffer via the cleaner without the need to use an commandline arguments.

Modifications:

- Add Cleaner interface
- Add CleanerJava9 which will be used when using Java9+ and take care of release direct ByteBuffer
- Let Cleaner0 implement Cleaner

Result:

Be able to free direct ByteBuffer on Java9+ again without any commandline arguments.
2017-04-17 19:40:52 +02:00
Vladimir Kostyukov
4c77e7c55a netty-codec: Manage read-flow explicitly in MessageAggregator 2017-04-17 19:37:43 +02:00
Brendt Lucas
dcd322dda2 Fix regression in QueryStringEncoder
Motivation:

Commit #d675febf07d14d4dff82471829f974369705655a introduced a regression in QueryStringEncoder, resulting in whitespace being converted into a literal `+` sign instead of `%20`.

Modification:

Modify `encodeComponent` to pattern match and replace on the result of the call to `URLEncoder#encode`

Result:

Fixes regression
2017-04-17 19:32:50 +02:00
Scott Mitchell
155983f1a1 DNS move JDK DNS resolution out of DnsServerAddresses static initialization
Motivation:
DnsServerAddresses loads the default DNS servers used for DNS resolution in a static initialization block. This is subject to blocking and may cause unexpected delays. We can move this initialization to DefaultDnsServerAddressStreamProvider where it is more expected to load the JDK's default configuration.

Modifications:
- Move all the static initialization from DnsServerAddresses to DefaultDnsServerAddressStreamProvider
- Deprecate static methods in DnsServerAddresses which have moved to DefaultDnsServerAddressStreamProvider
- Remove usage of deprecated methods in DnsServerAddresses

Result:
Usage of JDK's blocking DNS resolver is not required to use resolver-dns.
2017-04-06 18:09:58 -07:00
Trustin Lee
08646afc1e Do not fail a DNS query promise prematurely
Motivation:

DnsNameResolverContext completes its DNS query promise automatically
when no queries are in progress, which means there's no need to fail the
promise explicitly.

Modifications:

- Do not fail a DNS query promise explicitly but add an informational
  trace

Result:

- Fixes #6600
- Unexpected exception on one question type does not fail the promise
  too soon. If the other question succeeds, the query will succeed,
  making the resolver more robust.
2017-04-06 17:58:54 -07:00
Scott Mitchell
1bc5bc69e3 HTTP/2 Allow more time for EventLoopGroup to shutdown in test that use LocalChannel
Motivation:
The CI servers have reported leaks while building the HTTP/2 unit tests. The unit tests attempt to wait for the channels to be closed before exiting the test, but we should wait in case there are any tasks pending on the EventLoopGroup's task queues.

Modifications:
- Change the Future.sync() operations to Future.syncUninterruptibly()
- HTTP/2 unit tests which use local channel should wait for 5 seconds before shutting down the EventLoopGroups

Result:
More likely that any cleanup related tasks will execute before the unit tests are shutdown.
2017-04-06 17:56:21 -07:00
Scott Mitchell
c37267d682 NioEventLoop#rebuildSelector0 throws ClassCastException
Motivation:
Commit 795f318 simplified some code related to the special case Set for the selected keys and introduced a Selector wrapper to make sure this set was properly reset. However the JDK makes assumptions about the type of Selector and this type is not extensible. This means whenever we call into the JDK we must provide the unwrapped version of the Selector or we get a ClassCastException. We missed a case of unwrapping in NioEventLoop#rebuildSelector0.

Modificaitons:
- NioEventLoop#openSelector should return a tuple so we can atomically set the wrapped and unwrapped Selector
- NioEventLoop#rebuildSelector0 should use the unwrapped version of the selector

Result:
Fixes https://github.com/netty/netty/issues/6607.
2017-04-05 19:34:49 -07:00
Roger Kapsi
077a1988b9 OCSP stapling support for Netty using netty-tcnative.
https://github.com/netty/netty-tcnative/pull/215

Motivation

OCSP stapling (formally known as TLS Certificate Status Request extension) is alternative approach for checking the revocation status of X.509 Certificates. Servers can preemptively fetch the OCSP response from the CA's responder, cache it for some period of time, and pass it along during (a.k.a. staple) the TLS handshake. The client no longer has to reach out on its own to the CA to check the validity of a cetitficate. Some of the key benefits are:

1) Speed. The client doesn't have to crosscheck the certificate.
2) Efficiency. The Internet is no longer DDoS'ing the CA's OCSP responder servers.
3) Safety. Less operational dependence on the CA. Certificate owners can sustain short CA outages.
4) Privacy. The CA can lo longer track the users of a certificate.

https://en.wikipedia.org/wiki/OCSP_stapling
https://letsencrypt.org/2016/10/24/squarespace-ocsp-impl.html

Modifications

https://www.openssl.org/docs/man1.0.2/ssl/SSL_set_tlsext_status_type.html

Result

High-level API to enable OCSP stapling
2017-04-03 11:56:53 -07:00
Kevin Oliver
34e0007f07 LoggingHandler does not override channelReadComplete or channelWritabilityChanged
Motivation:

`io.netty.handler.logging.LoggingHandler` does not log when these
events happen.

Modifiations:

Add overrides with logging to these methods.

Result:

Logging now happens for these two events.
2017-04-03 11:46:01 -07:00
Scott Mitchell
225d10e1ad HTTP/2 Make DefaultHttp2HeadersDecoder's Http2Headers object creation extensible
Motivation:
It is generally useful to override DefaultHttp2HeadersDecoder's creation of a new Http2Headers object so more optimized versions can be substituted if the use case allows for it.

Modifications:
- DefaultHttp2HeadersDecoder should support an overridable method to generate the new Http2Headers object for each decode operation

Result:
DefaultHttp2HeadersDecoder is more extensible.
Fixes https://github.com/netty/netty/issues/6591.
2017-04-03 11:19:14 -07:00
Scott Mitchell
e8da5e5162 Revert "Expose HTTP/2 HpackDecoder (#6589)"
This reverts commit f4c635d30b.
2017-04-03 11:19:09 -07:00
Norman Maurer
4bcfa07a7d Fix OpenSslCertificateException error code validation
Motivation:

In OpenSslCertificateException we tried to validate the supplied error code but did not correctly account for all different valid error codes and so threw an IllegalArgumentException.

Modifications:

- Fix validation by updating to latest netty-tcnative and use CertificateVerifier.isValid
- Add unit tests

Result:

Validation of error code works as expected.
2017-04-03 11:12:15 -07:00
kennylbj
00bf06e97b Fix spelling mistake of HttpStaticFileServerHandler.java 2017-04-03 10:59:25 -07:00