Commit Graph

8705 Commits

Author SHA1 Message Date
Stephane Landelle
d60cd0231d HttpProxyHandler generates invalid CONNECT url and Host header when address is resolved
Motivation:

HttpProxyHandler uses `NetUtil#toSocketAddressString` to compute
CONNECT url and Host header.

The url is correct when the address is unresolved, as
`NetUtil#toSocketAddressString` will then use
`getHoststring`/`getHostname`. If the address is already resolved, the
url will be based on the IP instead of the hostname.

There’s an additional minor issue with the Host header: default port
443 should be omitted.

Modifications:

* Introduce NetUtil#getHostname
* Introduce HttpUtil#formatHostnameForHttp to format an
InetSocketAddress to
HTTP format
* Change url computation to favor hostname instead of IP
* Introduce HttpProxyHandler ignoreDefaultPortsInConnectHostHeader
parameter to ignore 80 and 443 ports in Host header

Result:

HttpProxyHandler performs properly when connecting to a resolved address
2018-03-27 09:43:11 +02:00
Norman Maurer
fc3b145cbb Correctly handle non IOException during read in NioServerSocketChannel
Motivation:

Our code was not correct in AbstractNioMessageChannel.closeOnReadError(....) which lead to the situation that we always tried to continue reading no matter what exception was thrown when using the NioServerSocketChannel. Also even on an IOException we should check if the Channel itself is still active or not and if not stop reading.

Modifications:

Fix closeOnReadError impl and added test.

Result:

Correctly stop reading on NioServerSocketChannel when error happens during read.
2018-03-25 17:31:59 +02:00
Norman Maurer
8189399e9d Ignore EINTR on close(...) as there is nothing sane we can do.
Motivation:

If close(...) reports EINTR there is nothing sane we can do so it makes no sense to even report it. See also:

https://github.com/apple/swift-nio/pull/217

Modifications:

Just ignore EINTR when calling close(...)

Result:

Less noise in the logs.
2018-03-23 07:39:17 +01:00
Norman Maurer
40af10b782 Skip NPN tests when libressl 2.6.1+ is used.
Motivation:

LibreSSL removed support for NPN in its 2.6.1+ releases.

Modifications:

Skip NPN tests in libressl 2.6.1+

Result:

Be able to run netty tests against libressl 2.6.1+ as well.
2018-03-22 08:30:49 +01:00
Norman Maurer
6e6cfa0604 Add tests for EmptyHeaders
Motivation:

6e5fd9311f fixed a bug in EmptyHeaders which was never noticed before because we had no tests.

Modifications:

Add tests for EmptyHeaders.

Result:

EmptyHeaders is tested now.
2018-03-20 15:59:08 +01:00
Scott Mitchell
6e5fd9311f EmptyHeaders get with default value returns null
Motivation:
EmptyHeaders#get with a default value argument returns null. It should never return null, and instead it should return the default value.

Modifications:
- EmptyHeaders#get with a default value should return that default value

Result:
More correct implementation of the Headers API.
2018-03-19 17:54:26 +01:00
Norman Maurer
352e36a179 Remove code duplication in ChunkedWriteHandler
Motivation:

We had some code duplication in ChunkedWriteHandler.

Modifications:

Factor out duplicated code into private methods and reuse it.

Result:

Less code duplication.
2018-03-19 09:14:07 +01:00
Norman Maurer
2c90b6235d Correctly include the stream id when convert from Http2HeadersFrame to HttpMessage
Motivation:

We did not correctly set the stream id in the headers of HttpMessage when converting a Http2HeadersFrame. This is based on https://github.com/netty/netty/pull/7778 so thanks to @jprante.

Modifications:

- Correctly set the id when possible in the header.
- Add test case

Result:

Correctly include stream id.
2018-03-17 09:46:01 +01:00
Norman Maurer
0adccfdb50 Simplify DefaultChannelGroup.contains(...) and so remove one instanceof check.
Motivation:

DefaultChannelGroup.contains(...) did one more instanceof check then needed.

Modifications:

Simplify contains(...) and remove one instanceof check.

Result:

Simplier and cheaper implementation.
2018-03-17 09:45:24 +01:00
Norman Maurer
de082bf4c7 Correctly record creation stacktrace in ResourceLeakDetector.
Motivation:

We missed to correctly record the stacktrace of the creation of an ResourceLeak record. This could either have the effect to log the wrote stacktrace for creation or not log a stacktrace at all if the object was dropped on the floor after it was created.

Modifications:

Correctly create a Record on creation of the object.

Result:

Fixes https://github.com/netty/netty/issues/7781.
2018-03-16 08:24:18 +01:00
Alexey Kachayev
b0823761f4 PendingWriteQueue to handle write operations with void future
Motivation:

Right now PendingWriteQueue.removeAndWriteAll collects all promises to
PromiseCombiner instance which sets listener to each given promise throwing
IllegalStateException on VoidChannelPromise which breaks while loop
and "reports" operation as failed (when in fact part of writes might be
actually written).

Modifications:

Check if the promise is not void before adding it to the PromiseCombiner
instance.

Result:

PendingWriteQueue.removeAndWriteAll succesfully writes all pendings
even in case void promise was used.
2018-03-16 08:23:40 +01:00
Norman Maurer
f6251c8256 IovArray.add(...) should check if buffer has memory address.
Motivation:

We currently not check if the buffer has a memory address and just assume this is the case if the nioBufferCount() == 1.

Modifications:

- Check hasMemoryAddress() before trying to access it.
- Add unit case.

Result:

More correct and robust code. Related to [#7752].
2018-03-13 08:51:01 +01:00
Norman Maurer
bd772d127e FixedCompositeByteBuf should allow to access memoryAddress / array when wrap a single buffer.
Motivation:

We should allow to access the memoryAddress / array of the FixedCompositeByteBuf when it only wraps a single ByteBuf. We do the same for CompositeByteBuf.

Modifications:

- Check how many buffers FixedCompositeByteBuf wraps and depending on it delegate the access to the memoryAddress / array
- Add unit tests.

Result:

Fixes [#7752].
2018-03-13 08:50:42 +01:00
Norman Maurer
6eb9674bf5 Replace finalizer() usage in Recycler.WeakOrderQueue with ObjectCleaner usage.
Motivation:

We recently introduced ObjectCleaner which can be used to ensure some cleanup action is done once an object becomes weakable reachable. We should use this in Recycler.WeakOrderQueue to reduce the overhead of using a finalizer() (which will cause the GC to process it two times).

Modifications:

Replace finalizer() usage with ObjectCleaner

Result:

Fixes [#7343]
2018-03-09 18:45:02 -08:00
Norman Maurer
d1055e0665 Add testcase for c11b23bbc1
Motivation:

c11b23bbc1 added a fix for closing the SSLEngine otbound but no test was provided.

Modifications:

Add testcase.

Result:

More tests.
2018-03-06 14:33:54 +09:00
Carl Mastrangelo
c11b23bbc1 Close SSLEngine when connection fails.
Motivation:
When using the JdkSslEngine, the ALPN class is used keep a reference
to the engine.   In the event that the TCP connection fails, the
SSLEngine is not removed from the map, creating a memory leak.

Modification:
Always close the SSLEngine regardless of if the channel became
active.  Also, record the SSLEngine was closed in all places.

Result:
Fixes: https://github.com/grpc/grpc-java/issues/3080
2018-03-04 06:55:51 -08:00
Norman Maurer
bf8cac4939 Workaround SSLEngine.unwrap(...) bug in Android 5.0
Motivation:

Android 5.0 sometimes not correctly update the bytesConsumed of the SSLEngineResult when consuming data from the input ByteBuffer. This will lead to handshake failures.

Modifications:

Add a workaround for Android 5.0

Result:

Be able to use netty on Android 5.0 by fixing https://github.com/netty/netty/issues/7758 .
2018-03-03 15:01:39 -08:00
Norman Maurer
48df2f66b8 HashedWheelTimer.newTimeout(...) may overflow
Motivation:

We dont protect from overflow and so the timer may fire too early if a large timeout is used.

Modifications:

Add overflow guard and a test.

Result:

Fixes https://github.com/netty/netty/issues/7760.
2018-03-03 15:00:47 -08:00
Norman Maurer
0a8e1aaf19 Flush task should not flush messages that were written since last flush attempt.
Motivation:

The flush task is currently using flush() which will have the affect of have the flush traverse the whole ChannelPipeline and also flush messages that were written since we gave up flushing. This is not really correct as we should only continue to flush messages that were flushed at the point in time when the flush task was submitted for execution if the user not explicit call flush() by him/herself.

Modification:

Call *Unsafe.flush0() via the flush task which will only continue flushing messages that were marked as flushed before.

Result:

More correct behaviour when the flush task is used.
2018-03-02 10:09:40 +09:00
kakashiio
12ccd40c5a Correctly throw IndexOutOfBoundsException when writerIndex < readerIndex
Motivation:

If someone invoke writeByte(), markWriterIndex(), readByte() in order first, and then invoke resetWriterIndex() should be throw a IndexOutOfBoundsException to obey the rule that the buffer declared "0 <= readerIndex <= writerIndex <= capacity".

Modification:

Changed the code writerIndex = markedWriterIndex; into writerIndex(markedWriterIndex); to make the check affect

Result:
Throw IndexOutOfBoundsException if any invalid happened in resetWriterIndex.
2018-03-02 10:05:33 +09:00
Francesco Nigro
ed46c4ed00 Copies from read-only heap ByteBuffer to direct ByteBuf can avoid stealth ByteBuf allocation and additional copies
Motivation:

Read-only heap ByteBuffer doesn't expose array: the existent method to perform copies to direct ByteBuf involves the creation of a (maybe pooled) additional heap ByteBuf instance and copy

Modifications:

To avoid stressing the allocator with additional (and stealth) heap ByteBuf allocations is provided a method to perform copies using the (pooled) internal NIO buffer

Result:

Copies from read-only heap ByteBuffer to direct ByteBuf won't create any intermediate ByteBuf
2018-02-27 09:54:21 +09:00
Norman Maurer
a53f716ca1 Call correct super methods in test.
Motivation:

We called the wrong super method in the test and also had a few unused imports.

Modifications:

Fix super method call and cleanup.

Result:

More correct test and cleanup.
2018-02-27 09:52:32 +09:00
teaey
06dcca1dbc When the response exceeds the threshold, it will be compressed
Motivation:

When the response is very small, compression will inflate the response.

Modifications:

Add filed io.netty.handler.codec.http.HttpContentCompressor#compressThreshold that control whether the HTTP response should be compressed.

Result:

Fixes #7660.
2018-02-25 16:43:45 +01:00
Norman Maurer
69582c0b6c [maven-release-plugin] prepare for next development iteration 2018-02-21 12:52:33 +00:00
Norman Maurer
786f35c6c9 [maven-release-plugin] prepare release netty-4.1.22.Final 2018-02-21 12:52:19 +00:00
Scott Mitchell
d2d3e6ef0c
KQueue write filter initial state (#7738)
Motivation:
KQueue implementations current have inconsistent behavior with Epoll implementations with respect to asynchronous sockets and connecting. In the Epoll transport we attempt to connect, if the connect call does not synchornously fail/succeed we set the EPOLLOUT which will be triggered by the kernel if the connection attempt succeeds or an error occurs. The connect API provides no way to asynchronously communicate an error so the Epoll implementation fires a EPOLLOUT event and puts the connect status in getsockopt(SO_ERROR). KQueue provides the same APIs but different behavior. If the EVFILT_WRITE is not enabled and the EVFILT_READ is enabled before connect is called, and there is an error the kernel may fire the EVFILT_READ filter and provide the Connection Refused error via read(). This is even true if we set the EVFILT_WRITE filter after calling connect because connect didn't synchornously complete. After the error has been delievered via read() a call to getsockopt(SO_ERROR) will return 0 indicating there is no error. This means we cannot rely upon the KQueue based kernel to deliver connection errors via the EVFILT_WRITE filter in the same way that the linux kernel does with the EPOLLOUT flag.
ce241bd introduced a change which depends upon the behavior of the EVFILT_WRITE being set and may prematurely stop writing to the OS as a result, becaues we assume the OS will notify us when the socket is writable. However the current work around for the above described behavior is to initialize the EVFILT_WRITE to true for connection oriented protocols. This leads to prematurely exiting from the flush() which may lead to deadlock.

Modifications:
- KQueue should check when an error is obtained from read() if the connectPromise has not yet been completed, and if not complete it with a ConnectException

Result:
No more deadlock in KQueue due to asynchronous connect workaround.
2018-02-20 11:01:49 -08:00
Norman Maurer
c6c0984b8e Ensure we always release the AddressEnvelope when doing DNS queries.
Motivation:

When we do DNS queries we need to ensure we always release the AddressEnvelope.

Modifications:

Also release the AddressEnvelope if the original resolution was done in the meantime and we did not cancel the extra query yet.

Result:

Should fix [#7713]
2018-02-20 12:06:21 +01:00
David Nault
f40ecc3f10 Fix Snappy decoding of large 2-byte literal lengths and copy offsets
Motivation:

The Snappy decoder was failing on valid inputs containing literals
with 2-byte lengths > 0x8000 or copies with 2-byte offsets >= 0x8000.

The decoder was also enforcing an artificially low offset limit of
0x7FFF, something the Snappy format description advises against,
and which prevents decoding valid inputs generated by other encoders.

Modifications:

Interpret 2-byte literal lengths and 2-byte copy offsets as unsigned
shorts, in accordance with the format description and reference
implementation.

Allow any positive offset value. Throw an appropriate exception
for negative values (which can theoretically occur due to arithmetic
overflow on 4-byte offsets, but are unlikely to occur in the wild).

Result:

The Snappy decoder can handle valid inputs that previously caused
it to throw exceptions.
2018-02-20 11:42:23 +01:00
Scott Mitchell
ce241bd11e Epoll flush/writabilityChange deadlock
Motivation:
b215794de3 recently introduced a change in behavior where writeSpinCount provided a limit for how many write operations were attempted per flush operation. However when the write quantum was meet the selector write flag was not cleared, and the channel unsafe flush0 method has an optimization which prematurely exits if the write flag is set. This may lead to no write progress being made under the following scenario:
- flush is called, but the socket can't accept all data, we set the write flag
- the selector wakes us up because the socket is writable, we write data and use the writeSpinCount quantum
- we then schedule a flush() on the EventLoop to execute later, however it the flush0 optimization prematurely exits because the write flag is still set

In this scenario the socket is still writable so the EventLoop may never notify us that the socket is writable, and therefore we may never attempt to flush data to the OS.

Modifications:
- When the writeSpinCount quantum is exceeded we should clear the selector write flag

Result:
Fixes https://github.com/netty/netty/issues/7729
2018-02-20 11:40:58 +01:00
Scott Mitchell
1e5fafe446 Simplify CharSequenceValueConverter#convertToBoolean
Motivation:
CharSequenceValueConverter#convertToBoolean has a few manual conditionals which can be removed if we use AsciiString.contentEqualsIgnoreCase. Also by comparing an AsciiString to a String we will incur conversions to char that can be avoided if we compare against AsciiString.

Modifications:
- Use AsciiString.contentEqualsIgnoreCase
- Compare against a AsciiString

Result:
Simplified CharSequenceValueConverter#convertToBoolean which favors AsciiString comparison.
2018-02-17 07:47:44 +01:00
Carl Mastrangelo
15560530d4 Propagate full Unsafe unavailability reason in PlatformDependent
Motivation:
It is not clear why Unsafe is unavailable when it is explicitly
disabled, or when Netty thinks it is running on Android.

Modification:
Change the "has" fields and methods to be causes.  A null cause
means Unsafe is present.  This catches all possible reason why
Unsafe might not be available.

Result:
Easier to debug Netty start up when logging cannot be turned on.
2018-02-16 10:43:06 -08:00
Norman Maurer
268b901844 SSL connection not closed properly after handshake failure
Motivation:

When SSL handshake fails, the connection should be closed. This is not true anymore after 978a46c.

Modifications:

- Ensure we always flush and close the channel on handshake failure.
- Add testcase.

Result:

Fixes [#7724].
2018-02-16 08:25:47 -08:00
Francesco Nigro
bc8e022601 Added exact utf8 length estimator and exposed writeUtf8 with custom space reservation on destination buffer
Motivation:

To avoid eager allocation of the destination and to perform length prefixed encoding of UTF-8 string with forward only access pattern

Modifications:

The original writeUtf8 is modified by allowing customization of the reserved bytes on the destination buffer and is introduced an exact UTF-8 length estimator.

Result:

Is now possible to perform length first encoding with UTF-8 well-formed char sequences following a forward only write access pattern on the destination buffer.
2018-02-16 11:52:35 +01:00
Norman Maurer
dc3036a202 Update jetty-alpn-agent for latest java8 release.
Motivation:

We need to update jetty-alpn-agent to support java 1.8.0_162 while running our tests / examples.

Modifications:

Update jetty-alpn-agent to 2.0.7

Result:

All tests alpn related tests work again on latest java8 version
2018-02-16 09:18:22 +01:00
Carl Mastrangelo
e00f24961a Get memory address from Unsafe for OpenSSL
Motivation:
Profiling tcnative SSL code showed a non trivial percentage (1%)
of time spent in JNI code for InstaceOf.   This turned out to be
from `Buffer.address` which makes a JNI call, which safely checks
on each call that The ByteBuffer is direct.

Modification:
Prefer using the address field of the pojo rather than looking it
up with JNI.  This is the same approach taken by the `OpenSsl`
class.

Result:
Less JNI overhead
2018-02-16 07:57:43 +01:00
Norman Maurer
756854e99a Correctly implement CharSequenceValueConvert.convertTimeMillis
Motivation:

If you pass the output of CharSequenceValueConvert.convertToTimeMillis to convertTimeMillis it will throw a ParseException.

Modifications:

- Correctly implement CharSequenceValueConverter.convertTimeMillis
- Add unit-tests for CharSequenceValueConverter

Result:

Correctly convert timemillis.
2018-02-16 07:44:13 +01:00
ryu1-sakai
c1d0d88f0a Implement DefaultHeaders.HeaderEntry.equals()
Motivation:

HeaderEntry.equals() inherets Object.equals() which simply check if two objects are the same.
So it returns false even when two HeaderEntry objects have the same name and value.

Modifications:

Implement HeaderEntry.equals() that follows the specification of Map.Entry.equals().
https://docs.oracle.com/javase/9/docs/api/java/util/Map.Entry.html#equals-java.lang.Object-

Result:

HeaderEntry.equals() returns true if two HeaderEntry objects have the same name and value.
2018-02-15 13:07:03 +01:00
Norman Maurer
ad6af3250c DefaultHeaders / CharSequenceValueConverter should treat boolean consistently.
Motivation:

HttpHeaders.getBoolean should return the same truth value for the same string value, regardless of the underlying type.

Modifications:

- Only treat values of true as Boolean.TRUE
- Add unit tests.

Result:

Consistent converting of values for all CharSequence implementations.
2018-02-15 08:37:46 +01:00
Shohei Kamimori
73f23c5faa Fix typos in docs.
Motivation:

There are same typos in the docs.

Modifications:

Fix typos. Docs only changing.

Result:

More correct docs.
2018-02-14 08:44:07 +01:00
shorea
650406c0a3 Http2MultiplexCodec now propagates SETTINGS and GOAWAY frames in pipeline.
Motivation:

Allow the observation of SETTINGS frame by other handlers in the pipeline. For my particular use case this allows me to observe the value of MAX_CONCURRENT_STREAMS for a ChannelPool abstraction that supports HTTP/2 multiplexing. Beside this also forward GOAWAY frames.

Modification:

Always forward SETTINGS and GOAWAY frames

Result:

Settings / Goaway can now be observed in the parent channel. Previously it was not possible (to my knowledge) to capture the settings when using Http2MultiplexCodec.
2018-02-14 08:39:01 +01:00
Norman Maurer
02b7507a62 Correctly handle the case when converting of value fails and return null or default value.
Motivation:

Headers.get* methods should not throw an exception but return null or the default value if converting of the value fails.

Modifications:

- Correctly handle the case when ValueConverter throws an Exception.
- Add testcase.

Result:

Fixes [#7710].
2018-02-14 08:35:55 +01:00
Scott Mitchell
4f982be91e
DefaultPromise internal state dependent on Signal
Motivation:
DefaultPromise's internal state depends upon specific Signal objects. These Signal objects can be used externally which causes the DefaultPromise object API to not function correct and state to become corrupted.

Modifications:
- DefaultPromise shouldn't depend upon Signal for its internal state

Result:
Fixes https://github.com/netty/netty/issues/7707
2018-02-12 14:24:46 -08:00
Scott Mitchell
108fbe5282
ByteBufUtil to not pool direct memory by default
Motivation:
ByteBufUtil by default will cache DirectByteBuffer objects, and the
associated direct memory (up to 64k). In combination with the Recycler which may
cache up to 32k elements per thread may lead to a large amount of direct
memory being retained per EventLoop thread. As traffic spikes come this
may be perceived as a memory leak because the memory in the Recycler
will never be reclaimed.

Modifications:
- By default we shouldn't cache DirectByteBuffer objects.

Result:
Less direct memory consumption due to caching DirectByteBuffer objects.
2018-02-12 10:49:17 -08:00
Scott Mitchell
7928a6fef3
NioDatagramChannel invalid usage of internalNioBuffer
Motivation:
NioDatagramChannel attempts to unpack a AddressedEnvelope and unconditionally uses internalNioBuffer. However if the ByteBuf is a CompositeByteBuf with more than 1 components, the write will fail and throw an exception.

Modifications:
- NioDatagramChannel should check the nioBufferCount before attempting
to use internalNioBuffer

Result:
No more failure to write UDP packets on NIO when a CompositeByteBuf is
used.
2018-02-12 09:31:36 -08:00
Eric Anderson
8b273983f0 Load Conscrypt method via reflection only once
Motivation:

The code did reflection every method call which made the code slower and
harder to read with additional cases to consider.

Modifications:

Instead of loading the method and then throwing it away, save the Method
reference instead of the Class reference. Then also use more precise
exception handling for the method invocation.

Result:

Simpler, speedier code.
2018-02-09 21:13:42 +01:00
Scott Mitchell
6cd5e8b0ca Reduce the default number of objects retained by the Recycler per thread
Motivation:
The Recycler currently retains 32k objects per thread by default. The Recycler is used in more than just one place and may result in large amounts of memory bloat if spikes of traffic are observed.

Modifications:
- Reduce the Recyclers default capacity from 32k to 4k.

Result:
- Lower default capacity of the Recycler and less memory retained.
2018-02-09 19:56:01 +01:00
Eric Anderson
0b0418b368 Bump Conscrypt version to 1.0.0
Motivation:

Conscrypt is now 1.0. No more need to depend on release candidates.

Modifications:

Just the version bump. Things seemed compatible.

Result:

Depending on first guaranteed-api-stable release of Conscrypt.
2018-02-09 19:55:03 +01:00
Johno Crawford
01e46ed03a System property util might return null
Motivation:

isAndroid0 should be robust.

Modifications:

yoda equals for string comparison.

Result:

No NPE.
2018-02-09 19:25:30 +01:00
Carl Mastrangelo
4ed961f4fe To detect Android, check the VM property rather than the classpath
Motivation:
Some java binaries include android classes on their classpath, even
if they aren't actually android.  When this is true, `Unsafe` no
longer works, disabling the Epoll functionality.  A sample case is
for binaries that use the j2objc library.

Modifications:
Check the `java.vm.name` instead of the classpath.   Numerous
Google-internal Android libraries / binaries check this property
rather than the class path.

It is believed this is safe and works with bother ART and Dalvik
VMs, safe for Robolectric, and j2objc.

Results:
Unusually built java server binaries can still use Netty Epoll.
2018-02-09 15:43:25 +01:00
Norman Maurer
501662a77f Use long for http2 ping payload.
Motivation:

At the moment we use a ByteBuf as the payload for a http2 frame. This complicates life-time management a lot with no real gain and also may produce more objects then needed. We should just use a long as it is required to be 8 bytes anyway.

Modifications:

Use long for ping payloads.

Result:

Fixes [#7629].
2018-02-08 10:23:34 +01:00