Commit Graph

10113 Commits

Author SHA1 Message Date
jrhee17
37948bc9de
Add support for HAProxyMessageEncoder (#10175)
Motivation:

Add support for HAProxyMessageEncoder.
This should help java based HAProxy server implementations propagate proxy information.

Modification:

Add public constructors for `HAProxyMessage`, `HAProxyTLV`, `HAProxySSLTLV`.
Add additional argument checks for `HAProxyMessage` and modify exceptions thrown when creating via public constructors directly.
Introduce a `@Sharable` `HAProxyMessageEncoder` which encodes a `HAProxyMessage` into a byte array.
Add an example `HAProxyServer` and `HAProxyClient` to `io.netty.example`


Result:

Fixes #10164
2020-04-16 09:35:06 +02:00
Norman Maurer
2c78b4c84f
Correctly propagate exceptions from inbound operations in all cases (#10176)
Motivation:

In AbstractChannelHandlerContext we had some code where we tried to guard against endless loops caused by exceptions thrown by exceptionCaught(...) that would trigger exceptionCaught again. This code was proplematic for two reasons:
 - It is quite expensive as we need to compare all the stacks
 - We may end up not notify another handlers exceptionCaught(...) if in our exeuction stack we triggered actions that will cause an exceptionCaught somewhere else in the pipeline

Modifications:

- Just remove the detection code as we already handle everything correctly when we invoke exceptionCaught(...)
- Add unit tests

Result:

Ensure we always notify correctly and also fixes performance issue reported as https://github.com/netty/netty/issues/10165
2020-04-16 08:56:45 +02:00
Norman Maurer
6dad12defa
Add workaround for possible classloader deadlock when trying to load JNI code (#10190)
Motivation:

netty_epoll_linuxsocket_JNI_OnLoad(...) may produce a deadlock with another thread that will load IOUtil in a static block. This seems to be a JDK bug which is not yet fixed. To workaround this we force IOUtil to be loaded from without java code before init the JNI code

Modifications:

Use Selector.open() as a workaround to load IOUtil

Result:

Fixes https://github.com/netty/netty/issues/10187
2020-04-16 08:40:59 +02:00
Linas Medžiūnas
fb5e2cd3aa
Efficient BytBuf search algorithms (#9914) (#9955)
Motivation:

We have found out that ByteBufUtil.indexOf can be inefficient for substring search on
ByteBuf, both in terms of algorithm complexity (worst case O(needle.readableBytes *
haystack.readableBytes)), and in constant factor (esp. on Composite buffers).
With implementation of more performant search algorithms we have seen improvements on
the order of magnitude.

Modifications:

This change introduces three search algorithms:
1. Knuth Morris Pratt - classical textbook algorithm, a good default choice.
2. Bit mask based algorithm - stable performance on any input, but limited to maximum
search substring (the needle) length of 64 bytes.
3. Aho–Corasick - worse performance and higher memory consumption than [1] and [2], but
it supports multiple substring (the needles) search simultaneously, by inspecting every
byte of the haystack only once.

Each algorithm processes every byte of underlying buffer only once, they are implemented
as ByteProcessor.

Result:

Efficient search algorithms with linear time complexity available in Netty (I will share
benchmark results in a comment on a PR).
2020-04-15 10:21:24 +02:00
时无两丶
ff6d2e4949
Remove duplicate code in PoolArena. (#10174)
Motivation:

Remove duplicate code in PoolArena.

Modification:

Replace duplicate code with `tinyIdx` and `smallIdx`.

Result:

Clean code.
2020-04-15 09:29:23 +02:00
feijermu
17f205fab3
Close the FileChannel in case of an IOException in AbstractDiskHttpData.addContent. (#10188)
Motivation:

`FileChannel.force` may throw an IOException. A fd leak may happen here.

Modification:

Close the fileChannel in a finally block.

Result:

Avoid fd leak.
2020-04-15 09:24:26 +02:00
Norman Maurer
4c9d7492fb
CodecOutputList should be recycled in a finally block (#10186)
Motivation:

To ensure we always recycle the CodecOutputList we should better do it in a finally block

Modifications:

Call CodecOutputList.recycle() in finally

Result:

Less chances of non-recycled lists. Related to https://github.com/netty/netty/issues/10183
2020-04-15 09:08:19 +02:00
feijermu
1b3e1985b4
Release the channel attribute--REOPEN_TASK after removing the TrafficShapingHandler from channel pipeline. (#10177)
Motivation:

The `AbstractTrafficShapingHandler` caches the `ReopenReadTimerTask` instance in the channel attribute. However, if this handler is removed from the channel pipeline, this `ReopenReadTimerTask` instance may not be released.

Modification:

Release the channel attribute `REOPEN_TASK` in `handlerRemoved` method.

Result:

Avoid a channel attribute leak.
2020-04-15 08:52:08 +02:00
Cenjie Ho
2d4d64dc42
Add the unit test of ChannelOutboundBuffer (#10180)
Motivation:

Parameter maxCount needs the unit test.

Modifications:

1. Change the conditional statement to avoid the ineffective maxCount (enhance the robustness of the code merely).
2. Add the unit test for maxCount.

Result:

Enable this parameter to be tested.
2020-04-15 08:22:51 +02:00
Norman Maurer
b996bf2151
Update Bouncycastle dependency (#10185)
Motivation:

We should update our optional bouncycastle dependency to ensure we use the latest which has all the security fixes

Modifications:

Update bouncycastle version

Result:

Fixes https://github.com/netty/netty/issues/10184
2020-04-14 14:23:45 +02:00
Stephane Landelle
d5e42f8553
Log protocol version during handshake (#10178)
Motivation:

Only cipher suite is logged during handshake. Picked protocol is interesting too.

Modification:

Log protocol as well.

Result:

More interesting information when debugging TLS handshakes.
2020-04-14 10:48:43 +02:00
Sérgio Santos
79ef0c4706
Allow Conscrypt to be used on Android (#10182)
**Motivation:**

Following up on https://github.com/netty/netty/issues/10181

I was able to successfully setup a Netty server with TLS on Android, using Conscrypt, with the change proposed here. Conscrypt publishes an [Android specific version](https://github.com/google/conscrypt#android). But the current availability check prevents Netty from using it (the call `PlatformDependent.javaVersion()` returns `6` on Android).

**Modification:**

Check whether the java version is above 8, or if it's running on an Android device, to flag Conscrypt as available (besides the remaining checks).

**Result:**

Netty with TLS runs fine on Android 👍
2020-04-14 10:42:51 +02:00
Andrey Mizurov
2b14775446
Stomp over WebSocket Chat example (#10152)
Motivation:

Often people want to use `stomp-codec` with WebSocket transport or other but cannot figure out how can do this staff on Netty.

Modification:

Create example for demonstrating integration between STOMP and WebSocket.
Inspired by https://github.com/jmesnil/stomp-websocket

Result:

Fixes #9383
2020-04-08 12:04:48 +02:00
minux
7d7e1f5ed7
Stop sending DNS queries if promise is cancelled (#10171)
Motivation:
A user might want to cancel DNS resolution when it takes too long.
Currently, there's no way to cancel the internal DNS queries especially when there's a lot of search domains.

Modification:
- Stop sending a DNS query if the original `Promise`, which was passed calling `resolve()`, is canceled.

Result:
- You can now stop sending DNS queries by cancelling the `Promise`.
2020-04-07 12:25:02 +02:00
minux
a4c96483d1
Fix a bug where making IPv6 DnsQuestion when it's not supported (#10170)
Motivation:
Related https://github.com/line/armeria/issues/2463
Here is an example that an NIC has only link local address for IPv6.
```
$ ipaddr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if18692: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1460 qdisc noqueue
    link/ether 1a:5e:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 10.xxx.xxx.xxx/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:xxxx/64 scope link
       valid_lft forever preferred_lft forever
```
If the NICs have only local or link local addresses, We should not send IPv6 DNS queris.

Modification:
- Ignore link-local IPv6 addresses which may exist even on a machine without IPv6 network.

Result:
- `DnsNameResolver` does not send DNS queries for AAAA when IPv6 is not available.
2020-04-07 12:08:04 +02:00
feijermu
3ebf3d7705
Close the file in case of an IOException in AbstractMemoryHttpData.renameTo. (#10163)
Motivation:

An `IOException` may be thrown from `FileChannel.write` or `FileChannel.force`, and cause the fd leak.

Modification:

Close the file in a finally block.

Result:

Avoid fd leak.
2020-04-06 10:12:46 +02:00
Cenjie Ho
363c4ecc7f
Add the missing comment (#10169)
Motivation:

The comment of SingleThreadEventExecutor#run() is missing

Modification:

Just fill it in.

Result:

Method comment is not blank now.
2020-04-06 09:05:29 +02:00
Dmitry Konstantinov
05b7af7ae4
Propagate ref to pool thread cache down in the allocation stack to avoid extra thread local lookup (#10166)
Motivation:
PoolChunk requires a link to a PoolThreadCache to init ByteBuf. Currently the link is retrieved from a thread local: arena.parent.threadCache().
It has some performance cost. At the beginning of the allocation call the PoolThreadCache is already retrieved from the thread local. The reference can be propagated through the calls and used.

Modifications:
Replace second lookup of PoolThreadCache during ByteBuf init by propagation of a reference to PoolThreadCache down in the allocation stack explicitly

Result:
Improve performance of ByteBuf allocation
--Before--
Benchmark                                            (size)  (tokens)  (useThreadCache)  Mode  Cnt    Score   Error  Units
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     123         0              true  avgt   20   57.112 ± 1.004  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     123       100              true  avgt   20  222.827 ± 1.307  ns/op

--After--
Benchmark                                            (size)  (tokens)  (useThreadCache)  Mode  Cnt    Score   Error  Units
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     123         0              true  avgt   20   50.732 ± 1.321  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     123       100              true  avgt   20  216.892 ± 3.806  ns/op
2020-04-06 08:01:01 +02:00
feijermu
bdaa935756
Release the temporary ByteBuf in case of unexpected exception in WebSocketUtil.base64. (#10160)
Motivation:

An exception may occur between ByteBuf's allocation and release. So I think it's a good idea to place the release operation in a finally block.

Modification:

Release the temporary ByteBuf in finally blocks.

Result:

Avoid ByteBuf leak.
2020-04-03 14:49:40 +02:00
Norman Maurer
c90acf4766
Hardcode TLS version used during blockhound tests (#10162)
Motivation:

Different versions of the JDK use different TLS versions by default. We should define the versions explicit

Modifications:

Explicit specify TLSv1.2

Result:

Blockhound tests pass on JDK14 as well
2020-04-03 14:40:15 +02:00
Norman Maurer
21f375fa90
SslHandler should fail handshake / close promise and notify pipeline on removal (#10161)
Motivation:

If the SslHandler is removed from the pipeline we also need to ensure we fail the handshake / close promise if it was not notified before as otherwise we may never do so.

Modifications:

- Correctly fail promise and notify pipeline if handshake was not done yet when the SslHandler is removed
- Add unit test

Result:

Fix https://github.com/netty/netty/issues/10158
2020-04-03 08:45:48 +02:00
Norman Maurer
db39b10249 Update link for JDK14 regression to point to the actual bugreport 2020-04-02 16:00:06 +02:00
feijermu
d4c268ae3e
Close the file when IOException occurs in AbstractMemoryHttpData. (#10157)
Motivation:

An IOException may be thrown from FileChannel.read, and cause the fd leak.

Modification:

Close the file when IOException occurs.

Result:
Avoid fd leak.
2020-04-02 14:16:01 +02:00
Dmitry Konstantinov
ea31b59037
Replace usage() with freeBytes() in thresholds within hot paths of PoolChunkList (#10141)
Motivation:
PoolChunk.usage() method has non-trivial computations. It is used currently in hot path methods invoked when an allocation and de-allocation are happened.
The idea is to replace usage() output comparison against percent thresholds by Chunk.freeBytes plain comparison against absolute thresholds. In such way the majority of computations from the threshold conditions are moved to init logic.

Modifications:
Replace PoolChunk.usage() conditions in PoolChunkList with equivalent conditions for PoolChunk.freeBytes()

Result:
Improve performance of allocation and de-allocation of ByteBuf from normal size cache pool
2020-03-31 22:11:16 +02:00
Norman Maurer
4b235a9600
Introduce DomainWildcardMappingBuilder to fix wildcard matching accor… (#10132)
Motivation:

How we did wildcard matching was not correct according to RFC6125. Beside this our implementation was quite CPU heavy.

Modifications:

- Add new DomainWildcardMappingBuilder which correctly does wildcard matching. See https://tools.ietf.org/search/rfc6125#section-6.4
- Add unit tests
- Deprecate old implementations

Result:

Correctly implement wildcard matching and improve performance
2020-03-31 16:57:42 +02:00
Andrey Mizurov
7309f2a13d
Fix allocate additional buffer for encoding stompFrame without readab… (#10150)
Motivation:

Not always STOMP frames contain any payload some times it just headers. So we wan't allocate additional buffer with NULL content for this situation.

Modification:

Modify StompSubframeEncoder to check if content is readable or not. If content is not readable just add NULL byte to encoded header buffer.

Result:

Less allocations
2020-03-31 16:02:06 +02:00
Norman Maurer
fbc6709686
Add profile to build on JDK 14 (#10148)
Motivation:

JDK 14 was released and need some special settings to be able to build with. Also there seems to be one regression that we need to workaround for now.

Modifications:

- Add maven profile for JDK 14
- Update blockhound version to be able to work on JDK 14
- Add workaround for possible JDK 14 regression

Result:

Be able to build on JDK 14
2020-03-30 21:25:08 +02:00
luffyke
7e823e3842
Replace MQTT deprecated API usage
Motivation:

Mqtt procotol defines packetId instead of messageId, replace deprecated method to match protocol definition

Modification:

Replace messageId() to packetId() in MqttEncoder

Result:

Match mqtt protocol definition and improve readability.
2020-03-30 21:23:23 +02:00
Romain Manni-Bucau
84ebf47515
making DefaultHttpDataFactory able to configure basedir and deleteonexit (#10146)
Motivation: 

currently (http) disk based attributes or uploads are globally configured in a single directory and can also only globally be deleted on exit or not. it does not fit well multiple cases, in particular the case you have multiple servers in the same JVM.

Modification: 
make it configurable per attribute/fileupload.

Result:

This PR duplicates Disk* constructor to add basedir and deleteonexit parameters and wires it in default http daa factory.
2020-03-30 13:11:46 +02:00
feijermu
aed5a74e09
Release the ByteBuf when IOException occurs in AbstractMemoryHttpData. (#10133)
Motivation:

An IOException may be thrown from InputStream.read or checkSize method, and cause the ByteBuf leak.

Modification:

Release the ByteBuf when IOException occurs.

Result:
Avoid ByteBuf leak.
2020-03-30 11:26:03 +02:00
Norman Maurer
81513c3728
Add JDK 14 and update JDK 13 for builds (#10136)
Motivation:

JDK 14 was released so we should include it in our build matrix. Beside this there was also a JDK 13 update

Modifications:

- Add docker-compose files for JDK 14
- Update JDK 13 version

Result:

Build with JDK 14 as well and use latest JDK 13 release
2020-03-26 10:26:20 +01:00
Norman Maurer
62e3c463c4
Don't produce multiple calls to exceptionCaught(...) on SSL handshake failure (#10134)
Motivation:

Before release 4.1.23, there was only ONE call to exceptionCaught method when an ssl handshake failure occurs, now we have two when using the JDK provider.

Modifications:

- Ensure we only propagate one exception fi we already failed the handshake and channelInactive(...) produce an exception again
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/10119
2020-03-26 09:12:39 +01:00
feijermu
0c3eae34ec
fix a potential ByteBuf leak in HttpProxyHandler. (#10130)
Motivation:

An exception may occur between ByteBuf's allocation and release. For example:
```java
java.lang.OutOfMemoryError: Java heap space
	at java.lang.String.<init>(String.java:325)
	at io.netty.buffer.ByteBufUtil.decodeString(ByteBufUtil.java:838)
	at io.netty.buffer.AbstractByteBuf.toString(AbstractByteBuf.java:1247)
	at io.netty.buffer.AbstractByteBuf.toString(AbstractByteBuf.java:1242)
	at io.netty.handler.proxy.HttpProxyHandler.<init>(HttpProxyHandler.java:105)
	at io.netty.handler.proxy.HttpProxyHandler.<init>(HttpProxyHandler.java:90)
	at io.netty.handler.proxy.HttpProxyHandler.<init>(HttpProxyHandler.java:85)
```
It may cause the ByteBuf variable authz and authzBase64's leak.

Modification:

Release the ByteBuf in a finally block as soon as possible.

Result:

Fix a potential ByteBuf leak.
2020-03-24 11:17:40 +01:00
Dmitry Konstantinov
913cae6fef
Optimize log2 in PoolThreadCache (#10128)
Motivation:
The current implementation of log2 in PoolThreadCache uses a loop and less efficient than an version based on Integer.numberOfLeadingZeros (intrinsic).

Modifications:
Replace the current log2 implementation in PoolThreadCache with a version based on Integer.numberOfLeadingZeros

Result:
It can improve performance slightly during allocation and de-allocation of ByteBuf using pooled allocator.
2020-03-24 11:13:52 +01:00
水木今山
4d7d249cc3
Fix a mistake in javadoc of ReplayingDecoder (#10129)
Motivation:

In the code example of ReplayingDecoder, an input parameter List<Object> out is missing.

Modification:

Just add this parameter.

Result:

The right doc.
2020-03-23 13:09:21 +01:00
Stephane Landelle
fcf7144f7c
Add some HTTP header constants (#10127)
Motivation:

Add some missing HTTP header names and values constants.

Modification:

* names:
  * dnt (Do Not Track)
  * upgrade-insecure-requests
  * x-requested-with
* values:
  * application/xhtml+xml
  * application/xml
  * text/css
  * text/html
  * text/event-stream
  * XmlHttpRequest

Result:

More constants available
2020-03-23 13:05:58 +01:00
Norman Maurer
99aca1879f
Update to netty-tcnative 2.0.30.Final to fix small memory leak (#10118)
Motivation:

A new netty-tcnative version was just released which fixes a small memory leak.

Modifications:

Update to 2.0.30.Final

Result:

Small memory leak fixed
2020-03-19 12:47:46 +01:00
root
14e4afeba2 [maven-release-plugin] prepare for next development iteration 2020-03-17 09:20:54 +00:00
root
c10c697e5b [maven-release-plugin] prepare release netty-4.1.48.Final 2020-03-17 09:18:28 +00:00
Norman Maurer
0fb58d3c54
HttpObjectDecoder should limit the number of control chars (#10112)
Motivation:

At the moment HttpObjectDecoder does not limit the number of controls chars. These should be counted with the initial line and so a limit should be exposed

Modifications:

- Change LineParser to also be used when skipping control chars and so enforce a limit
- Add various tests to ensure that limit is enforced

Result:

Fixes https://github.com/netty/netty/issues/10111
2020-03-17 10:16:45 +01:00
Bo Zhang
286f14f04a
Close Channel upon exceptions during bootstrap to avoid leaking (#10110)
Motivation:

We need to ensure we always close the Channel when we see an exception during bootstrapping it.

Modification:

Ensure we correct close the Channel if we see an exception during retrieving the Resolver from the group while bootstrapping.

Result:

Fixes #10109


Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2020-03-16 13:37:29 +01:00
Norman Maurer
c74469bc6c
Ensure WebSocket*Handshaker can not corrupt pipeline when HttpProxyHa… (#10103)
Motivation:

HttpProxyHandler itself will add a HttpClientCodec into the ChannelPipeline and so confuse the WebSocket*Handshaker when it tries to modify the pipeline as it will replace the wrong HttpClientCodec.

Modifications:

Wrap the internal HttpClientCodec that is added by HttpProxyHandler so it will not be replaced by HttpProxyHandler.

Result:

Fixes  https://github.com/netty/netty/issues/5201 and https://github.com/netty/netty/issues/5070
2020-03-16 11:38:41 +01:00
feijermu
427a641299
Replace oio's default thread factory with netty's DefaultThreadFactory. (#10107)
Motivation:

Netty's DefaultThreadFactory that creates FastThreadLocalThread instance is widely used in NioEventLoopGroup, EpollEventLoopGroup, etc, but not OioEventLoopGroup. Although oio is quite stale, I still think this change may be useful.

Modification:

Replace oio's default thread factory with netty's DefaultThreadFactory just like NioEventLoopGroup, EpollEventLoopGroup, etc.

Result:

Faster access to FastThreadLocal in oio.
2020-03-16 11:37:29 +01:00
feijermu
679c606f13
Add test case for Http2StreamChannelId. (#10108)
Motivation:

Http2StreamChannelId is Serializable. A test case is needed to ensure that it works.

Modification:

Add a test case about serialization.

Result:

Improve test coverage slightly.
2020-03-16 09:31:38 +01:00
Norman Maurer
4b22d8ab35
Use WebSocketVersion.toAsciiString() as header value when possible (#10105)
Motivation:

In our WebSocketClientHandshaker* implementations we "hardcode" the version number to use. This is error-prone, we should better use the WebSocketVersion so we dont need to maintain the value multiple times. Beside this we can also use an AsciiString to improve performance

Modifications:

- Use WebSocketVersion.toAsciiString

Result:

Less stuff to maintain and small performance win
2020-03-13 07:48:58 +01:00
Norman Maurer
260540b25a
Don't override HOST header if provided by user already in WebSocketClientHandshaker (#10104)
Motivation:

The user may need to provide a specific HOST header. We should not override it when specified during handshake.

Modifications:

Check if a custom HOST header is already provided by the user and if so dont override it

Result:

Fixes https://github.com/netty/netty/issues/10101
2020-03-12 20:25:53 +01:00
David Latorre
e4af5c3631
Added support for the SameSite attribute in Cookies (#10050)
Motivation:

Netty currently does not support the SameSite attribute for response cookies (see issue #8161 for discussion).

Modifications:

The attribute has been added to the DefaultCookie class as a quick fix since adding new methods to the Cookie interface would be backwards-incompatible.
ServerCookieEncoder and ClientCookieDecoder have been updated accordingly to process this value. No validation for allowed values (Lax, None, Strict) has been implemented.

Result:

Response cookies with the SameSite attribute set can be read or written by Netty.

Co-authored-by: David Latorre <a-dlatorre@hotels.com>
2020-03-12 09:48:30 +01:00
时无两丶
60cbe8b7b2
Let LzfEncoder support length aware ability. (#10082)
Motivation:

Since the LZF support non-compress and compress format, we can let LzfEncoder support length aware ability. It can let the user control compress.

Modification:

When the data length over compressThreshold, LzfEncoder use compress format to compress data. Otherwise, only use non-compress format. Whatever compress format the encoder use, the LzfDecoder can decompress data well.

Result:

Gives users control over compression capabilities
2020-03-11 21:05:22 +01:00
Stephane Landelle
2576a2dd74
Fix WebSocketClientHandshaker not generating correct handshake request when path is empty (#10095)
Motivation:

WebSocketClientHandshaker#upgradeUrl doesn't comperly compute relative url when path is empty and produces url such as `?access_token=foo` instead of `/?access_token=foo`.

Modifications:

* fix WebSocketClientHandshaker#upgradeUrl
* add tests for urls without path, with and without query

Result:

WebSocketClientHandshaker properly connects to url without path.
2020-03-10 15:17:22 +01:00
root
c623a50d19 [maven-release-plugin] prepare for next development iteration 2020-03-09 12:13:56 +00:00