Commit Graph

1401 Commits

Author SHA1 Message Date
Norman Maurer
d58d1add34 Fix checkstyle errors introduced by 33de96f448 2020-10-24 14:50:00 +02:00
Andrey Mizurov
33de96f448
Provide new client and server websocket handshake exceptions (#10646)
Motivation:

At the moment we have only one base `WebSocketHandshakeException` for handling WebSocket upgrade issues.
Unfortunately, this message contains only a string message about the cause of the failure, which is inconvenient in handling.

Modification:

Provide new `WebSocketClientHandshakeException` with `HttpResponse` field  and `WebSocketServerHandshakeException` with `HttpRequest` field both of them without content for avoid reference counting 
problems. 

Result:

More information for more flexible handling.

Fixes #10277 #4528 #10639.
2020-10-24 14:41:11 +02:00
Artem Smotrakov
e5951d46fc
Enable nohttp check during the build (#10708)
Motivation:

HTTP is a plaintext protocol which means that someone may be able
to eavesdrop the data. To prevent this, HTTPS should be used whenever
possible. However, maintaining using https:// in all URLs may be
difficult. The nohttp tool can help here. The tool scans all the files
in a repository and reports where http:// is used.

Modifications:

- Added nohttp (via checkstyle) into the build process.
- Suppressed findings for the websites
  that don't support HTTPS or that are not reachable

Result:

- Prevent using HTTP in the future.
- Encourage users to use HTTPS when they follow the links they found in
  the code.
2020-10-23 14:44:18 +02:00
Stuart Douglas
303baf5c0b
Minor performance improvement in websocket upgrade (#10710)
Motivation:

I noticed WebSocketServerExtensionHandler taking up a non-trivial
amount of CPU time for a non-websocket based menchmark. This attempts
to speed it up.

Modifications:

- It is faster to check for a 101 response than to look at headers,
so an initial response code check is done
- Move all the actual upgrade code into its own method to increase
chance of this method being inlined
- Add an extra contains() check for the upgrade header, to avoid
allocating an iterator if there is no upgrade header

Result:

A small but noticable performance increase.

Signed-off-by: Stuart Douglas <stuart.w.douglas@gmail.com>
2020-10-21 12:09:32 +02:00
Norman Maurer
ffbddcd842
Replace deprecated Assert.assertThat(...) with MatcherAssert.assertThat(...) (#10699)
Motivation:

junit deprecated Assert.assertThat(...)

Modifications:

Use MatcherAssert.assertThat(...) as replacement for deprecated method

Result:

Less deprecation warnings
2020-10-18 14:30:52 +02:00
Artem Smotrakov
1ca7d5db81
Fix or suppress LGTM findings (#10689)
Motivation:

LGTM reports multiple issues. They need to be triaged,
and real ones should be fixed.

Modifications:
- Fixed multiple issues reported by LGTM, such as redundant conditions,
  resource leaks, typos, possible integer overflows.
- Suppressed false-positives.
- Added a few testcases.

Result:

Fixed several possible issues, get rid of false alarms in the LGTM report.
2020-10-17 09:49:44 +02:00
Norman Maurer
6b613682ba
Ensure we don't leak the ClassLoader in the backtrace (#10691)
Motivation:

We have a few classes in which we store and reuse static instances of various exceptions. When doing so it is important to also override fillInStacktrace() and so prevent the leak of the ClassLoader in the internal backtrace field.

Modifications:

- Add overrides of fillInStracktrace when needed
- Move ThrowableUtil usage in the static methods

Result:

Fixes https://github.com/netty/netty/pull/10686
2020-10-15 20:41:29 +02:00
Aayush Atharva
08ff3442ec
Update OWASP Links in Cookie class (#10677)
Motivation:
Fix Broken Link of OWASP HttpOnly Cookie in Cookie class.

Modification:
Updated the broken link.

Result:
Broken Link Fix for better Documentation.
2020-10-15 14:08:36 +02:00
Aayush Atharva
6cb72be2dd
Use ObjectUtil for multiple operations (#10679)
Motivation:
We should use ObjectUtil for checking if Compression parameters are in range. This will reduce LOC and make code more readable.

Modification:
Used ObjectUtil

Result:
More readable code
2020-10-14 12:01:42 +02:00
root
01b7e18632 [maven-release-plugin] prepare for next development iteration 2020-10-13 06:29:26 +00:00
root
d4a0050ef3 [maven-release-plugin] prepare release netty-4.1.53.Final 2020-10-13 06:29:02 +00:00
Artem Smotrakov
5b55ee434b
Avoid casting numbers to narrower types (#10645)
Motivation:

Avoid implicit conversions to narrower types in
AbstractMemoryHttpData and Bzip2HuffmanStageEncoder classes
reported by LGTM.

Modifications:

Updated the classes to avoid implicit casting to narrower types.
It doesn't look like that an integer overflow is possible there,
therefore no checks for overflows were added.

Result:

No warnings about implicit conversions to narrower types.
2020-10-12 09:33:47 +02:00
Artem Smotrakov
00f21845f8
Suppress warnings about weak hash algorithms (#10647)
Motivation:

LGTM reported that WebSocketUtil uses MD5 and SHA-1
that are considered weak. Although those algorithms
are insecure, they are required by draft-ietf-hybi-thewebsocketprotocol-00
specification that is implemented in the corresponding WebSocket
handshakers. Once the handshakers are removed, WebSocketUtil can be
updated to stop using those weak hash functions.

Modifications:

Added SuppressWarnings annotations.

Result:

Suppressed warnings.
2020-10-12 09:24:17 +02:00
Doyun Geum
d01471917b
Add validation check about websocket path (#10583)
Add validation check about websocket path

Motivation:

I add websocket handler in custom server with netty.
I first add WebSocketServerProtocolHandler in my channel pipeline.
It does work! but I found that it can pass "/websocketabc". (websocketPath is "/websocket")

Modification:
`isWebSocketPath()` method of `WebSocketServerProtocolHandshakeHandler` now checks that "startsWith" applies to the first URL path component, rather than the URL as a string.

Result:
Requests to "/websocketabc" are no longer passed to handlers for requests that starts-with "/websocket".
2020-10-08 12:06:43 +02:00
Divij Vaidya
79a7c157a3
Clear scheduled timeout if channel is closed with incomplete WebSocket handshake (#10510)
Motivation:

Consider a scenario when the client iniitiates a WebSocket handshake but before the handshake is complete,
the channel is closed due to some reason. In such scenario, the handshake timeout scheduled on the executor
is not cleared. The reason it is not cleared is because in such cases the handshakePromise is not completed.

Modifications:

This change completes the handshakePromise exceptinoally on channelInactive callback, if it has not been
completed so far. This triggers the callback on completion of the promise which clears the timeout scheduled
on the executor.

This PR also adds a test case which reproduces the scenario described above. The test case fails before the
fix is added and succeeds when the fix is applied.

Result:

After this change, the timeout scheduled on the executor will be cleared, thus freeing up thread resources.
2020-09-16 09:40:42 +02:00
Kevin Wu
95ce1b95ea
Fix DeleteOnExitHook cause memory leak (#10560)
Motivation:

If DeleteOnExitHook is in the open state and the program runs for a long time, the DeleteOnExitHook file keeps increasing.
This results in a memory leak


Modification:

I re-customized a DeleteOnExitHook hook. If DeleteOnExitHook is turned on, this hook will be added when creating a temporary file. After the request ends and the corresponding resources are released, the current file will be removed from this hook, so that it will not increase all the time.

Result:

Fixes https://github.com/netty/netty/issues/10351
2020-09-15 16:38:42 +02:00
root
957ef746d8 [maven-release-plugin] prepare for next development iteration 2020-09-08 05:26:25 +00:00
root
ada9c38c0a [maven-release-plugin] prepare release netty-4.1.52.Final 2020-09-08 05:26:05 +00:00
Nick Hill
8e8f01e01d
Use ByteBuf#isAccessible() in more places (#10506)
Motivation

ByteBuf has an isAccessible method which was introduced as part of ref
counting optimizations but there are some places still doing
accessibility checks by accessing the volatile refCnt() directly.

Modifications

- Have PooledNonRetained(Duplicate|Sliced)ByteBuf#isAccessible() use
their refcount delegate's isAccessible() method
- Add static isAccessible(buf) and ensureAccessible(buf) methods to
ByteBufUtil
(since ByteBuf#isAccessible() is package-private)
- Adjust DefaultByteBufHolder and similar classes to use these methods
rather than access refCnt() directly

Result

- More efficient accessibility checks in more places
2020-08-28 09:18:13 +02:00
Andrey Mizurov
f51ae686c4
Small fix that takes into account the remainder when assigning the size (see #10453) (#10491)
Motivation:

This is small fixes for #10453 PR according @njhill and @normanmaurer conversation.

Modification:

Simple refactor and takes into account remainder when calculate size.

Result:

Behavior is correct
2020-08-21 09:18:48 +02:00
Violeta Georgieva
b27914e302
AbstractDiskHttpData#getChunk closes fileChannel only if everything w… (#10481)
Motivation:
AbstractDiskHttpData#getChunk opens and closes fileChannel every time when it is invoked,
as a result the uploaded file is corrupted. This is a regression caused by #10270.

Modifications:

- Close the fileChannel only if everything was read or an exception is thrown
- Add unit test

Result:
AbstractDiskHttpData#getChunk closes fileChannel only if everything was read or an exception is thrown
2020-08-14 10:54:43 +02:00
Andrey Mizurov
686df17b1a
Fix #10449, buffer.getBytes(...) not change a file channel position (#10453)
Motivation:

Regression appeared after making changes in fix #10360 .
The main problem here that `buffer.getBytes(buffer.readerIndex(), fileChannel, fileChannel.position(), localsize)`
doesn't change channel position after writes.

Modification:

Manually set position according to the written bytes.

Result:

Fixes #10449 .
2020-08-07 13:53:16 +02:00
root
bfbeb2dec6 [maven-release-plugin] prepare for next development iteration 2020-07-09 12:27:06 +00:00
root
646934ef0a [maven-release-plugin] prepare release netty-4.1.51.Final 2020-07-09 12:26:30 +00:00
Bennett Lynch
9557c88da2
Add option to HttpObjectDecoder to allow duplicate Content-Lengths (#10349)
Motivation:

Since https://github.com/netty/netty/pull/9865 (Netty 4.1.44) the
default behavior of the HttpObjectDecoder has been to reject any HTTP
message that is found to have multiple Content-Length headers when
decoding. This behavior is well-justified as per the risks outlined in
https://github.com/netty/netty/issues/9861, however, we can see from the
cited RFC section that there are multiple possible options offered for
responding to this scenario:

> If a message is received that has multiple Content-Length header
> fields with field-values consisting of the same decimal value, or a
> single Content-Length header field with a field value containing a
> list of identical decimal values (e.g., "Content-Length: 42, 42"),
> indicating that duplicate Content-Length header fields have been
> generated or combined by an upstream message processor, then the
> recipient MUST either reject the message as invalid or replace the
> duplicated field-values with a single valid Content-Length field
> containing that decimal value prior to determining the message body
> length or forwarding the message.

https://tools.ietf.org/html/rfc7230#section-3.3.2

Netty opted for the first option (rejecting as invalid), which seems
like the safest, but the second option (replacing duplicate values with
a single value) is also valid behavior.

Modifications:

* Introduce "allowDuplicateContentLengths" parameter to
HttpObjectDecoder (defaulting to false).
* When set to true, will allow multiple Content-Length headers only if
they are all the same value. The duplicated field-values will be
replaced with a single valid Content-Length field.
* Add new parameterized test class for testing different variations of
multiple Content-Length headers.

Result:

This is a backwards-compatible change with no functional change to the
existing behavior.

Note that the existing logic would result in NumberFormatExceptions
for header values like "Content-Length: 42, 42". The new logic correctly
reports these as IllegalArgumentException with the proper error message.

Additionally note that this behavior is only applied to HTTP/1.1, but I
suspect that we may want to expand that to include HTTP/1.0 as well...
That behavior is not modified here to minimize the scope of this change.
2020-07-06 10:25:13 +02:00
feijermu
8b3abf3b49
Fix a javadoc mistake. (#10364)
Motivation:

There exists a `javadoc` mistake in `HttpHeaderValues.java`.

Modification:

Just correct this `javadoc` mistake...
2020-06-23 09:23:59 +02:00
Norman Maurer
48938e2d99
Fix memory leak in AbstractDiskHttpData when CompositeByteBuf is used (#10360)
Motivation:

AbstractDiskHttpData may cause a memory leak when a CompositeByteBuf is used. This happened because we may call copy() but actually never release the newly created ByteBuf.

Modifications:

- Remove copy() call and just use ByteBuf.getBytes(...) which will internally handle the writing to the FileChannel without any extra copies that need to be released later on.
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/10354
2020-06-22 13:54:20 +02:00
Bennett Lynch
bcb62be62b
Consolidate HttpObjectDecoder default values into constants (#10344)
Motivation

HttpObjectDecoder and its associated classes make frequent use of
default values for maxInitialLineLength, maxHeaderSize, maxChunkSize,
etc. Today, these defaults are defined in-line in constructors and
duplicated across many classes. This repetition is more prone to error
and inconsistencies.

Furthermore, due to the current lack of builder support, if a user wants
to change just one of these values (e.g., maxHeaderSize), they are also
required to know and repeat the other default values (e.g.,
maxInitialLineLength and maxChunkSize).

The primary motivation for this change is as we are considering adding
another constructor parameter (for multiple content length behavior),
appending this parameter may require some users to have prior knowledge
of the default initialBufferSize, and it would be cleaner to allow them
to reference the default constant.

Modifications

* Consolidate the HttpObjectDecoder default values into public constants
* Reference these constants where possible

Result

No functional change. Additional telescoping constructors will be easier
and safer to write. Users may have an easier experience changing single
parameters.
2020-06-12 07:39:10 +02:00
Lin Gao
de134da720
More values other than chunked defined in Transfer-Encoding header leads to decode failure (#10321)
Motivation:

`containsValue()` will check if there are multiple values defined in the specific header name, we need to use this method instead of `contains()` for the `Transfer-Encoding` header to cover the case that multiple values defined, like: `Transfer-Encoding: gzip, chunked`

Modification:

Change from `contains()` to `containsValue()` in `HttpUtil.isTransferEncodingChunked()` method.

Result:

Fixes #10320
2020-06-02 14:29:20 +02:00
Andrey Mizurov
0dc94e4965
Set (and override) reserved websocket handshake response headers after custom to avoid duplication (#10319)
Motivation:
Currently we passing custom websocket handshaker response headers to a `WebSocketServerHandshaker` but they can contain a reserved headers (e.g. Connection, Upgrade, Sec-Websocket-Accept) what lead to duplication because we use response.headers().add(..) instead of response.headers().set(..).

Modification:
In each `WebSocketServerHandshaker00`, ... `WebSocketServerHandshaker13` implementation replace the method add(..) to set(..) for reserved response headers.

Result:

Less error-prone
2020-06-02 11:56:22 +02:00
prgitpr
2183b37892
Fix a potential fd leak in AbstractDiskHttpData.getChunk (#10270)
Motivation:

`FileChannel.read()` may throw an IOException. We must deal with this in case of the occurrence of `I/O` error.

Modification:

Place the `FileChannel.read()` method call in the `try-finally` block.

Result:

Advoid fd leak.


Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2020-05-14 10:16:16 +02:00
Fabien Renaud
d5087deec6
Fix regression in HttpPostStandardRequestDecoder to always decode + to whitespace (#10285)
Motivations
-----------
HttpPostStandardRequestDecoder was changed in 4.1.50 to provide its own
ByteBuf UrlDecoder. Prior to this change, it was using the decodeComponent
method from QueryStringDecoder which decoded + characters to
whitespaces. This behavior needs to be preserved to maintain backward
compatibility.

Modifications
-------------
Changed HttpPostStandardRequestDecoder to detect + bytes and decode them
toe whitespaces. Added a test.

Results
-------
Addresses issues#10284
2020-05-14 09:28:40 +02:00
root
caf51b7284 [maven-release-plugin] prepare for next development iteration 2020-05-13 06:00:23 +00:00
root
8c5b72aaf0 [maven-release-plugin] prepare release netty-4.1.50.Final 2020-05-13 05:59:55 +00:00
Norman Maurer
f00160bca3
Don't reuse ChannelPromise in WebSocketProtocolHandler (#10248)
Motivation:

We cant reuse the ChannelPromise as it will cause an error when trying to ful-fill it multiple times.

Modifications:

- Use a new promise and chain it with the old one
- Add unit test

Result:

Fixes https://github.com/netty/netty/issues/10240
2020-05-07 08:11:33 +02:00
Norman Maurer
8f7ca2b4ef
Reuse the same allocator as used by the ByteBuf that is used during… (#10226)
Motivation:

We should not use Unpooled to allocate buffers if possible to ensure we can make use of pooling etc.

Modifications:

- Only allocate a buffer if really needed
- Use the ByteBufAllocator of the offered ByteBuf
- Ensure we not use buffer.copy() but explicitly allocate a buffer and then copy into it to not hit the limit of maxCapacity()

Result:

Improve allocations
2020-04-29 14:39:14 +02:00
Norman Maurer
ff36f2826c
Remove some debugging cruft (#10229)
Motivation:

RtspDecoderTest did include a println(...) call which was a left over from debugging.

Modifications:

Remove println(...)

Result:

Cleanup
2020-04-29 11:35:22 +02:00
Norman Maurer
987a68eb02
Fix memory leak in HttpPostMultipartRequestDecoder (#10227)
Motivation:

We need to release all ByteBufs that we allocate to prevent leaks. We missed to release the ByteBufs that are used to aggregate in two cases

Modifications:

Add release() calls

Result:

No more memory leak in HttpPostMultipartRequestDecoder
2020-04-29 08:23:41 +02:00
Fabien Renaud
c354fa48e1
HttpPostRequestDecoder: retain instead of copy when first buf is last (#10209)
Motivations
-----------
There is no need to copy the "offered" ByteBuf in HttpPostRequestDecoder
when the first HttpContent ByteBuf is also the last (LastHttpContent) as
the full content can immediately be decoded. No extra bookeeping needed.

Modifications
-------------
HttpPostMultipartRequestDecoder
 - Retain the first ByteBuf when it is both the first HttpContent offered
to the decoder and is also LastHttpContent.
 - Retain slices of the final buffers values

Results
-------
ByteBufs of FullHttpMessage decoded by HttpPostRequestDecoder are no longer
unnecessarily copied. Attributes are extracted as retained slices when
the content is multi-part. Non-multi-part content continues to return
Unpooled buffers.

Partially addresses issue #10200
2020-04-28 09:43:05 +02:00
feijermu
9751bb3ebc
Move up the size check in AbstractDiskHttpData.setContent. (#10222)
Motivation:

`AbstractHttpData.checkSize` may throw an IOException if we set the max size limit via `AbstractHttpData.setMaxSize`. However, if this exception happens, the `AbstractDiskHttpData.file` and the `AbstractHttpData.size` are still be modified. In other words, it may break the failure atomicity here.

Modification:

Just move up the size check.

Result:

Keep the failure atomicity even if `AbstractHttpData.checkSize` fails.
2020-04-28 09:34:33 +02:00
feijermu
eb3721b971
Fix a potential fd leak in AbstractDiskHttpData.delete (#10212)
Motivation:

An unexpected IOException may be thrown from `FileChannel.force`. If it happens, the `FileChannel.close` may not be invoked.

Modification:

Place the `FileChannel.close` in a finally block.

Result:

Avoid fd leak.
2020-04-27 07:03:45 +02:00
root
9c5008b109 [maven-release-plugin] prepare for next development iteration 2020-04-22 09:57:54 +00:00
root
d0ec961cce [maven-release-plugin] prepare release netty-4.1.49.Final 2020-04-22 09:57:26 +00:00
feijermu
32e32fe98f
Fix a potential fd leak in AbstractDiskHttpData.setContent (#10198)
Motivation:

`RandomAccessFile.setLength` may throw an IOException. We must deal with this in case of the occurrence of `I/O` error.

Modification:

Place the `RandomAccessFile.setLength` method call in the `try-finally` block.

Result:

Avoid fd leak.
2020-04-21 10:54:09 +02:00
feijermu
17f205fab3
Close the FileChannel in case of an IOException in AbstractDiskHttpData.addContent. (#10188)
Motivation:

`FileChannel.force` may throw an IOException. A fd leak may happen here.

Modification:

Close the fileChannel in a finally block.

Result:

Avoid fd leak.
2020-04-15 09:24:26 +02:00
feijermu
3ebf3d7705
Close the file in case of an IOException in AbstractMemoryHttpData.renameTo. (#10163)
Motivation:

An `IOException` may be thrown from `FileChannel.write` or `FileChannel.force`, and cause the fd leak.

Modification:

Close the file in a finally block.

Result:

Avoid fd leak.
2020-04-06 10:12:46 +02:00
feijermu
bdaa935756
Release the temporary ByteBuf in case of unexpected exception in WebSocketUtil.base64. (#10160)
Motivation:

An exception may occur between ByteBuf's allocation and release. So I think it's a good idea to place the release operation in a finally block.

Modification:

Release the temporary ByteBuf in finally blocks.

Result:

Avoid ByteBuf leak.
2020-04-03 14:49:40 +02:00
feijermu
d4c268ae3e
Close the file when IOException occurs in AbstractMemoryHttpData. (#10157)
Motivation:

An IOException may be thrown from FileChannel.read, and cause the fd leak.

Modification:

Close the file when IOException occurs.

Result:
Avoid fd leak.
2020-04-02 14:16:01 +02:00
Romain Manni-Bucau
84ebf47515
making DefaultHttpDataFactory able to configure basedir and deleteonexit (#10146)
Motivation: 

currently (http) disk based attributes or uploads are globally configured in a single directory and can also only globally be deleted on exit or not. it does not fit well multiple cases, in particular the case you have multiple servers in the same JVM.

Modification: 
make it configurable per attribute/fileupload.

Result:

This PR duplicates Disk* constructor to add basedir and deleteonexit parameters and wires it in default http daa factory.
2020-03-30 13:11:46 +02:00
feijermu
aed5a74e09
Release the ByteBuf when IOException occurs in AbstractMemoryHttpData. (#10133)
Motivation:

An IOException may be thrown from InputStream.read or checkSize method, and cause the ByteBuf leak.

Modification:

Release the ByteBuf when IOException occurs.

Result:
Avoid ByteBuf leak.
2020-03-30 11:26:03 +02:00