Motivation:
We cannot control when "the system" reuses file descriptors.
This makes any test that assert on the behaviour of closed file descriptors inherently racy.
Modification:
Allow the EpollSocketChannelConfigTest socketoption tests a few tries to get the correct assertion on the behaviour of closed socket file descriptors.
Result:
The EpollSocketChannelConfigTest should now be much less flaky.
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.
See https://github.com/hyperxpro/Brotli4j/pull/27
Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.
Result:
Up-to-date Broti4j library.
Motivation:
As the release of JDK17 is getting closer and there are ea builds already we should ensure we can actually build netty with it.
Modifications:
- Add profile for JDK17
- Remove test-code that would fail with JDK17 due the changes in 4f4d0f5366.
Result:
Be able to build and run testsuite with JDK17
Motivation:
japicmp did fix some issues related to Java16+. Let's update so we can also enable it in later java versions
Modifications:
Update to 0.15.3
Result:
Be able to use japicmp with all java versions
Motivation:
New versions of Bouncy Castle libraries are out and we should upgrade to them.
Modification:
Upgraded all Bouncy Castle libraries to the latest version.
Result:
The latest versions of Bouncy Castle libraries.
Motivation:
We need to add `--add-exports java.base/sun.security.x509=ALL-UNNAMED` when running the tests for codec-http2 as some of the tests use SelfSignedCertificate.
Modifications:
- Add `--add-exports java.base/sun.security.x509=ALL-UNNAMED` when running the tests for codec-http2
- Ensure we export correct when running with JDK12, 13, 14 and 15 as well
Result:
No more tests failure due not be able to access classes
Motivation:
This caused test failures due to the deprecation warning and produced a
dumpstream.
Modification:
Replace deprecated flag with recommended one.
Result:
Fix deprecation and cause of test failure in codec project.
Motivation:
In Netty 5 we wish to have a simpler, safe, future proof, and more consistent buffer API.
We developed such an API in the incubating buffer repository, and taking it through multiple rounds of review and adjustments.
This PR/commit bring the results of that work into the Netty 5 branch of the main Netty repository.
Modifications:
* `Buffer` is an interface, and all implementations are hidden behind it.
There is no longer an inheritance hierarchy of abstract classes and implementations.
* Reference counting is gone.
After a buffer has been allocated, calling `close` on it will deallocate it.
It is then up to users and integrators to ensure that the life-times of buffers are managed correctly.
This is usually not a problem as buffers tend to flow through the pipeline to be released after a terminal IO operation.
* Slice and duplicate methods are replaced with `split`.
By removing slices, duplicate, and reference counting, there is no longer a possibility that a buffer and/or its memory can be shared and accessible through multiple routes.
This solves the problem of data being accessed from multiple places in an uncoordinated way, and the problem of buffer memory being closed while being in use by some unsuspecting piece of code.
Some adjustments will have to be made to other APIs, idioms, and usages, since `split` is not always a replacement for `slice` in some use cases.
* The `split` has been added which allows memory to be shared among multiple buffers, but in non-overlapping regions.
When the memory regions don't overlap, it will not be possible for the different buffers to interfere with each other.
An internal, and completely transparent, reference counting system ensures that the backing memory is released once the last buffer view is closed.
* A Send API has been introduced that can be used to enforce (in the type system) the transfer of buffer ownership.
This is not expected to be used in the pipeline flow itself, but rather for other objects that wrap buffers and wish to avoid becoming "shared views" — the absence of "shared views" of memory is important for avoiding bugs in the absence of reference counting.
* A new BufferAllocator API, where the choice of implementation determines factors like on-/off-heap, pooling or not.
How access to the different allocators will be exposed to integrators will be decided later.
Perhaps they'll be directly accessible on the `ChannelHandlerContext`.
* The `PooledBufferAllocator` has been copied and modified to match the new allocator API.
This includes unifying its implementation that was previously split across on-heap and off-heap.
* The `PooledBufferAllocator` implementation has also been adjusted to allocate 4 MiB chunks by default, and a few changes have been made to the implementation to make a newly created, empty allocator use significantly less heap memory.
* A `Resource` interface has been added, which defines the life-cycle methods and the `send` method.
The `Buffer` interface extends this.
* Analogues for `ByteBufHolder` has been added in the `BufferHolder` and `BufferRef` classes.
* `ByteCursor` is added as a new way to iterate the data in buffers.
The byte cursor API is designed to be more JIT friendly than an iterator, or the existing `ByteProcessor` interface.
* `CompositeBuffer` no longer permit the same level of access to its internal components.
The composite buffer enforces its ownership of its components via the `Send` API, and the components can only be individually accessed with the `forEachReadable` and `forEachWritable` methods.
This keeps the API and behavioral differences between composite and non-composite buffers to a minimum.
* Two implementations of the `Buffer` interface are provided with the API: One based on `ByteBuffer`, and one based on `sun.misc.Unsafe`.
The `ByteBuffer` implementation is used by default.
More implementations can be loaded from the classpath via service loading.
The `MemorySegment` based implementation is left behind in the incubator repository.
* An extensive and highly parameterised test suite has been added, to ensure that all implementations have consistent and correct behaviour, regardless of their configuration or composition.
Result:
We have a new buffer API that is simpler, better tested, more consistent in behaviour, and safer by design, than the existing `ByteBuf` API.
The next legs of this journey will be about integrating this new API into Netty proper, and deprecate (and eventually remove) the `ByteBuf` API.
This fixes#11024, #8601, #8543, #8542, #8534, #3358, and #3306.
Motivation:
We need to use a GraalVM dependency which uses GPL2 + CE.
Modifications:
- Update all graalvm dependencies to new GAV which introduces a license change from GPL2 to GPL2 + CE
- This also required a small bump on the general version from 19.2 to 19.3, which should be fine as 19.3 is an official maintained LTS version, while 19.2 wasn't.
Result:
Fixes: #11398
Signed-off-by: Paulo Lopes <pmlopes@gmail.com>
Motivation:
We don't publish any tarballs these days so we can just remove the module
Modifications:
Remove tarball module and also adjust release scripts
Result:
Less code / config to mantain
Motivation:
There is a small typo in `pom.xml`. Typo is: `acclerating`, however it should be `accelerating`.
Modification:
Corrected the typo.
Result:
Typo-free `pom.xml`.
Motivation:
We only release a static compiled version of netty-tcnative for windows so we need to ensure we use it when on windows and not try to use a dynamic build version.
Modifications:
Automatically renable the boringssl profile when on windows
Result:
Fixes https://github.com/netty/netty/issues/11302
Motivation:
We shouldn't include benchmarks and tests in the generated javadocs to keep things clean
Modifications:
Add some more excludes
Result:
Javadocs for benchmarks and tests are not published
Motivation:
Older version of logback are affected by [CVE-2017-5929]. While its an optional dependency we should upgrade
Modifications:
Upgrade to 1.2.3
Result:
No more reports about using affected logback version
Motivation:
When changing the netty-all artifact to not include any sources we also removed the ability to generate the javadocs / xref files for our website
Modifications:
- Add new profile which will generate the files
- Add script which generates all the files and copy these over to the netty-website
Result:
Easier to generate files for website
Motivation:
Just use MAVEN_OPTS to setup all the timeouts etc for dependency downloads. This way we at least can be sure these are applied.
Modifications:
- Use MAVEN_OPTS
- Remove ci profile
- Remove unused settings.xml file
- Always use ./mvnw
Result:
Build stability improvements
Motivation:
When trying to compile with java16 we should use adopt@1.16*
Modifications:
- Use adopt@1.16.0-1-
- Upgrade to blockhoud 1.0.6 to be able to support java16
Result:
Use correct java version / flavor
Motivation:
Conscrypt not correctly filters out non support TLS versions which may lead to test failures.
Related to https://github.com/google/conscrypt/issues/1013
Modifications:
- Bump up to latest patch release
- Add workaround
Result:
No more test failures caused by conscrypt
Motivation:
Netty lacks client side support for decompressing Brotli compressed response bodies.
Modification:
* Introduce optional dependency to brotli4j by @hyperxpro. It will be up to the user to provide the brotli4j libraries for the target platform in the classpath. brotli4j is currently available for Linux, OSX and Windows, all for x86 only.
* Introduce BrotliDecoder in codec module
* Plug it onto `HttpContentDecompressor` for HTTP/1 and `DelegatingDecompressorFrameListener` for HTTP/2
* Add test in `HttpContentDecoderTest`
* Add `BrotliDecoderTest` that doesn't extend `AbstractDecoderTest` that looks flaky
Result:
Netty now support decompressing Brotli compressed response bodies.
Motivation:
0f25213918 introduced some properties that were used to make builds more stable on the ci. All of these properties were duplicated everywhere, this made it hard to maintain
Modifications:
- Add profile which sets the properties.
- Just use the profile when build on the ci
Result:
Easier to maintain custom properties for the ci build
Motivation:
It turned out we didnt run the openssl tests on the CI when we used the non-static version of netty-tcnative.
Modifications:
- Upgrade netty-tcnative to fix segfault when using shared openssl
- Adjust tests to only run session cache tests when openssl supports it
- Fix some more tests to only depend on KeyManager if the underlying openssl version supports it
Result:
Run all openssl test on the CI even when shared library is used
Motivation:
We should skip the deployment of jars that are not meant to be consumed by the user as there is no public API.
Modifications:
Let's skip deployment for modules that are not useful for users
Result:
Build cleanup
netty-jni-util 0.0.2.Final is incompatible with static linking. Before
the netty-jni-util dependency was introduced netty-tcnative supported
static linking via NETTY_BUILD_STATIC. netty-jni-util 0.0.3.Final adds
static linking compatibility.
Modifications:
Bump netty-jni-util to version 0.0.3.Final and update to its new API
which requires the caller to manage packagePrefix.
Result:
Using latest version of netty-jni-util and restored static linking
compatibility.
Motivation:
netty-build is now called netty-build-common
Modifications:
Rename netty-build to netty-build-common
Result:
Be able to compile branch again
Motivation:
The JDK deflate implementation added support for operating on ByteBuffers in Java 11 or so.
This means that we don't need to restrict that implementation to ByteBufs that are heap based and can expose arrays.
Modification:
Add clauses to JdkZlibEncoder and JdkZlibDecoder for handling ByteBufs that don't have arrays, but do have one nioByteBuffer.
Expand the test coverage in JdkZlibTest to include all relevant combinations of buffer types and data types.
Result:
The JdkZlibEncoder and JdkZlibDecoder should now work on basically all non-composite ByteBufs, and likely also composite ByteBufs that have exactly one component.
Motivation:
The current netty's graalvm dependency version is too low, so you need to upgrade the plugin
Modification:
Upgrade Graalvm version to the latest version, please review this pr, thank you
Result:
Use up-to-date version.
Co-authored-by: xingrufei <xingrufei@sogou-inc.com>
* Revert "Add a profile for debugging tests that run from Maven (#11011)"
This reverts commit 83895f0f
The same functionality is already natively available in surefire, by adding the `-Dmaven.surefire.debug` flag to Maven.
* Update surefire/failsafe version
These new versions copes better when our tests prints to STDOUT, and disturbs the progress processing that these plugins do.
Motivation:
As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions.
Modifications:
- Add new maven module which tests loading of native modules
- Add job that will also test loading on aarch64
Result:
Less likely to introduce regressions related to loading native code in the future
Motivation:
New versions of `Bouncy Castle` libraries are out and we should upgrade to them.
Modification:
Upgraded all `Bouncy Castle` libraries to the latest version.
Result:
The latest versions of `Bouncy Castle` libraries.
Fixes#10905.
Motivation:
netty-jni-util is now also hosted on maven central. Let's use it
Modifications:
Adjust plugins to just unpack netty-jni-util and use it
Result:
Be able to use what is in the maven cache for netty-jni-util
Motivation:
In #10630, field substitutions were introduced for NetUtil.LOCALHOST4, NetUtil.LOCALHOST6 and NetUtil.LOCALHOST fields. They were required to allow a native image be built with most of Netty (including NetUtil) initialized at build time.
The substitutions created in #10630 only define getters, so the 3 fields can only be read in a native image.
But when NetUtil is initialized at run-time (this is what happens in #10797), its static initialization block is executed, and this block writes to all 3 fields. As the substitutions do not provide any setters, field stores are not valid, and such builds fail.
Modifications:
- Add netty-testsuite-native-image-client-runtime-init Maven module that builds a native image deferring NetUtil class initialization till run-time; this module is used to demonstrate the problem and verify the problem is gone with the fix
- Add no-op setters to substitutions for NetUtil.LOCALHOST4, NetUtil.LOCALHOST6 and NetUtil.LOCALHOST
Result:
A native image initializing NetUtil at run-time builds successfully.
Fixes#10797
Motivation:
https in xmlns URIs does not work and will let the maven release plugin fail:
```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.779 s
[INFO] Finished at: 2020-11-10T07:45:21Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.3:prepare (default-cli) on project netty-parent: Execution default-cli of goal org.apache.maven.plugins:maven-release-plugin:2.5.3:prepare failed: The namespace xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" could not be added as a namespace to "project": The namespace prefix "xsi" collides with an additional namespace declared by the element -> [Help 1]
[ERROR]
```
See also https://issues.apache.org/jira/browse/HBASE-24014.
Modifications:
Use http for xmlns
Result:
Be able to use maven release plugin
Motivation:
JUnit 5 is the new hotness. It's more expressive, extensible, and composable in many ways, and it's better able to run tests in parallel. But most importantly, it's able to directly run JUnit 4 tests.
This means we can update and start using JUnit 5 without touching any of our existing tests.
I'm also introducing a dependency on assertj-core, which is like hamcrest, but arguably has a nicer and more discoverable API.
Modification:
Add the JUnit 5 and assertj-core dependencies, without converting any tests at time time.
Result:
All our tests are now executed through the JUnit 5 Vintage Engine.
Also, the JUnit 5 test APIs are available, and any JUnit 5 tests that are added from now on will also be executed.
Motivation:
Conscrypt 2.5.1 is available and it's a good idea to upgrade to the latest version.
Modification:
Upgraded Conscrypt 2.4.0 to 2.5.1
Result:
Newer Conscrypt version.
Motivation:
SLF4J 1.7.30 is the latest version in 1.7.x and we should upgrade to it from 1.7.21.
Modification:
Changed 1.7.21 to 1.7.30
Result:
Newer version of SLF4J
Motivation:
Since GraalVM version 19.3.0, instances of java.net.InetAddress (and its subclasses Inet4Address and Inet6Address) are not allowed in native image heap (that is, they cannot be stored in static fields of classes initialized at build time or be reachable through static fields of such classes). When building a native image, it makes sense to initialize at build time as many classes as possible.
But some fields of some classes in Netty (for example, NetUtil.LOCALHOST4) contain InetAddress instances. If a program is using code path that makes it possible to reach such fields at build time initialization, it becomes impossible to build a native image initializing core Netty classes initialized at runtime. An example of such a program is a client that uses netty-dns.
Modifications:
- Add netty-testsuite-native-image-client Maven module to test that such an example program can be built after the corresponding fixes
- Add native-image.properties to resolver-dns module to move initialization of some classes to runtime (some of them are parsing configuration during initialization, so it makes no sense to initialize them at build time; for others, it's needed to avoid InetAddress reachability at build time)
- Add substitutions for NetUtil.LOCALHOST4, NetUtil.LOCALHOST6 and NetUtil.LOCALHOST to overcome the InetAddress-related prohibition
- Extract some initialization code from NetUtil to NetUtilInitializations to allow it to be used by the substitutions
Result:
A client program using netty-dns with --initialize-at-build-time=io.netty builds successfully
Motivation:
HTTP is a plaintext protocol which means that someone may be able
to eavesdrop the data. To prevent this, HTTPS should be used whenever
possible. However, maintaining using https:// in all URLs may be
difficult. The nohttp tool can help here. The tool scans all the files
in a repository and reports where http:// is used.
Modifications:
- Added nohttp (via checkstyle) into the build process.
- Suppressed findings for the websites
that don't support HTTPS or that are not reachable
Result:
- Prevent using HTTP in the future.
- Encourage users to use HTTPS when they follow the links they found in
the code.
Motivation:
26f3cd89ef did introduce a profile for compiling on JDK15 but did miss to set the compiler settings correctly for the requirements of this branch
Modifications:
Correct compiler settings.
Result:
Be able to build with JDK15
Motivation:
Java 16 will come around eventually anyway, and this makes it easier for people to experiment with Early Access builds.
Modification:
- Added Maven profiles for JDK 16 to relevant pom files.
- Removed the `--add-exports java.base/sun.security.x509=ALL-UNNAMED` argument when running tests; we've not needed it since the Java11-as-baseline PR landed.
Result:
Netty now builds on JDK 16 pre-releases (provided they've not broken compatibility in some way).
