Commit Graph

698 Commits

Author SHA1 Message Date
Norman Maurer
bf0beb772c Fix IllegalArgumentException when release a wrapped ByteBuffer on Java9
Motivation:

Unsafe.invokeCleaner(...) checks if the passed in ByteBuffer is a slice or duplicate and if so throws an IllegalArgumentException on Java9. We need to ensure we never try to free a ByteBuffer that was provided by the user directly as we not know if its a slice / duplicate or not.

Modifications:

Never try to free a ByteBuffer that was passed into UnpooledUnsafeDirectByteBuf constructor by an user (via Unpooled.wrappedBuffer(....)).

Result:

Build passes again on Java9
2017-04-20 19:19:11 +02:00
Nikolay Fedorovskikh
0692bf1b6a fix the typos 2017-04-20 04:56:09 +02:00
Norman Maurer
e482d933f7 Add 'io.netty.tryAllocateUninitializedArray' system property which allows to allocate byte[] without memset in Java9+
Motivation:

Java9 added a new method to Unsafe which allows to allocate a byte[] without memset it. This can have a massive impact in allocation times when the byte[] is big. This change allows to enable this when using Java9 with the io.netty.tryAllocateUninitializedArray property when running Java9+. Please note that you will need to open up the jdk.internal.misc package via '--add-opens java.base/jdk.internal.misc=ALL-UNNAMED' as well.

Modifications:

Allow to allocate byte[] without memset on Java9+

Result:

Better performance when allocate big heap buffers and using java9.
2017-04-19 11:45:39 +02:00
Scott Mitchell
21562d8808 Retained[Duplicate|Slice] operations should not increase the reference count for UnreleasableByteBuf
Motivation:
UnreleasableByteBuf operations are designed to not modify the reference count of the underlying buffer. The Retained[Duplicate|Slice] operations violate this assumption and can cause the underlying buffer's reference count to be increased, but never allow for it to be decreased. This may lead to memory leaks.

Modifications:
- UnreleasableByteBuf's Retained[Duplicate|Slice] should leave the reference count of the parent buffer unchanged after the operation completes.

Result:
No more memory leaks due to usage of the Retained[Duplicate|Slice] on an UnreleasableByteBuf object.
2017-03-31 17:45:29 -07:00
Scott Mitchell
ef21d5f4ca UnsafeByteBufUtil errors and simplification
Motiviation:
UnsafeByteBufUtil has some bugs related to using an incorrect index, and also omitting the array paramter when dealing with byte[] objects. There is also some simplification possible with respect to type casting, and minor formatting consistentcy issues.

Modifications:
- Ensure indexing is correct when dealing with native memory
- Fix the native access and endianness for the medium/unsigned medium methods
- Ensure array is used when dealing with heap memory
- Remove unecessary casts when using long
- Fix formating and alignment

Result:
UnsafeByteBufUtil is more correct and won't access direct memory when heap arrays are used.
2017-03-30 11:52:03 -07:00
Norman Maurer
6036b3f6ea Fix buffer leak in EmptyByteBufTest introduced by aa2f16f314 2017-03-27 05:20:02 +02:00
Bryce Anderson
aa2f16f314 EmptyByteBuf allows writing ByteBufs with 0 readable bytes
Motivation:

The contract of `ByteBuf.writeBytes(ByteBuf src)` is such that it will
throw an `IndexOutOfBoundsException if `src.readableBytes()` is greater than
`this.writableBytes()`. The EmptyByteBuf class will throw the exception,
even if the source buffer has zero readable bytes, in violation of the
contract.

Modifications:

Use the helper method `checkLength(..)` to check the length and throw
the exception, if appropriate.

Result:

Conformance with the stated behavior of ByteBuf.
2017-03-21 22:00:54 -07:00
Norman Maurer
2b8c8e0805 [maven-release-plugin] prepare for next development iteration 2017-03-10 07:46:17 +01:00
Norman Maurer
1db58ea980 [maven-release-plugin] prepare release netty-4.1.9.Final 2017-03-10 07:45:28 +01:00
Nikolay Fedorovskikh
2993760e92 Fix misordered 'assertEquals' arguments in tests
Motivation:

Wrong argument order in some 'assertEquals' applying.

Modifications:

Flip compared arguments.

Result:

Correct `assertEquals` usage.
2017-03-08 22:48:37 -08:00
Norman Maurer
3ad3356892 Expose ByteBufAllocator metric in a more general way
Motivation:

PR [#6460] added a way to access the used memory of an allocator. The used naming was not very good and how things were exposed are not consistent.

Modifications:

- Add a new ByteBufAllocatorMetric and ByteBufAllocatorMetricProvider interface
- Let the ByteBufAllocator implementations implement ByteBufAllocatorMetricProvider
- Move exposed stats / metric from PooledByteBufAllocator to PooledByteBufAllocatorMetric and mark old methods as `@Deprecated`.

Result:

More consistent way to expose metric / stats for ByteBufAllocator
2017-03-08 20:07:58 +01:00
Scott Mitchell
2cff918044 Correct usages of internalNioBuffer
Motivation:
There are numerous usages of internalNioBuffer which hard code 0 for the index when the intention was to use the readerIndex().

Modifications:
- Remove hard coded 0 for the index and use readerIndex()

Result:
We are less susceptible to using the wrong index, and don't make assumptions about the ByteBufAllocator.
2017-03-02 12:51:22 -08:00
Norman Maurer
461f9a1212 Allow to obtain informations of used direct and heap memory for ByteBufAllocator implementations
Motivation:

Often its useful for the user to be able to get some stats about the memory allocated via an allocator.

Modifications:

- Allow to obtain the used heap and direct memory for an allocator
- Add test case

Result:

Fixes [#6341]
2017-03-01 18:53:43 +01:00
Norman Maurer
a7fe6c0153 Metrics exposed by PooledByteBufAllocator needs to be correctly synchronized
Motivation:

As we may access the metrics exposed of PooledByteBufAllocator from another thread then the allocations happen we need to ensure we synchronize on the PoolArena to ensure correct visibility.

Modifications:

Synchronize on the PoolArena to ensure correct visibility.

Result:

Fix multi-thread issues on the metrics
2017-03-01 06:26:08 +01:00
Norman Maurer
deb90923a2 Ensure PooledByteBuf.initUnpooled(...) correctly set the allocator
Motivation:

Commit 8dda984afe introduced a regression which lead to the situation that the allocator is not set when PooledByteBuf.initUnpooled(...) is called. Thus it was possible that PooledByteBuf.alloc() returns null or the wrong allocator if multiple PooledByteBufAllocator are used in an application.

Modifications:

- Correctly set the allocator
- Add test-case

Result:

Fixes [#6436].
2017-02-23 19:53:17 +01:00
Nikolay Fedorovskikh
0623c6c533 Fix javadoc issues
Motivation:

Invalid javadoc in project

Modifications:

Fix it

Result:

More correct javadoc
2017-02-22 07:31:07 +01:00
Norman Maurer
fbf0e5f4dd Prefer JDK ThreadLocalRandom implementation over ours.
Motivation:

We have our own ThreadLocalRandom implementation to support older JDKs . That said we should prefer the JDK provided when running on JDK >= 7

Modification:

Using ThreadLocalRandom implementation of the JDK when possible.

Result:

Make use of JDK implementations when possible.
2017-02-16 15:44:00 -08:00
Norman Maurer
6ac5f35077 Use Unsafe to read ByteBuffer.address field to make it work on Java9 as well.
Motivation:

Java9 does not allow changing access level via reflection by default. This lead to the situation that netty disabled Unsafe completely as ByteBuffer.address could not be read.

Modification:

Use Unsafe to read the address field as this works on all Java versions.

Result:

Again be able to use Unsafe optimisations when using Netty with Java9
2017-02-16 20:40:59 +01:00
Norman Maurer
8a3a3245df Ensure Unsafe buffer implementations are used when sun.misc.Unsafe is present
Motivation:

When sun.misc.Unsafe is present we want to use *Unsafe*ByteBuf implementations. We missed to do so in PooledByteBufAllocator when the heapArena is null.

Modifications:

- Correctly use UnpooledUnsafeHeapByteBuf
- Add unit tests

Result:

Use most optimal ByteBuf implementation.
2017-02-16 07:48:33 +01:00
Norman Maurer
f09a721d7f Expose the chunkSize used by PooledByteBufAllocator.
Motivation:

Sometimes it may be useful to know the used chunkSize.

Modifications:

Add method to expose chunkSize.

Result:

More exposed details.
2017-02-14 08:37:05 +01:00
Norman Maurer
371c0ca0f8 Eliminate unnessary wrapping when call ByteBuf.asReadOnly() in some cases
Motivation:

We can eliminate unnessary wrapping when call ByteBuf.asReadOnly() in some cases to reduce indirection.

Modifications:

- Check if asReadOnly() needs to create a new instance or not
- Add test cases

Result:

Less object creation / wrapping.
2017-02-14 08:35:16 +01:00
fenik17
0cf3f54a8d Adding 'final' keyword for private fields where possible
Motivation

Missing 'final' keyword for fields

Modifications

Add 'final' for fields where possible

Result

More safe and consistent code
2017-02-14 08:29:15 +01:00
Norman Maurer
34ea09e552 Add missing assumeTrue(...) that were not added in 54339c08ac 2017-02-14 08:17:33 +01:00
Norman Maurer
9b2b3e2512 Ensure tests pass when sun.misc.Unsafe is not present
Motivation:

We need to ensure we pass all tests when sun.misc.Unsafe is not present.

Modifications:

- Make *ByteBufAllocatorTest work whenever sun.misc.Unsafe is present or not
- Let Lz4FrameEncoderTest not depend on AbstractByteBufAllocator implementation details which take into account if sun.misc.Unsafe is present or not

Result:

Tests pass even without sun.misc.Unsafe.
2017-02-14 07:52:07 +01:00
Norman Maurer
54339c08ac Only try to calculate direct memory offset when sun.misc.Unsafe is present
Motivation:

We should only try to calculate the direct memory offset when sun.misc.Unsafe is present as otherwise it will fail with an NPE as PlatformDependent.directBufferAddress(...) will throw it.
This problem was introduced by 66b9be3a46.

Modifications:

Use offset of 0 if no sun.misc.Unsafe is present.

Result:

PooledByteBufAllocator also works again when no sun.misc.Unsafe is present.
2017-02-14 07:49:24 +01:00
Norman Maurer
d8596d2d90 Two tests are missing @Test annotations
Motivation:

ReadOnlyByteBufTest contains two tests which are missing the `@Test` annotation and so will never run.

Modifications:

Add missing annotation.

Result:

Tests run as expected.
2017-02-14 07:48:37 +01:00
Norman Maurer
a7c0ff665c Only use Mockito for mocking.
Motivation:

We used various mocking frameworks. We should only use one...

Modifications:

Make usage of mocking framework consistent by only using Mockito.

Result:

Less dependencies and more consistent mocking usage.
2017-02-07 08:47:22 +01:00
Kiril Menshikov
66b9be3a46 Allow to allign allocated Buffers
Motivation:

64-byte alignment is recommended by the Intel performance guide (https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors) for data-structures over 64 bytes.
Requiring padding to a multiple of 64 bytes allows for using SIMD instructions consistently in loops without additional conditional checks. This should allow for simpler and more efficient code.

Modification:

At the moment cache alignment must be setup manually. But probably it might be taken from the system. The original code was introduced by @normanmaurer https://github.com/netty/netty/pull/4726/files

Result:

Buffer alignment works better than miss-align cache.
2017-02-06 07:58:29 +01:00
Norman Maurer
756b78b7df Add common tests for ByteBufAllocator / AbstractByteBufAllocator implementations.
Motivation:

We not had tests for ByteBufAllocator implementations in general.

Modifications:

Added ByteBufAllocatorTest, AbstractByteBufAllocatorTest and UnpooledByteBufAllocatorTest

Result:

More tests for allocator implementations.
2017-02-06 07:51:10 +01:00
Dmitriy Dumanskiy
b9abd3c9fc Cleanup : for loops for arrays to make code easier to read and removed unnecessary toLowerCase() 2017-02-06 07:47:59 +01:00
Norman Maurer
66b1731041 PooledByteBuf.capacity(...) not enforces maxCapacity()
Motivation:

PooledByteBuf.capacity(...) miss to enforce maxCapacity() and so its possible to increase the capacity of the buffer even if it will be bigger then maxCapacity().

Modifications:

- Correctly enforce maxCapacity()
- Add unit tests for capacity(...) calls.

Result:

Correctly enforce maxCapacity().
2017-02-01 18:45:54 +01:00
Carl Mastrangelo
ead9938980 Include Http 1 request in error message
Motivation:

When An HTTP server is listening in plaintext mode, it doesn't have
a chance to negotiate "h2" in the tls handshake.  HTTP 1 clients
that are not expecting an HTTP2 server will accidentally a request
that isn't an upgrade, which the HTTP/2 decoder will not
understand.  The decoder treats the bytes as hex and adds them to
the error message.

These error messages are hard to understand by humans, and result
in extra, manual work to decode.

Modification:

If the first bytes of the request are not the preface, the decoder
will now see if they are an HTTP/1 request first.  If so, the error
message will include the method and path of the original request in
the error message.

In case the path is long, the decoder will check up to the first
1024 bytes to see if it matches.  This could be a DoS vector if
tons of bad requests or other garbage come in.  A future optimization
would be to treat the first few bytes as an AsciiString and not do
any Charset decoding.  ByteBuf.toCharSequence alludes to such an
optimization.

The code has been left simple for the time being.

Result:

Faster identification of errant HTTP requests.
2017-01-30 09:46:38 -08:00
Norman Maurer
735d6dd636 [maven-release-plugin] prepare for next development iteration 2017-01-30 15:14:02 +01:00
Norman Maurer
76e22e63f3 [maven-release-plugin] prepare release netty-4.1.8.Final 2017-01-30 15:12:36 +01:00
ming.ma
f10f8a3131 Calculate correct count for tiny/small/normal allocation
Motivation:

Disable ThreadLocal Cache, then allocate Pooled ByteBuf and release all these buffers, PoolArena's tiny/small/normal allocation count is incorrect.

Modifications:

- Calculate PoolArena's tiny/small/normal allocation one time
- Add testAllocationCounter TestCase

Result:

Fixes #6282 .
2017-01-30 10:32:58 +01:00
Norman Maurer
8dda984afe Null out references to tmpNioBuf and chunk to allow quicker collecting
Motivation:

In PooledByteBuf we missed to null out the chunk and tmpNioBuf fields before recycle it to the Recycler. This could lead to keep objects longer alive then necessary which may hold a lot of memory.

Modifications:

Null out tmpNioBuf and chunk before recycle.

Result:

Possible to earlier GC objects.
2017-01-26 22:06:47 +01:00
ming.ma
44add3c525 Log correct value for useCacheForAllThreads
Motivation:

Log about "-Dio.netty.allocator.useCacheForAllThreads" is missing log placeholder, and so can't output correct value.

Modification:

- Add placeholder

Result:

Fixes #6265 .
2017-01-25 08:01:23 +01:00
Norman Maurer
7f01da8d0f [maven-release-plugin] prepare for next development iteration 2017-01-12 11:36:51 +01:00
Norman Maurer
7a21eb1178 [maven-release-plugin] prepare release netty-4.1.7.Final 2017-01-12 11:35:58 +01:00
Scott Mitchell
583a59abb1 ByteBufUtil.compare int underflow
Motivation:
ByteBufUtil.compare uses long arithmetic but doesn't check for underflow on when converting from long to int to satisfy the Comparable interface. This will result in incorrect comparisons and violate the Comparable interface contract.

Modifications:
- ByteBufUtil.compare should protect against int underflow

Result:
Fixes https://github.com/netty/netty/issues/6169
2017-01-10 11:43:59 -08:00
Norman Maurer
89e93968ac Remove usage of own Atomic*FieldUpdater in favor of JDKs
Motivation:

In later Java8 versions our Atomic*FieldUpdater are slower then the JDK implementations so we should not use ours anymore. Even worse the JDK implementations provide for example an optimized version of addAndGet(...) using intrinsics which makes it a lot faster for this use-case.

Modifications:

- Remove methods that return our own Atomic*FieldUpdaters.
- Use the JDK implementations everywhere.

Result:

Faster code.
2016-12-15 08:09:06 +00:00
Norman Maurer
712c16ad83 Ensure leak aware buffers correctly close the ResourceLeakTracker
Motivation:

We should assert that the leak aware buffers correctly close the ResourceLeakTracker in the unit tests.

Modifications:

- Keep track of NoopResourceLeakTrackers and check if these were closed once the test completes
- Fix bugs in tests so the buffers are all released.

Result:

Better tests for leak aware buffers
2016-12-08 19:33:20 +01:00
Norman Maurer
24b39bc287 Only schedule a ThreadDeathWatcher task if caches are used.
Motivation:

If caches are disabled it does not make sense to schedule a task that will free up memory consumed by the caches.

Modifications:

Do not schedule if caches are disabled.

Result:

Less overhead.
2016-12-08 10:36:29 +01:00
Norman Maurer
13a8ebade4 Correctly handle the case when no ResourceLeakTracker was created for derived pooled buffers. This was missed in c2f4daa739 2016-12-04 23:09:23 +01:00
Norman Maurer
c2f4daa739 Fix false-positives when using ResourceLeakDetector.
Motivation:

We need to ensure the tracked object can not be GC'ed before ResourceLeak.close() is called as otherwise we may get false-positives reported by the ResourceLeakDetector. This can happen as the JIT / GC may be able to figure out that we do not need the tracked object anymore and so already enqueue it for collection before we actually get a chance to close the enclosing ResourceLeak.

Modifications:

- Add ResourceLeakTracker and deprecate the old ResourceLeak
- Fix some javadocs to correctly release buffers.
- Add a unit test for ResourceLeakDetector that shows that ResourceLeakTracker has not the problems.

Result:

No more false-positives reported by ResourceLeakDetector when ResourceLeakDetector.track(...) is used.
2016-12-04 09:01:39 +01:00
Norman Maurer
243b2b9f19 PooledByteBufAllocatorTest may has memory visiblity issues as it uses non concurrent queue
Motivation:

PooledByteBufAllocatorTest uses an ArrayQueue but access it from multiple threads (not concurrently but still from different threads). This may leak to memory visibility issues.

Modifications:

- Use a concurrent queue
- Some cleanup

Result:

Non racy test code.
2016-12-02 07:42:19 +01:00
Norman Maurer
2b8fd8d43b Allow to disable caching in PooledByteBufAllocator for non FastThreadLocalThreads
Motivation:

If a user allocates a lot from outside the EventLoop we may end up creating a lot of caches in the PooledByteBufAllocator. This may be wasteful and so it may be useful for an other to configure that caches should only be used from within EventLoops.

Modifications:

Add new constructor which allows to configure the caching behaviour.

Result:

More flexible configuration of PooledByteBufAllocator possible
2016-12-02 07:40:33 +01:00
Norman Maurer
5bc447c539 Allow to build netty when sun.misc.Unsafe is not avaible or -Dio.netty.noUnsafe=true is used.
Motivation:

We support using Netty without sun.misc.Unsafe, so we should also support building it without it. This way we can also run all tests without sun.misc.Unsafe and so see if it works as expected.

Modifications:

Correctly skip tests that depend on sun.misc.Unsafe if its not present or -Dio.netty.noUnsafe=true is used.

Result:

Be able to build netty without sun.misc.Unsafe
2016-12-01 21:24:49 +01:00
Norman Maurer
feae0435b5 SwappedByteBuf.unwrap() should return wrapped buffer.
Motivation:

SwappedByteBuf.unwrap() not returned the wrapped buffer but the buffer that was wrapped by the original buffer. This is not correct.

Modifications:

Correctly return wrapped buffer and fix test.

Result:

SwappedByteBuf.unwrap() works as expected.
2016-12-01 21:22:30 +01:00
Norman Maurer
f70757da2a [#6015] Fix racy PooledByteBufAllocatorTests
Motivation:

We had a few tests PooledByteBufAllocatorTests which used parkNanos(...) to give a resource enough time to get destroyed. This is race and may not be good enough.

Modifications:

Ensure the ThreadCache is really destroyed.

Result:

No more racy tests that depend on ThreadCaches.
2016-12-01 10:16:12 +01:00