Allow users of Netty to plug in their own leak detector for the purpose
of instrumentation.
Motivation:
We are rolling out a large Netty deployment and want to be able to
track the amount of leaks we're seeing in production via custom
instrumentation. In order to achieve this today, I had to plug in a
custom `ByteBufAllocator` into the bootstrap and have it initialize a
custom `ResourceLeakDetector`. Due to these classes mostly being marked
`final` or having private or static methods, a lot of the code had to
be copy-pasted and it's quite ugly.
Modifications:
* I've added a static loader method for the `ResourceLeakDetector` in
`AbstractByteBuf` that tries to instantiate the class passed in via the
`-Dio.netty.customResourceLeakDetector`, otherwise falling back to the
default one.
* I've modified `ResourceLeakDetector` to be non-final and to have the
reporting broken out in to methods that can be overridden.
Result:
You can instrument leaks in your application by just adding something
like the following:
```java
public class InstrumentedResourceLeakDetector<T> extends
ResourceLeakDetector<T> {
@Monitor("InstanceLeakCounter")
private final AtomicInteger instancesLeakCounter;
@Monitor("LeakCounter")
private final AtomicInteger leakCounter;
public InstrumentedResourceLeakDetector(Class<T> resource) {
super(resource);
this.instancesLeakCounter = new AtomicInteger();
this.leakCounter = new AtomicInteger();
}
@Override
protected void reportTracedLeak(String records) {
super.reportTracedLeak(records);
leakCounter.incrementAndGet();
}
@Override
protected void reportUntracedLeak() {
super.reportUntracedLeak();
leakCounter.incrementAndGet();
}
@Override
protected void reportInstancesLeak() {
super.reportInstancesLeak();
instancesLeakCounter.incrementAndGet();
}
}
```
Motivation:
The current DnsNameResolver fails to resolve an A+CNAME answer. For example:
dig moose.rmq.cloudamqp.com
...
;; ANSWER SECTION:
moose.rmq.cloudamqp.com. 1800 IN CNAME ec2-54-152-221-139.compute-1.amazonaws.com.
ec2-54-152-221-139.compute-1.amazonaws.com. 583612 IN A 54.152.221.139
...
The resolver constructs a map of cnames but forgets the trailing "." in the values which lead to not resolve the A record.
Modifications:
Reuse the code of DefaltDnsRecordDecoder which correctly handles the trailing dot.
Result:
Correctly resolve.
Related: #4333#4421#5128
Motivation:
slice(), duplicate() and readSlice() currently create a non-recyclable
derived buffer instance. Under heavy load, an application that creates a
lot of derived buffers can put the garbage collector under pressure.
Modifications:
- Add the following methods which creates a non-recyclable derived buffer
- retainedSlice()
- retainedDuplicate()
- readRetainedSlice()
- Add the new recyclable derived buffer implementations, which has its
own reference count value
- Add ByteBufHolder.retainedDuplicate()
- Add ByteBufHolder.replace(ByteBuf) so that..
- a user can replace the content of the holder in a consistent way
- copy/duplicate/retainedDuplicate() can delegate the holder
construction to replace(ByteBuf)
- Use retainedDuplicate() and retainedSlice() wherever possible
- Miscellaneous:
- Rename DuplicateByteBufTest to DuplicatedByteBufTest (missing 'D')
- Make ReplayingDecoderByteBuf.reject() return an exception instead of
throwing it so that its callers don't need to add dummy return
statement
Result:
Derived buffers are now recycled when created via retainedSlice() and
retainedDuplicate() and derived from a pooled buffer
Motivation:
Some codecs should be considered unstable as these are relative new. For this purpose we should introduce an annotation which these codecs should us to be marked as unstable in terms of API.
Modifications:
- Add UnstableApi annotation and use it on codecs that are not stable
- Move http2.hpack to http2.internal.hpack as it is internal.
Result:
Better document unstable APIs.
Motivation:
Before release 4.1.0.Final we should update all our dependencies.
Modifications:
Update dependencies.
Result:
Up-to-date dependencies used.
Motivation:
- The decoded name should always end with a dot (.), but we currently
strip it, which is incorrect.
- (O) 0 -> "."
- (X) 0 -> ""
- (O) 5 netty 2 io 0 -> "netty.io."
- (X) 5 netty 2 io 0 -> "netty.io"
- The encoded name should end with a null-label, which is a label whose
length is 0, but we currently append an extra NUL, causing FORMERR(1)
on a strict DNS server:
- (O) . -> 0
- (X) . -> 0 0
- (O) netty.io. -> 5 netty 2 io 0
- (X) netty.io. -> 5 netty 2 io 0 0
Modifications:
- Make sure to append '.' when decoding a name.
- Improve index checks so that the decoder can raise
CorruptFrameException instead of IIOBE
- Do not encode extra NUL
- Add more tests
Result:
Robustness and correctness
Motivation:
Zero-length names needs to be "prefixed" by the length as well when encoded into a ByteBuf. Also some servers not correctly prefix these so we should ensure we can workaround this and even decode in such case.
Modifications:
- Always encode the length of the name into the ByteBuf even if its zero-length.
- If there are no readable bytes for the name just asume its an empty name to workaround dns servers that not fully respect the RFC.
Result:
Correctly encode zero-length names and be able to decode empty names even when the rfc is not strictly followed.
Motivation:
Compile crash w/ JDK8:
```
[ERROR]
/Users/slandelle/Documents/dev/workspaces/workspace-ahc2/async-http-clie
nt-project/netty-bp/codec-dns/src/main/java/io/netty/handler/codec/dns/D
nsMessageUtil.java:[176,16] reference to append is ambiguous
both method append(java.lang.String) in java.lang.StringBuilder and
method append(java.lang.StringBuffer) in java.lang.StringBuilder match
```
Modification:
Force type explicitly
Result:
Class compile w/ JDK8
Motivation:
We need to handle the trailing dot in the correct manner when creating DNS questions and responses.
Modifications:
- Add a trailing dot if not given to the hostname when construct a AbstractDnsRecord (this is the same as dig does).
Result:
Correctly handle trailing dots.
Motivation:
Domain name labels must be converted to ASCII and not be longer then 63 chars.
Modifications:
Correctly convert to ASCII which also will enforce the 63 chars length.
Result:
Correctly guard against invalid input.
Motivation:
The current implementation will provide a DnsRawRecord, which, while
containing the host name it resolves to, would require the user to
decode the name using the decode method currently private to
DefaultDnsRecordDecoder, which in fact means copying it.
Modifications:
Introduce DnsPtrRecord, which is a specialization of DnsRecord which
provides a decoded host name.
Result:
PTR Records are much easier to work with, as the name is decoded already.
Motivation:
codec-dns has great function to solve dns packet, but only make a query, not answer query from other client.
i make a change of add two classes to fill last pieces of map, finish the server function.
Modifications:
in this change, add two classes of DatagramDnsQueryDecoder and DatagramDnsResponseEncoder to handle client query, reply answer.
Result:
nothing code change after this commit, except two new classes.
Related issues:
- #3971
- #3973
- #3976
- #4035
Motivation:
1. Previously, DnsNameResolver.query() retried the request query by its
own. It prevents a user from deciding when to retry or stop. It is also
impossible to get the response object whose code is not NOERROR.
2. NameResolver does not have an operation that resolves a host name
into multiple addresses, like InetAddress.getAllByName()
Modifications:
- Changes related with DnsNameResolver.query()
- Make query() not retry
- Move the retry logic to DnsNameResolver.resolve() instead.
- Make query() fail the promise only when I/O error occurred or it
failed to get a response
- Add DnsNameResolverException and use it when query() fails so that
the resolver can give more information about the failure
- query() does not cache anymore.
- Changes related with NameResolver.resolveAll()
- Add NameResolver.resolveAll()
- Add SimpleNameResolver.doResolveAll()
- Changes related with DnsNameResolver.resolve() and resolveAll()
- Make DnsNameResolveContext abstract so that DnsNameResolver can
decide to get single or multiple addresses from it
- Re-implement cache so that the cache works for resolve() and
resolveAll()
- Add 'traceEnabled' property to enable/disable trace information
- Miscellaneous changes
- Use ObjectUtil.checkNotNull() wherever possible
- Add InternetProtocolFamily.addressType() to remove repetitive
switch-case blocks in DnsNameResolver(Context)
- Do not raise an exception when decoding a truncated DNS response
Result:
- Full control over query()
- A user can now retrieve all addresses via (Dns)NameResolver.resolveAll()
- DNS cache works only for resolve() and resolveAll() now.
Motiviation:
The current read loops don't fascilitate reading a maximum amount of bytes. This capability is useful to have more fine grain control over how much data is injested.
Modifications:
- Add a setMaxBytesPerRead(int) and getMaxBytesPerRead() to ChannelConfig
- Add a setMaxBytesPerIndividualRead(int) and getMaxBytesPerIndividualRead to ChannelConfig
- Add methods to RecvByteBufAllocator so that a pluggable scheme can be used to control the behavior of the read loop.
- Modify read loop for all transport types to respect the new RecvByteBufAllocator API
Result:
The ability to control how many bytes are read for each read operation/loop, and a more extensible read loop.
Motivation:
ResourceLeak.close() must be called when a reference-counted resource is
deallocated, but AbstractDnsMessage.deallocate() forgot to call it.
Modifications:
Call ResourceLeak.close() for the tracked AbstractDnsMessage instances
Result:
Fix the false resource leak warnings
Motivation:
There are various known issues in netty-codec-dns:
- Message types are not interfaces, which can make it difficult for a
user to implement his/her own message implementation.
- Some class names and field names do not match with the terms in the
RFC.
- The support for decoding a DNS record was limited. A user had to
encode and decode by him/herself.
- The separation of DnsHeader from DnsMessage was unnecessary, although
it is fine conceptually.
- Buffer leak caused by DnsMessage was difficult to analyze, because the
leak detector tracks down the underlying ByteBuf rather than the
DnsMessage itself.
- DnsMessage assumes DNS-over-UDP.
- To send an EDNS message, a user have to create a new DNS record class
instance unnecessarily.
Modifications:
- Make all message types interfaces and add default implementations
- Rename some classes, properties, and constants to match the RFCs
- DnsResource -> DnsRecord
- DnsType -> DnsRecordType
- and many more
- Remove DnsClass and use an integer to support EDNS better
- Add DnsRecordEncoder/DnsRecordDecoder and their default
implementations
- DnsRecord does not require RDATA to be ByteBuf anymore.
- Add DnsRawRecord as the catch-all record type
- Merge DnsHeader into DnsMessage
- Make ResourceLeakDetector track AbstractDnsMessage
- Remove DnsMessage.sender/recipient properties
- Wrap DnsMessage with AddressedEnvelope
- Add DatagramDnsQuest and DatagramDnsResponse for ease of use
- Rename DnsQueryEncoder to DatagramDnsQueryEncoder
- Rename DnsResponseDecoder to DatagramDnsResponseDecoder
- Miscellaneous changes
- Add StringUtil.TAB
Result:
- Cleaner APi
- Can support DNS-over-TCP more easily in the future
- Reduced memory footprint in the default DnsQuery/Response
implementations
- Better leak tracking for DnsMessages
- Possibility to introduce new DnsRecord types in the future and provide
full record encoder/decoder implementation.
- No unnecessary instantiation for an EDNS pseudo resource record
Motivation:
DnsQueryEncoder does not encode the 'additional resources' section at all, which contains the pseudo-RR as defined in RFC 2671.
Modifications:
- Modify DnsQueryEncoder to encode the additional resources
- Fix a bug in DnsQueryEncoder where an empty name is encoded incorrectly
Result:
A user can send an EDNS query.
Related issue: #1133
Motivation:
There is no support for client socket connections via a proxy server in
Netty.
Modifications:
- Add a new module 'handler-proxy'
- Add ProxyHandler and its subclasses to support SOCKS 4a/5 and HTTP(S)
proxy connections
- Add a full parameterized test for most scenarios
- Clean up pom.xml
Result:
A user can make an outgoing connection via proxy servers with only
trivial effort.
Motivation:
There were two buffer leaks in the codec-dns.
Modifications:
- Fix buffer leak in DnsResponseTest.readResponseTest()
- Correctly release DnsResources on Exception
Result:
No more buffer leaks in the codec-dns module.
Motivation:
DnsResource.duplicate() should return DnsResource and not ByteBufHolder
Modifications:
Change return type from ByteBufHolder to DnsResource
Result:
No need to cast to the correct type when using duplicate()
Related issue: #2688
- DnsClass and DnsType
- Make DnsClass and DnsType implement Comparable
- Optimize the message generation of IllegalArgumentException,
by pre-populating the list of the expected parameters
- Move the static methods up
- Relax the validation rule of DnsClass so that a user can define a
CLASS which is not listed in the RFC 1035
- valueOf(int) does not throw IllegalArgumentException anymore as long
as the specified value is an unsigned short.
- Rename create() and forName() to valueOf() so that they look like a
real enum
- Rename type() and clazz() to intValue() so that they conform to our
naming convention
- Add missing null checks in DnsEntry
Motivation:
DNS class and type were represented as integers rather than an enum or a
similar dedicated value type. This can be a potential source of a
parameter order bug which might be difficult to track down.
Modifications:
Add DnsClass and DnsType to replace integer parameters
Result:
Type safety and less error-proneness
Motivation:
When decoding the NAME field in a DNS Resource Record, DnsResponseDecoder
can raise a NullPointerException if the NAME field contains a loop.
Modification:
Instead of raising an NPE, raise CorruptedFrameException so that the
exception itself has meaning.
Result:
Less confusing when a malformed DNS RR is received
Motivation:
NullPointerException is raised when a DNS response conrains a resource
record whose NAME is empty, which is the case for the authority section.
Modification:
Allow an empty name for DnsEntry. Only disallow an empty name for
DnsQuestion.
Result:
Fixes#2686
Motivation:
DNS packets that contain pointers in a loop will cause
DnsResponseDecoder.readName() to infinite loop.
Modifications:
Fixed DnsResponseDecoder.readName() to detect when packets have loops
and return null instead.
Result:
It is no longer possible to cause Netty to infinite loop by sending it malformed
DNS packets with a loop.
Motivation:
As part of GSOC 2013 we had @mbakkar working on a DNS codec but did not integrate it yet as it needs some cleanup. This commit is based on @mbakkar's work and provide the codec for DNS.
Modifications:
Add DNS codec
Result:
Reusable DNS codec will be included in netty.
This PR also includes a AsynchronousDnsResolver which allows to resolve DNS entries in a non blocking way by make use
of the dns codec and netty transport itself.
