andreacavalli/netty5
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

OpenSslEngine wrap may not consume all data

Browse Source 
Motivation:
If the OpenSslEngine has bytes pending in the non-application buffer and also generates wrapped data during the handshake then the handshake data will be missed. This will lead to a handshake stall and eventually timeout. This can occur if the non-application buffer becomes full due to a large certificate/hello message.

Modification:
- ReferenceCountedOpenSslEngine should not assume if no data is flushed from the non-application buffer that no data will be produced by the handshake.

Result:
New unit tests with larger certificate chains don't fail.
This commit is contained in:
Scott Mitchell 2017-02-14 17:24:03 -08:00 committed by Norman Maurer
parent c521c72178
commit d60e37cb3d
1 changed files with 23 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
handler/src/main/java/io/netty/handler/ssl/ReferenceCountedOpenSslEngine.java
View File

@ -483,12 +483,13 @@ public class ReferenceCountedOpenSslEngine extends SSLEngine implements Referenc
// Flush any data that may have been written implicitly during the handshake by OpenSSL.
bytesProduced = SSL.bioFlushByteBuffer(networkBIO);
// Check to see if the engine wrote data into the network BIO.
if (bytesProduced > 0) {
if (bytesProduced > 0 && handshakeException != null) {
// We produced / consumed some data during the handshake, signal back to the caller.
// If there is a handshake exception and we have produced data, we should send the data before
// we allow handshake() to throw the handshake exception.
if (handshakeException == null) {
return newResult(NEED_WRAP, 0, bytesProduced);
}
status = handshake();
if (renegotiationPending && status == FINISHED) {
@ -506,15 +507,12 @@ public class ReferenceCountedOpenSslEngine extends SSLEngine implements Referenc
// we may have freed up space by flushing above.
bytesProduced = bioLengthBefore - SSL.bioLengthByteBuffer(networkBIO);
if (bytesProduced > 0) {
// It's important we call this before wrapStatus() as wrapStatus() may shutdown the engine.
return newResult(mayFinishHandshake(status != FINISHED ?
getHandshakeStatus(SSL.bioLengthNonApplication(networkBIO)) : FINISHED),
0, bytesProduced);
}
return newResult(NEED_WRAP, 0, bytesProduced);
}
status = handshake();
if (status == NEED_UNWRAP) {
// Signal if the outbound is done or not.