andreacavalli/netty5
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

OpenSslEngine return NEED_WRAP if the destination buffered filled

Browse Source 
Motivation:
If the destination buffer is completely filled during a call to OpenSslEngine#wrap(..) we may return NEED_UNWRAP because there is no data pending in the SSL buffers. However during a handshake if the SSL buffers were just drained, and filled up the destination buffer it is possible OpenSSL may produce more data on the next call to SSL_write. This means we should keep trying to call SSL_write as long as the destination buffer is filled and only return NEED_UNWRAP when the destination buffer is not full and there is no data pending in OpenSSL's buffers.

Modifications:
- If the handshake produces data in OpenSslEngine#wrap(..) we should return NEED_WRAP if the destination buffer is completely filled

Result:
OpenSslEngine returns the correct handshake status from wrap().
Fixes https://github.com/netty/netty/issues/6796.
This commit is contained in:
Scott Mitchell 2017-06-01 15:40:05 -07:00
parent 270e9d66c5
commit 24f801c7d1
1 changed files with 9 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
handler/src/main/java/io/netty/handler/ssl/ReferenceCountedOpenSslEngine.java
View File

@ -571,8 +571,11 @@ public class ReferenceCountedOpenSslEngine extends SSLEngine implements Referenc
bytesProduced = bioLengthBefore - SSL.bioLengthByteBuffer(networkBIO);
if (bytesProduced > 0) {
// It's important we call this before wrapStatus() as wrapStatus() may shutdown the engine.
// If we have filled up the dst buffer and we have not finished the handshake we should try to
// wrap again. Otherwise we should only try to wrap again if there is still data pending in
// SSL buffers.
return newResult(mayFinishHandshake(status != FINISHED ?
bytesProduced == bioLengthBefore ? NEED_WRAP :
getHandshakeStatus(SSL.bioLengthNonApplication(networkBIO)) : FINISHED),
0, bytesProduced);
}
@ -648,8 +651,12 @@ public class ReferenceCountedOpenSslEngine extends SSLEngine implements Referenc
bytesProduced += bioLengthBefore - SSL.bioLengthByteBuffer(networkBIO);
// If we have filled up the dst buffer and we have not finished the handshake we should
// try to wrap again. Otherwise we should only try to wrap again if there is still data
// pending in SSL buffers.
SSLEngineResult.HandshakeStatus hs = mayFinishHandshake(
status != FINISHED ? getHandshakeStatus(SSL.bioLengthNonApplication(networkBIO))
status != FINISHED ? bytesProduced == dst.remaining() ? NEED_WRAP
: getHandshakeStatus(SSL.bioLengthNonApplication(networkBIO))
: FINISHED);
return newResult(hs, bytesConsumed, bytesProduced);
}