andreacavalli/netty5
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Correctly return NEED_WRAP if we produced some data even if we could not consume any during SSLEngine.wrap(...) (#10396)

Browse Source 
Motivation:

At the moment we may report BUFFER_OVERFLOW when wrap(...) fails to consume data but still prodce some. This is not correct and we should better report NEED_WRAP as we already have produced some data (for example tickets). This way the user will try again without increasing the buffer size which is more correct and may reduce the number of allocations

Modifications:

Return NEED_WRAP when we produced some data but not consumed any.

Result:

Fix ReferenceCountedOpenSslEngine.wrap(...) state machine
This commit is contained in:
Norman Maurer 2020-07-09 08:52:07 +02:00
parent fca62510f3
commit 165ade5d9f
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
handler/src/main/java/io/netty/handler/ssl/ReferenceCountedOpenSslEngine.java
View File

@ -929,6 +929,11 @@ public class ReferenceCountedOpenSslEngine extends SSLEngine implements Referenc
// In practice this means the destination buffer doesn't have enough space for OpenSSL
// to write encrypted data to. This is an OVERFLOW condition.
// [1] https://www.openssl.org/docs/manmaster/ssl/SSL_write.html
if (bytesProduced > 0) {
// If we produced something we should report this back and let the user call
// wrap again.
return newResult(NEED_WRAP, bytesConsumed, bytesProduced);
}
return newResult(BUFFER_OVERFLOW, status, bytesConsumed, bytesProduced);
} else if (sslError == SSL.SSL_ERROR_WANT_X509_LOOKUP ||
sslError == SSL.SSL_ERROR_WANT_CERTIFICATE_VERIFY ||