andreacavalli/netty5
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Get uploaded size while upload is in progress

Browse Source 
Proposal to fix issue #3636

Motivations:
Currently, while adding the next buffers to the decoder
(`decoder.offer()`), there is no way to access to the current HTTP
object being decoded since it can only be available currently once fully
decoded by `decoder.hasNext()`.
Some could want to know the progression on the overall transfer but also
per HTTP object.
While overall progression could be done using (if available) the global
Content-Length of the request and taking into account each HttpContent
size, the per HttpData object progression is unknown.

Modifications:
1) For HTTP object, `AbstractHttpData` has 2 protected properties named
`definedSize` and `size`, respectively the supposely final size and the
current (decoded until now) size.
This provides a new method `definedSize()` to get the current value for
`definedSize`. The `size` attribute is reachable by the `length()`
method.

Note however there are 2 different ways that currently managed the
`definedSize`:
a) `Attribute`: it is reset each time the value is less than actual
(when a buffer is added, the value is increased) since the final length
is not known (no Content-Length)
b) `FileUpload`: it is set at startup from the lengh provided

So these differences could lead in wrong perception;
a) `Attribute`: definedSize = size always
b) `FileUpload`: definedSize >= size always

Therefore the comment tries to explain clearly the different behaviors.

2) In the InterfaceHttpPostRequestDecoder (and the derived classes), I
add a new method: `decoder.currentPartialHttpData()` which will return a
`InterfaceHttpData` (if any) as the current `Attribute` or `FileUpload`
(the 2 generic types), which will allow then the programmer to check
according to the real type (instance of) the 2 methods `definedSize()`
and `length()`.

This method check if currentFileUpload or currentAttribute are null and
returns the one (only one could be not null) that is not null.

Note that if this method returns null, it might mean 2 situations:
a) the last `HttpData` (whatever attribute or file upload) is already
finished and therefore accessible through `next()`
b) there is not yet any `HttpData` in decoding (body not yet parsed for
instance)

Result:
The developper has more access and therefore control on the current
upload.
The coding from developper side could looks like in the example in
HttpUloadServerHandler.
This commit is contained in:
Frederic Bregier 2015-04-18 15:07:38 +02:00 committed by Norman Maurer
parent 3308510bc9
commit c4d861844c
15 changed files with 184 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
codec-http/src/main/java/io/netty/handler/codec/http/multipart/AbstractHttpData.java
View File

@ -105,6 +105,11 @@ public abstract class AbstractHttpData extends AbstractReferenceCounted implemen
return size;
}
@Override
public long definedLength() {
return definedSize;
}
@Override
public ByteBuf content() {
try {

21
codec-http/src/main/java/io/netty/handler/codec/http/multipart/DefaultHttpDataFactory.java
View File

@ -141,6 +141,27 @@ public class DefaultHttpDataFactory implements HttpDataFactory {
return attribute;
}
@Override
public Attribute createAttribute(HttpRequest request, String name, long definedSize) {
if (useDisk) {
Attribute attribute = new DiskAttribute(name, definedSize, charset);
attribute.setMaxSize(maxSize);
List<HttpData> fileToDelete = getList(request);
fileToDelete.add(attribute);
return attribute;
}
if (checkSize) {
Attribute attribute = new MixedAttribute(name, definedSize, minSize, charset);
attribute.setMaxSize(maxSize);
List<HttpData> fileToDelete = getList(request);
fileToDelete.add(attribute);
return attribute;
}
MemoryAttribute attribute = new MemoryAttribute(name, definedSize);
attribute.setMaxSize(maxSize);
return attribute;
}
/**
* Utility method
*/

8
codec-http/src/main/java/io/netty/handler/codec/http/multipart/DiskAttribute.java
View File

@ -43,10 +43,18 @@ public class DiskAttribute extends AbstractDiskHttpData implements Attribute {
this(name, HttpConstants.DEFAULT_CHARSET);
}
public DiskAttribute(String name, long definedSize) {
this(name, definedSize, HttpConstants.DEFAULT_CHARSET);
}
public DiskAttribute(String name, Charset charset) {
super(name, charset, 0);
}
public DiskAttribute(String name, long definedSize, Charset charset) {
super(name, charset, definedSize);
}
public DiskAttribute(String name, String value) throws IOException {
this(name, value, HttpConstants.DEFAULT_CHARSET);
}

16
codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpData.java
View File

@ -98,6 +98,22 @@ public interface HttpData extends InterfaceHttpData, ByteBufHolder {
*/
long length();
/**
* Returns the defined length of the HttpData.
*
* If no Content-Length is provided in the request, the defined length is
* always 0 (whatever during decoding or in final state).
*
* If Content-Length is provided in the request, this is this given defined length.
* This value does not change, whatever during decoding or in the final state.
*
* This method could be used for instance to know the amount of bytes transmitted for
* one particular HttpData, for example one {@link FileUpload} or any known big {@link Attribute}.
*
* @return the defined length of the HttpData
*/
long definedLength();
/**
* Deletes the underlying storage for a file item, including deleting any
* associated temporary disk file.

8
codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpDataFactory.java
View File

@ -37,6 +37,14 @@ public interface HttpDataFactory {
*/
Attribute createAttribute(HttpRequest request, String name);
/**
* @param request associated request
* @param name name of the attribute
* @param definedSize defined size from request for this attribute
* @return a new Attribute with no value
*/
Attribute createAttribute(HttpRequest request, String name, long definedSize);
/**
* @param request associated request
* @return a new Attribute

29
codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java
View File

@ -393,6 +393,15 @@ public class HttpPostMultipartRequestDecoder implements InterfaceHttpPostRequest
return null;
}
@Override
public InterfaceHttpData currentPartialHttpData() {
if (currentFileUpload != null) {
return currentFileUpload;
} else {
return currentAttribute;
}
}
/**
* This getMethod will parse as much as possible data and fill the list and map
*
@ -501,9 +510,25 @@ public class HttpPostMultipartRequestDecoder implements InterfaceHttpPostRequest
}
Attribute nameAttribute = currentFieldAttributes.get(HttpHeaderValues.NAME);
if (currentAttribute == null) {
Attribute lengthAttribute = currentFieldAttributes
.get(HttpHeaderNames.CONTENT_LENGTH);
long size;
try {
currentAttribute = factory.createAttribute(request,
cleanString(nameAttribute.getValue()));
size = lengthAttribute != null? Long.parseLong(lengthAttribute
.getValue()) : 0L;
} catch (IOException e) {
throw new ErrorDataDecoderException(e);
} catch (NumberFormatException e) {
size = 0;
}
try {
if (size > 0) {
currentAttribute = factory.createAttribute(request,
cleanString(nameAttribute.getValue()), size);
} else {
currentAttribute = factory.createAttribute(request,
cleanString(nameAttribute.getValue()));
}
} catch (NullPointerException e) {
throw new ErrorDataDecoderException(e);
} catch (IllegalArgumentException e) {

5
codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoder.java
View File

@ -238,6 +238,11 @@ public class HttpPostRequestDecoder implements InterfaceHttpPostRequestDecoder {
return decoder.next();
}
@Override
public InterfaceHttpData currentPartialHttpData() {
return decoder.currentPartialHttpData();
}
@Override
public void destroy() {
decoder.destroy();

6
codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostRequestEncoder.java
View File

@ -507,6 +507,9 @@ public class HttpPostRequestEncoder implements ChunkedInput<HttpContent> {
Attribute attribute = (Attribute) data;
internal.addValue(HttpHeaderNames.CONTENT_DISPOSITION + ": " + HttpHeaderValues.FORM_DATA + "; "
+ HttpHeaderValues.NAME + "=\"" + attribute.getName() + "\"\r\n");
// Add Content-Length: xxx
internal.addValue(HttpHeaderNames.CONTENT_LENGTH + ": " +
attribute.length() + "\r\n");
Charset localcharset = attribute.getCharset();
if (localcharset != null) {
// Content-Type: text/plain; charset=charset
@ -655,6 +658,9 @@ public class HttpPostRequestEncoder implements ChunkedInput<HttpContent> {
+ HttpHeaderValues.NAME + "=\"" + fileUpload.getName() + "\"; "
+ HttpHeaderValues.FILENAME + "=\"" + fileUpload.getFilename() + "\"\r\n");
}
// Add Content-Length: xxx
internal.addValue(HttpHeaderNames.CONTENT_LENGTH + ": " +
fileUpload.length() + "\r\n");
// Content-Type: image/gif
// Content-Type: text/plain; charset=ISO-8859-1
// Content-Transfer-Encoding: binary

5
codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostStandardRequestDecoder.java
View File

@ -350,6 +350,11 @@ public class HttpPostStandardRequestDecoder implements InterfaceHttpPostRequestD
return null;
}
@Override
public InterfaceHttpData currentPartialHttpData() {
return currentAttribute;
}
/**
* This getMethod will parse as much as possible data and fill the list and map
*

10
codec-http/src/main/java/io/netty/handler/codec/http/multipart/InterfaceHttpPostRequestDecoder.java
View File

@ -121,6 +121,16 @@ public interface InterfaceHttpPostRequestDecoder {
*/
InterfaceHttpData next();
/**
* Returns the current InterfaceHttpData if currently in decoding status,
* meaning all data are not yet within, or null if there is no InterfaceHttpData
* currently in decoding status (either because none yet decoded or none currently partially
* decoded). Full decoded ones are accessible through hasNext() and next() methods.
*
* @return the current InterfaceHttpData if currently in decoding status or null if none.
*/
InterfaceHttpData currentPartialHttpData();
/**
* Destroy the {@link InterfaceHttpPostRequestDecoder} and release all it resources. After this method
* was called it is not possible to operate on it anymore.

8
codec-http/src/main/java/io/netty/handler/codec/http/multipart/MemoryAttribute.java
View File

@ -33,10 +33,18 @@ public class MemoryAttribute extends AbstractMemoryHttpData implements Attribute
this(name, HttpConstants.DEFAULT_CHARSET);
}
public MemoryAttribute(String name, long definedSize) {
this(name, definedSize, HttpConstants.DEFAULT_CHARSET);
}
public MemoryAttribute(String name, Charset charset) {
super(name, charset, 0);
}
public MemoryAttribute(String name, long definedSize, Charset charset) {
super(name, charset, definedSize);
}
public MemoryAttribute(String name, String value) throws IOException {
this(name, value, HttpConstants.DEFAULT_CHARSET); // Attribute have no default size
}

22
codec-http/src/main/java/io/netty/handler/codec/http/multipart/MixedAttribute.java
View File

@ -36,11 +36,20 @@ public class MixedAttribute implements Attribute {
this(name, limitSize, HttpConstants.DEFAULT_CHARSET);
}
public MixedAttribute(String name, long definedSize, long limitSize) {
this(name, definedSize, limitSize, HttpConstants.DEFAULT_CHARSET);
}
public MixedAttribute(String name, long limitSize, Charset charset) {
this.limitSize = limitSize;
attribute = new MemoryAttribute(name, charset);
}
public MixedAttribute(String name, long definedSize, long limitSize, Charset charset) {
this.limitSize = limitSize;
attribute = new MemoryAttribute(name, definedSize, charset);
}
public MixedAttribute(String name, String value, long limitSize) {
this(name, value, limitSize, HttpConstants.DEFAULT_CHARSET);
}
@ -91,7 +100,7 @@ public class MixedAttribute implements Attribute {
checkSize(attribute.length() + buffer.readableBytes());
if (attribute.length() + buffer.readableBytes() > limitSize) {
DiskAttribute diskAttribute = new DiskAttribute(attribute
.getName());
.getName(), attribute.definedLength());
diskAttribute.setMaxSize(maxSize);
if (((MemoryAttribute) attribute).getByteBuf() != null) {
diskAttribute.addContent(((MemoryAttribute) attribute)
@ -148,6 +157,11 @@ public class MixedAttribute implements Attribute {
return attribute.length();
}
@Override
public long definedLength() {
return attribute.definedLength();
}
@Override
public boolean renameTo(File dest) throws IOException {
return attribute.renameTo(dest);
@ -164,7 +178,7 @@ public class MixedAttribute implements Attribute {
if (buffer.readableBytes() > limitSize) {
if (attribute instanceof MemoryAttribute) {
// change to Disk
attribute = new DiskAttribute(attribute.getName());
attribute = new DiskAttribute(attribute.getName(), attribute.definedLength());
attribute.setMaxSize(maxSize);
}
}
@ -177,7 +191,7 @@ public class MixedAttribute implements Attribute {
if (file.length() > limitSize) {
if (attribute instanceof MemoryAttribute) {
// change to Disk
attribute = new DiskAttribute(attribute.getName());
attribute = new DiskAttribute(attribute.getName(), attribute.definedLength());
attribute.setMaxSize(maxSize);
}
}
@ -188,7 +202,7 @@ public class MixedAttribute implements Attribute {
public void setContent(InputStream inputStream) throws IOException {
if (attribute instanceof MemoryAttribute) {
// change to Disk even if we don't know the size
attribute = new DiskAttribute(attribute.getName());
attribute = new DiskAttribute(attribute.getName(), attribute.definedLength());
attribute.setMaxSize(maxSize);
}
attribute.setContent(inputStream);

5
codec-http/src/main/java/io/netty/handler/codec/http/multipart/MixedFileUpload.java
View File

@ -151,6 +151,11 @@ public class MixedFileUpload implements FileUpload {
return fileUpload.length();
}
@Override
public long definedLength() {
return fileUpload.definedLength();
}
@Override
public boolean renameTo(File dest) throws IOException {
return fileUpload.renameTo(dest);

10
codec-http/src/test/java/io/netty/handler/codec/http/multipart/HttpPostRequestEncoderTest.java
View File

@ -52,12 +52,14 @@ public class HttpPostRequestEncoderTest {
String expected = "--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"foo\"" + "\r\n" +
CONTENT_LENGTH + ": 3" + "\r\n" +
CONTENT_TYPE + ": text/plain; charset=UTF-8" + "\r\n" +
"\r\n" +
"bar" +
"\r\n" +
"--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"quux\"; filename=\"file-01.txt\"" + "\r\n" +
CONTENT_LENGTH + ": " + file1.length() + "\r\n" +
CONTENT_TYPE + ": text/plain" + "\r\n" +
CONTENT_TRANSFER_ENCODING + ": binary" + "\r\n" +
"\r\n" +
@ -88,6 +90,7 @@ public class HttpPostRequestEncoderTest {
String expected = "--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"foo\"" + "\r\n" +
CONTENT_LENGTH + ": 3" + "\r\n" +
CONTENT_TYPE + ": text/plain; charset=UTF-8" + "\r\n" +
"\r\n" +
"bar" + "\r\n" +
@ -97,6 +100,7 @@ public class HttpPostRequestEncoderTest {
"\r\n" +
"--" + multipartMixedBoundary + "\r\n" +
CONTENT_DISPOSITION + ": attachment; filename=\"file-02.txt\"" + "\r\n" +
CONTENT_LENGTH + ": " + file1.length() + "\r\n" +
CONTENT_TYPE + ": text/plain" + "\r\n" +
CONTENT_TRANSFER_ENCODING + ": binary" + "\r\n" +
"\r\n" +
@ -104,6 +108,7 @@ public class HttpPostRequestEncoderTest {
"\r\n" +
"--" + multipartMixedBoundary + "\r\n" +
CONTENT_DISPOSITION + ": attachment; filename=\"file-02.txt\"" + "\r\n" +
CONTENT_LENGTH + ": " + file2.length() + "\r\n" +
CONTENT_TYPE + ": text/plain" + "\r\n" +
CONTENT_TRANSFER_ENCODING + ": binary" + "\r\n" +
"\r\n" +
@ -135,17 +140,20 @@ public class HttpPostRequestEncoderTest {
String expected = "--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"foo\"" + "\r\n" +
CONTENT_LENGTH + ": 3" + "\r\n" +
CONTENT_TYPE + ": text/plain; charset=UTF-8" + "\r\n" +
"\r\n" +
"bar" + "\r\n" +
"--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"quux\"; filename=\"file-01.txt\"" + "\r\n" +
CONTENT_LENGTH + ": " + file1.length() + "\r\n" +
CONTENT_TYPE + ": text/plain" + "\r\n" +
CONTENT_TRANSFER_ENCODING + ": binary" + "\r\n" +
"\r\n" +
"File 01" + StringUtil.NEWLINE + "\r\n" +
"--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"quux\"; filename=\"file-02.txt\"" + "\r\n" +
CONTENT_LENGTH + ": " + file2.length() + "\r\n" +
CONTENT_TYPE + ": text/plain" + "\r\n" +
CONTENT_TRANSFER_ENCODING + ": binary" + "\r\n" +
"\r\n" +
@ -174,12 +182,14 @@ public class HttpPostRequestEncoderTest {
String expected = "--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"foo\"" + "\r\n" +
CONTENT_LENGTH + ": 3" + "\r\n" +
CONTENT_TYPE + ": text/plain; charset=UTF-8" + "\r\n" +
"\r\n" +
"bar" +
"\r\n" +
"--" + multipartDataBoundary + "\r\n" +
CONTENT_DISPOSITION + ": form-data; name=\"quux\"; filename=\"file-01.txt\"" + "\r\n" +
CONTENT_LENGTH + ": " + file1.length() + "\r\n" +
CONTENT_TYPE + ": text/plain" + "\r\n" +
CONTENT_TRANSFER_ENCODING + ": binary" + "\r\n" +
"\r\n" +

32
example/src/main/java/io/netty/example/http/upload/HttpUploadServerHandler.java
View File

@ -42,6 +42,7 @@ import io.netty.handler.codec.http.multipart.DefaultHttpDataFactory;
import io.netty.handler.codec.http.multipart.DiskAttribute;
import io.netty.handler.codec.http.multipart.DiskFileUpload;
import io.netty.handler.codec.http.multipart.FileUpload;
import io.netty.handler.codec.http.multipart.HttpData;
import io.netty.handler.codec.http.multipart.HttpDataFactory;
import io.netty.handler.codec.http.multipart.HttpPostRequestDecoder;
import io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.EndOfDataDecoderException;
@ -70,6 +71,8 @@ public class HttpUploadServerHandler extends SimpleChannelInboundHandler<HttpObj
private boolean readingChunks;
private HttpData partialContent;
private final StringBuilder responseContent = new StringBuilder();
private static final HttpDataFactory factory =
@ -216,6 +219,11 @@ public class HttpUploadServerHandler extends SimpleChannelInboundHandler<HttpObj
while (decoder.hasNext()) {
InterfaceHttpData data = decoder.next();
if (data != null) {
// check if current HttpData is a FileUpload and previously set as partial
if (partialContent == data) {
logger.info(" 100% (FinalSize: " + partialContent.length() + ")");
partialContent = null;
}
try {
// new value
writeHttpData(data);
@ -224,6 +232,30 @@ public class HttpUploadServerHandler extends SimpleChannelInboundHandler<HttpObj
}
}
}
// Check partial decoding for a FileUpload
InterfaceHttpData data = decoder.currentPartialHttpData();
if (data != null) {
StringBuilder builder = new StringBuilder();
if (partialContent == null) {
partialContent = (HttpData) data;
if (partialContent instanceof FileUpload) {
builder.append("Start FileUpload: ")
.append(((FileUpload) partialContent).getFilename()).append(" ");
} else {
builder.append("Start Attribute: ")
.append(partialContent.getName()).append(" ");
}
builder.append("(DefinedSize: ").append(partialContent.definedLength()).append(")");
}
if (partialContent.definedLength() > 0) {
builder.append(" ").append(partialContent.length() * 100 / partialContent.definedLength())
.append("% ");
logger.info(builder.toString());
} else {
builder.append(" ").append(partialContent.length()).append(" ");
logger.info(builder.toString());
}
}
} catch (EndOfDataDecoderException e1) {
// end
responseContent.append("\r\n\r\nEND OF CONTENT CHUNK BY CHUNK\r\n\r\n");