问题记录:minio上传文件大于2G,文件被截断

1,204 阅读2分钟

问题

隔壁新建的开发组发生了一个bug,文件上传接口,上传2G大小文件时,文件被截断为2G整。

分析

因为配置的是MinioClient,原本设置地上传文件大小为100m,后来有需求要增加文件大小,来不及开发切片上传,想要临时更改配置撑过用户使用阶段,所以更改了服务器文件大小限制、nginx文件大小限制,结果发现大于2G的文件被截断为2G。自己手动存到minio的文件不受影响,故开始排查

1.nginx文件大小限制

Nginx 限制文件大小可以通过 client_max_body_size 指令来设置,该指令通常在 http、server 或
location 块中设置,如果不设置,默认上传大小为1M。

client_max_body_size 10M;

2.springboot文件大小限制

multipart参数接收文件的大小限制

spring.servlet.multipart.max-file-size=50MB
spring.servlet.multipart.max-request-size=50MB

3.minio文件大小限制

minio是不限制文件大小的

结果

经过排查,发现文件的大小在上传minio的时候被固定为2G,原因在于


try {
    this.client.putObject((PutObjectArgs)((PutObjectArgs.Builder)((PutObjectArgs.Builder)PutObjectArgs.builder().bucket(this.getBucketName(bucketName))).object(fileName)).stream(stream, (long)stream.available(), -1L).contentType(contentType).build());
} catch (Exception var16) {
    var16.printStackTrace();
} finally {
    try {
        if (stream != null) {
            stream.close();
        }
    } catch (Exception var15) {
        var15.printStackTrace();
    }

}

上述代码中的(long)stream.available(),源码为

/**
 * Returns an estimate of the number of bytes that can be read (or
 * skipped over) from this input stream without blocking by the next
 * invocation of a method for this input stream. The next invocation
 * might be the same thread or another thread.  A single read or skip of this
 * many bytes will not block, but may read or skip fewer bytes.
 *
 * <p> Note that while some implementations of {@code InputStream} will return
 * the total number of bytes in the stream, many will not.  It is
 * never correct to use the return value of this method to allocate
 * a buffer intended to hold all data in this stream.
 *
 * <p> A subclass' implementation of this method may choose to throw an
 * {@link IOException} if this input stream has been closed by
 * invoking the {@link #close()} method.
 *
 * <p> The {@code available} method for class {@code InputStream} always
 * returns {@code 0}.
 *
 * <p> This method should be overridden by subclasses.
 *
 * @return     an estimate of the number of bytes that can be read (or skipped
 *             over) from this input stream without blocking or {@code 0} when
 *             it reaches the end of the input stream.
 * @exception  IOException if an I/O error occurs.
 */
public int available() throws IOException {
    return 0;
}

可以看到方法返回的值为int,然后强转为long当作文件大小去上传参数使用,那么int的最大值是等于2G的,问题就在这里

210241024*1024 = 2,147,483,648 > int范围