前言

前几个月写了一篇minio学习记录，最近在深入学习这个轻便快捷的对象存储框架，发现了一个好玩的功能，分片上传！！ 但是鉴于自己还没接触过这技术，所以到处去找资料，而阿里云OSS的技术文档里面刚刚好有这一点的代码示例，这无疑给我提供了不错的思路。阿里云对象存储文档。

思路

原思路

前端上传文件到web后台。
后端对文件进行切割，并且记录切割段数。
后端调用minio上传api。
等待分片全部上传后再调用合并文件api进行文件合并。

但是这样后端做了太多事了，这样会占用后端的资源，而阿里云那些服务商是可以从后端得到OSS的url返回前端，前端直接上传然后回调的。

优化后的思路

前端对文件进行切片，并且记录切片总数
访问后端预上传接口，该接口仅仅处理目标文件上传url并且返回给前端，没有太多的资源占用。
前端获取到返回的预上传url后，循环分片进行上传。
在前端上传完分片后，请求后端文件合并接口对目标分片进行合并。

这样可以最大限度的利用前端的性能，而不是只发请求到后端，并且实现了前端直接对接minio服务器的功能。

实现

源码关键API

createMultipartUpload    //创建文件上传id
getPresignedObjectUrl    //获取文件预上传url
listParts                //获取分片上传列表
completeMultipartUpload  // 合并分片文件

依赖

核心依赖只有这个minio的依赖，其他的和普通springboot项目依赖差别不大。

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.3.3</version>
</dependency>

注意：这里用的是minio的8.x版本，非8.x版本api会有所不同！！！

配置

minio:
  endpoint: xxx          #minio服务器地址
  accessKey: xxx         #你的minio服务器accessKey
  secretKey: xxx         #你的minio服务器secretKey
  bucketName: xxx        #目标桶名
  downloadUri: xxx       #你的minio服务器地址，如果有nginx转发或分布式配置请根据你的实际项目进行替换
  path: xxx              #自定义地址

踩坑配置

@Configuration
public class minioConfig {
    @Value("${minio.endpoint}")
    private String endpoint;
    @Value("${minio.accessKey}")
    private String accessKey;
    @Value("${minio.secretKey}")
    private String secretKey;

    @Bean
    public MinioClient minioClient(){
        return MinioClient.builder().endpoint(endpoint)
                .credentials(accessKey,secretKey)
                .build();
    }
}

此处配置是可以成功调用API的，但是上面的几个关键API会调用失败，比如createMultipartUpload

这就很纳闷了，知道了关键API但是调用不了，怎么回事呢？于是我点进MinioClient的源码一探究竟。

在默认的MinioClient类里面我们没有找到createMultipartUpload方法，但是我注意到，这个类继承了S3Base这个类，估计createMultipartUpload方法就在里面了，让我们一起去看看。

果然！那现在知道了原因所在了就好办了，答案只有一个，继承MinioClient类！！你问为什么不直接继承S3Base这个类？因为MinioClient是S3Base的子类，它有属于它自己的一些方法，如果只继承S3Base类就意味着放弃某些功能。

分片正确配置

继承MinioClient类

public class MyMinioClient extends MinioClient {

    public MyMinioClient(MinioClient client) {
        super(client);
    }

    /**
     * 创建分片上传请求
     *
     * @param bucketName       存储桶
     * @param region           区域
     * @param objectName       对象名
     * @param headers          消息头
     * @param extraQueryParams 额外查询参数
     */
    @SneakyThrows
    @Override
    public CreateMultipartUploadResponse createMultipartUpload(String bucketName, String region, String objectName, Multimap<String, String> headers, Multimap<String, String> extraQueryParams) throws NoSuchAlgorithmException, InsufficientDataException, IOException, XmlParserException, ErrorResponseException, InvalidResponseException {
        return super.createMultipartUpload(bucketName, region, objectName, headers, extraQueryParams);
    }

    /**
     * 完成分片上传，执行合并文件
     *
     * @param bucketName       存储桶
     * @param region           区域
     * @param objectName       对象名
     * @param uploadId         上传ID
     * @param parts            分片
     * @param extraHeaders     额外消息头
     * @param extraQueryParams 额外查询参数
     */
    @SneakyThrows
    @Override
    public ObjectWriteResponse completeMultipartUpload(String bucketName, String region, String objectName, String uploadId, Part[] parts, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws NoSuchAlgorithmException, InsufficientDataException, IOException, XmlParserException, ErrorResponseException, InvalidResponseException {
        return super.completeMultipartUpload(bucketName, region, objectName, uploadId, parts, extraHeaders, extraQueryParams);
    }

    /**
     * 查询分片数据
     *
     * @param bucketName       存储桶
     * @param region           区域
     * @param objectName       对象名
     * @param uploadId         上传ID
     * @param extraHeaders     额外消息头
     * @param extraQueryParams 额外查询参数
     */
    @SneakyThrows
    public ListPartsResponse listMultipart(String bucketName, String region, String objectName, Integer maxParts, Integer partNumberMarker, String uploadId, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws NoSuchAlgorithmException, InsufficientDataException, IOException{
        return super.listParts(bucketName, region, objectName, maxParts, partNumberMarker, uploadId, extraHeaders, extraQueryParams);
    }

配置类

@Configuration
@EnableConfigurationProperties(MinioProperties.class)
public class MinioConfiguration {

    @Autowired
    private MinioProperties minioProperties;

    @Bean
    public MyMinioClient minioClient() {
        MinioClient minioClient = MinioClient.builder()
                .endpoint(minioProperties.getEndpoint())
                .credentials(minioProperties.getAccessKey(), minioProperties.getSecretKey())
                .build();
        return new MyMinioClient(minioClient);
    }
}

这样就可以通过注入MyMinioClient调用createMultipartUpload等方法。

封装的核心工具类

@Slf4j
@Component
public class MinioCoreUtils {

    @Autowired
    private MyMinioClient client;
    @Autowired
    public MinioProperties minioProperties;


    /**
     * 上传单个文件
     * @param multipartFile
     * @return
     */
    public FileUploadResponse uploadFile(MultipartFile multipartFile) throws IOException, ServerException, InsufficientDataException, ErrorResponseException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {

        boolean found = client.bucketExists(BucketExistsArgs.builder().bucket(minioProperties.getBucketName()).build());
        if (!found) {
            log.info("create bucket: [{}]", minioProperties.getBucketName());
            client.makeBucket(MakeBucketArgs.builder().bucket(minioProperties.getBucketName()).build());
        } else {
            log.info("bucket '{}' already exists.", minioProperties.getBucketName());
        }

        try (InputStream inputStream = multipartFile.getInputStream()) {

            // 上传文件的名称
            String uploadName =  UUID.fastUUID().toString(true) + "_" + DateUtil.format(new Date(), "yyyy_MM_dd_HH_mm_ss") + "_" +
                    multipartFile.getOriginalFilename().substring(multipartFile.getOriginalFilename().lastIndexOf("."));

            // PutObjectOptions，上传配置(文件大小，内存中文件分片大小)
            PutObjectArgs putObjectOptions = PutObjectArgs.builder()
                    .bucket(minioProperties.getBucketName())
                    .object(uploadName)
                    .contentType(multipartFile.getContentType())
                    .stream(inputStream, multipartFile.getSize(), -1)
                    .build();
            client.putObject(putObjectOptions);

            final String url = minioProperties.getEndpoint() + "/" + minioProperties.getBucketName() + "/" + UriUtils.encode(uploadName, StandardCharsets.UTF_8);

            // 返回访问路径
            return FileUploadResponse.builder()
                    .uploadName(uploadName)
                    .url(url)
                    .realName(multipartFile.getOriginalFilename())
                    .size(multipartFile.getSize())
                    .bucket(minioProperties.getBucketName())
                    .build();
        }
    }
        /**生成上传id
         * 
         * @param multipartUploadCreate
         * @return
         */
    public CreateMultipartUploadResponse uploadId(MultipartUploadCreate multipartUploadCreate){
        try {
            return client.createMultipartUpload(multipartUploadCreate.getBucketName(), multipartUploadCreate.getRegion(), multipartUploadCreate.getObjectName(), multipartUploadCreate.getHeaders(), multipartUploadCreate.getExtraQueryParams());
        } catch (Exception e) {
            log.error("获取上传编号失败", e);
            throw BusinessException.newBusinessException(ResultCode.KNOWN_ERROR.getCode(), e.getMessage());
        }
    }



    /**
     * 合并分片
     * @param multipartUploadCreate
     * @return
     */
    public ObjectWriteResponse completeMultipartUpload(MultipartUploadCreate multipartUploadCreate) {
        try {
            return client.completeMultipartUpload(multipartUploadCreate.getBucketName(), multipartUploadCreate.getRegion(), multipartUploadCreate.getObjectName(), multipartUploadCreate.getUploadId(), multipartUploadCreate.getParts(), multipartUploadCreate.getHeaders(), multipartUploadCreate.getExtraQueryParams());
        } catch (Exception e) {
            log.error("合并分片失败", e);
            throw BusinessException.newBusinessException(ResultCode.KNOWN_ERROR.getCode(), e.getMessage());
        }
    }

   
    /**
     * 查询分片
     * @param multipartUploadCreate
     * @return
     */
    public ListPartsResponse listMultipart(MultipartUploadCreate multipartUploadCreate){
        try {
            return client.listMultipart(multipartUploadCreate.getBucketName(), multipartUploadCreate.getRegion(), multipartUploadCreate.getObjectName(), multipartUploadCreate.getMaxParts(), multipartUploadCreate.getPartNumberMarker(), multipartUploadCreate.getUploadId(), multipartUploadCreate.getHeaders(), multipartUploadCreate.getExtraQueryParams());
        } catch (Exception e) {
            log.error("查询分片失败", e);
            throw BusinessException.newBusinessException(ResultCode.KNOWN_ERROR.getCode(), e.getMessage());
        }
    }

   
/**
 * 获取预上传url
 * @param bucketName
 * @param objectName
 * @param queryParams
 * @return
 */
    public String getPresignedObjectUrl(String bucketName, String objectName, Map<String, String> queryParams) {
        try {
            return client.getPresignedObjectUrl(
                    GetPresignedObjectUrlArgs.builder()
                            .method(Method.PUT)
                            .bucket(bucketName)
                            .object(objectName)
                            .expiry(60 * 60 * 24)
                            .extraQueryParams(queryParams)
                            .build());
        } catch (Exception e) {
            log.error("查询分片失败", e);
            throw BusinessException.newBusinessException(ResultCode.KNOWN_ERROR.getCode(), e.getMessage());
        }
    }
}

核心代码

特别说明：只有文件大于5m才会分片上传。这是minio开源项目下的Discussions的截图

前端核心代码

const chunkSize = 5 * 1024 * 1024;

uploadFile = async () => {
    //获取用户选择的文件
    const file = document.getElementById("upload").files[0];
    console.log(file)
    //文件大小(大于5m再分片哦，否则直接走普通文件上传的逻辑就可以了，这里只实现分片上传逻辑)
    const fileSize = file.size

    if (fileSize <= chunkSize){
        console.log("上传的文件大于5m才能分片上传")
    }

    //计算当前选择文件需要的分片数量
    const chunkCount = Math.ceil(fileSize / chunkSize)
    console.log("文件大小：",(file.size / 1024 / 1024) + "Mb","分片数：",chunkCount)

    //获取文件md5
    const fileMd5 = await getFileMd5(file);
    
    //向后端请求本次分片上传初始化
    const initUploadParams = JSON.stringify({chunkSize: chunkCount,fileName: file.name,contentType:file.type})
    $.ajax({url: "你的后台预上传url", type: 'POST', contentType: "application/json", processData: false, data: initUploadParams,
        success: async function (res) {
      
            const chunkUploadUrls = res.data.chunks

            //若删除await会使用并发上传方式，当前分片上传完成后打印出来的完成提示是不准确的。而且会异步调用composeFile方法
            //但是当分片没上传完毕时调用此方法会发生null or empty异常。如需异步并发上传，composeFile方法调用需改造
            for (item of chunkUploadUrls) {
                //分片开始位置
                let start = (item.partNumber) * chunkSize
                //分片结束位置
                let end = Math.min(fileSize, start + chunkSize)
                //取文件指定范围内的byte，从而得到分片数据
                let _chunkFile = file.slice(start, end)
                console.log("开始上传第" + item.partNumber + "个分片")
                await $.ajax({url: item.uploadUrl, type: 'PUT', contentType: true, processData: false, data: _chunkFile,
                    success: function (res) {
                        console.log("第" + item.partNumber + "个分片上传完成")
                    }
                }).catch(err=>{console.log(err)})
            }
            console.log(file.type)
            console.log(file.contentType)
            //请求后端合并文件
            composeFile(res.data.uploadId,file.name, chunkCount, fileSize, file.type)
        }
    })
}
/**
 * 请求后端合并文件
 * @param fileMd5
 * @param fileName
 */
composeFile = (uploadId,fileName, chunkSize, fileSize, contentType) => {
    console.log("开始请求后端合并文件")
    const composeParams = JSON.stringify({uploadId: uploadId,fileName: fileName,chunkSize: chunkSize, fileSize: fileSize, contentType: contentType,pass:"minio123", expire: 12, maxGetCount: 2})
    $.ajax({url:  "你的后台合并文件url", type: 'POST', contentType: "application/json", processData: false, data: composeParams,
        success: function (res) {
            console.log("合并文件完成",res.data)
        }
    })
}
/**
 * 获取文件MD5
 * @param file
 * @returns {Promise<unknown>}
 */
getFileMd5 = (file) => {
    let fileReader = new FileReader()
    fileReader.readAsBinaryString(file)
    let spark = new SparkMD5()
    return new Promise((resolve) => {
        fileReader.onload = (e) => {
            spark.appendBinary(e.target.result)
            resolve(spark.end())
        }
    })
}

后端核心代码

service类

@Slf4j
@Service
public class FileUploadService {
   
    @Autowired
    private final MinioCoreUtils minioCoreUtils;
    
    /**
     * 普通上传
     *
     * @param file
     * @return
     */
    public FileUploadResponse upload(MultipartFile file) {
        Assert.notNull(file, "文件不能为空");
        log.info("start file upload");

        //文件上传
        try {
            return minioCoreUtils.uploadFile(file);
        } catch (IOException e) {
            log.error("file upload error.", e);
            throw BusinessException.newBusinessException(ResultCode.FILE_IO_ERROR.getCode());
        } catch (ServerException e) {
            log.error("minio server error.", e);
            throw BusinessException.newBusinessException(ResultCode.MINIO_SERVER_ERROR.getCode());
        } catch (InsufficientDataException e) {
            log.error("insufficient data throw exception", e);
            throw BusinessException.newBusinessException(ResultCode.MINIO_INSUFFICIENT_DATA.getCode());
        } catch (Exception e) {
            log.error(e.getMessage(), e);
            throw BusinessException.newBusinessException(ResultCode.KNOWN_ERROR.getCode());
        }
    }


    /**
     * 创建分片上传
     *
     * @param createRequest
     * @return
     */
    public MultipartUploadCreateResponse createMultipartUpload(MultipartUploadCreateRequest createRequest) {
        log.info("创建分片上传开始, createRequest: [{}]", createRequest);
        //设置分片文件类型
        Multimap<String, String> headers = HashMultimap.create();
        headers.put("Content-Type", createRequest.getContentType());
        MultipartUploadCreateResponse response = new MultipartUploadCreateResponse();
        response.setChunks(new LinkedList<>());
        final MultipartUploadCreate uploadCreate = MultipartUploadCreate.builder()
                .bucketName(minioCoreUtils.minioProperties.getBucketName())
                .objectName(createRequest.getFileName()).headers(headers)
                .build();

        final CreateMultipartUploadResponse uploadId = minioCoreUtils.uploadId(uploadCreate);
        uploadCreate.setUploadId(uploadId.result().uploadId());
        response.setUploadId(uploadCreate.getUploadId());
        Map<String, String> reqParams = new HashMap<>();
        reqParams.put("uploadId", uploadId.result().uploadId());

        for (int i = 0; i < createRequest.getChunkSize(); i++) {
            reqParams.put("partNumber", String.valueOf(i));
            String presignedObjectUrl = minioCoreUtils.getPresignedObjectUrl(uploadCreate.getBucketName(), uploadCreate.getObjectName(), reqParams);
            if (StringUtils.isNotBlank(minioCoreUtils.minioProperties.getPath())) {//如果线上环境配置了域名解析，可以进行替换
                presignedObjectUrl = presignedObjectUrl.replace(minioCoreUtils.minioProperties.getEndpoint(), minioCoreUtils.minioProperties.getPath());
            }
            MultipartUploadCreateResponse.UploadCreateItem item = new MultipartUploadCreateResponse.UploadCreateItem();
            item.setPartNumber(i);
            item.setUploadUrl(presignedObjectUrl);
            response.getChunks().add(item);
        }
        log.info("创建分片上传结束, createRequest: [{}]", createRequest);
        return response;
    }

    /**
     * 分片合并
     *
     * @param uploadRequest
     */
    public FileUploadResponse completeMultipartUpload(CompleteMultipartUploadRequest uploadRequest) {
        log.info("文件合并开始, uploadRequest: [{}]", uploadRequest);
        //headers.put("Content-Type",uploadRequest.getContentType());
        Multimap<String, String> headers = HashMultimap.create();
        headers.put("Content-Type", uploadRequest.getContentType());
        try {
            final ListPartsResponse listMultipart = minioCoreUtils.listMultipart(MultipartUploadCreate.builder()
                    .bucketName(minioCoreUtils.minioProperties.getBucketName())
                    .objectName(uploadRequest.getFileName())
                    .maxParts(uploadRequest.getChunkSize() + 10)
                    .uploadId(uploadRequest.getUploadId()).type(uploadRequest.getContentType())
                    .partNumberMarker(0)
                    .build());
            final ObjectWriteResponse objectWriteResponse = minioCoreUtils.completeMultipartUpload(MultipartUploadCreate.builder()
                    .bucketName(minioCoreUtils.minioProperties.getBucketName())
                    .uploadId(uploadRequest.getUploadId())
                    .objectName(uploadRequest.getFileName()).headers(headers)
                    .type(uploadRequest.getContentType())
                    .parts(listMultipart.result().partList().toArray(new Part[]{}))
                    .build());

            return FileUploadResponse.builder()
                    .url(minioCoreUtils.minioProperties.getDownloadUri() + "/" + minioCoreUtils.minioProperties.getBucketName() + "/" + uploadRequest.getFileName())
                    .build();
        } catch (Exception e) {
            log.error("合并分片失败", e);
        }
        log.info("文件合并结束, uploadRequest: [{}]", uploadRequest);
        return null;
    }
}

Controller类主要方法

@ApiOperation("创建分片上传")
@PostMapping("/multipart/create")
public MultipartUploadCreateResponse createMultipartUpload(@RequestBody @Validated
                MultipartUploadCreateRequest multipartUploadCreateRequest) {
    return fileUploadService.createMultipartUpload(multipartUploadCreateRequest);
}

@ApiOperation("合并分片")
@PostMapping("/multipart/complete")
public FileUploadResponse completeMultipartUpload(@RequestBody @Validated
                CompleteMultipartUploadRequest uploadRequest) {
    return fileUploadService.completeMultipartUpload(uploadRequest);
}