本文目的:记录常用 s3cmd 命令,方便以后查找使用。
一、基本使用
(1)安装
# 安装最新版 s3cmd,可通过 pip 在线安装,或从 FTP 或百度云上下载离线包离线安装
python -m pip install s3cmd==2.1.0 --upgrade -i https://pypi.douban.com/simple
# python -m pip install -r requirements.txt --no-index --find-links=/opt/software/basic/ansible/packages/
# 验证安装结果
s3cmd --version
# 为 s3 访问创建一个用户,记录 uid、access_key,secret_key
radosgw-admin user create --uid="dolphin" --display-name="dolphin"
# 初始化
s3cmd --configure
# 该过程会提示输入很多设置参数,填写好 access_key 和 secret_key,其他均采用默认值
# 填写完不要执行 Test,直接保存退出
Test access with supplied credentials? [Y/n] n
Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
# 再次修改配置 ~/.s3cfg 中以下配置
cloudfront_host = {radosgw_ip}:{rados_port}
host_base = {radosgw_ip}:{rados_port}
host_bucket = {radosgw_ip}:{rados_port}
# 检查是否可以正常使用
s3cmd mb s3://test
s3cmd ls s3://
s3cmd rb s3://test
(2)基本使用
# 查看手册
s3cmd -h
# 查看 bucket 信息
s3cmd info s3://groupchat
# 列出所有的桶
s3cmd ls
# 创建一个名为 dolphin 的桶
s3cmd mb s3://dolphin
# 列出指定桶里所有的对
s3cmd ls s3://dolphin
# 存储一个名为ml_map.png的文件:
s3cmd put ml_map.png s3://dolphin
#上传并将权限设置为所有人可读
s3cmd put --acl-public file.txt s3://dolphin /file.txt
# 批量上传文件
s3cmd put ./* s3://dolphin/
# 获得刚刚上传的文件
s3cmd get s3://dolphin/ml_map.png ml_map.png1
# 删除指定对象
s3cmd del s3://dolphin/hello.txt
# 删除 bucket 下的文件夹
s3cmd del s3://dolphin/dolphin/ --recursive
# 删除指定桶
s3cmd rb s3://dolphin
# 获得对应的bucket所占用的空间大小
s3cmd du -H s3://dolphin
# 设置桶权限
s3cmd setacl s3://my-bucket-name/ --acl-public --recursive
[dolphin@bfd-yiz-1p24 ~]$ s3cmd setacl s3://oceantemplate/ --acl-public --recursive
s3://oceantemplate/tempalte.xlsx: ACL set to Public [1 of 4]
s3://oceantemplate/template.docx: ACL set to Public [2 of 4]
s3://oceantemplate/template.pptx: ACL set to Public [3 of 4]
s3://oceantemplate/template.xlsx: ACL set to Public [4 of 4]
(3)修改 Bucket
生命周期
可查看:stackoverflow.com/questions/4…
桶设置过期天数(),文件有个创建时间,
# 存在 bucket,但没有设置生命周期
# 创建文件
$ vim lifecycle_policy.xml
# 输入
# ID 不给,会随机生成
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>60</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
# 给 groupchat 设置生命周期
$ s3cmd setlifecycle lifecycle_policy.xml s3://groupchat
# 查看
$ s3cmd getlifecycle s3://groupchat
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>7yik56gat3je2h9yexktwv75pzsxx4wzbl4mgeqdh4we6t4t</ID>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>60</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
# 不设置过期
$ s3cmd dellifecycle s3://groupchat
# 举例如下:
[dolphin@bfd-yiz-1p24 ~]$ s3cmd dellifecycle s3://groupchat
s3://groupchat/: Lifecycle Policy deleted
$ s3cmd dellifecycle s3://chat
s3://chat/: Lifecycle Policy deleted
# 查看
$ s3cmd info s3://chat/picture/f803477a-d94d-4581-bc16-701175da5231_scaled.jpg
s3://chat/picture/f803477a-d94d-4581-bc16-701175da5231_scaled.jpg (object):
File size: 114441
Last mod: Mon, 12 Apr 2021 07:00:30 GMT
MIME type: image/jpeg
Storage: STANDARD
MD5 sum: 4273cd0eafced905744498727363db57
SSE: none
Policy: none
CORS: none
ACL: *anon*: READ
ACL: dolphin: FULL_CONTROL
URL: http://172.18.1.23:7480/chat/picture/f803477a-d94d-4581-bc16-701175da5231_scaled.jpg
二、更新生命周期
测试:删除 bucket
生命周期,文件是否过期。
结果:删除 bucket
生命周期后,文件不会过期被删除。
# 创建 test-lifecycle 桶
$ s3cmd mb s3://test-lifecycle
Bucket 's3://test-lifecycle/' created
# 查看桶生命周期
$ s3cmd getlifecycle s3://test-lifecycle
ERROR: S3 error: 404 (NoSuchLifecycleConfiguration)
# 设置桶生命周期
# 1. 创建文件
$ vim lifecycle_policy.xml
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>1</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
# 2. 设置规则
$ s3cmd setlifecycle lifecycle_policy.xml s3://test-lifecycle
s3://test-lifecycle/: Lifecycle Policy updated
# 查看生命周期
$ s3cmd getlifecycle s3://test-lifecycle
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>iax0mb1qe44ty27e7mjh260evtmp3kxyzawx8nykxitsu3d4</ID>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>1</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
# 上传文件
$ s3cmd put lifecycle_policy.xml s3://test-lifecycle
upload: 'lifecycle_policy.xml' -> 's3://test-lifecycle/lifecycle_policy.xml' [1 of 1]
295 of 295 100% in 0s 16.13 KB/s done
# 列举文件
$ s3cmd ls s3://test-lifecycle
2021-06-11 08:08 295 s3://test-lifecycle/lifecycle_policy.xml
# 查看文件
$ s3cmd info s3://test-lifecycle/lifecycle_policy.xml
s3://test-lifecycle/lifecycle_policy.xml (object):
File size: 295
Last mod: Fri, 11 Jun 2021 08:08:49 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: 66a2e4e5d187b1f0e5056f82b4b0d18f
SSE: none
Policy: none
CORS: none
ACL: dolphin: FULL_CONTROL
# 删除桶生命周期
$ s3cmd dellifecycle s3://test-lifecycle
s3://test-lifecycle/: Lifecycle Policy deleted
# 一天后查看文件:
# 发现文件还在
$ s3cmd ls s3://test-lifecycle
2021-06-11 08:08 295 s3://test-lifecycle/lifecycle_policy.xml
创建时候,不设置过期时间,即可。
三、创建文件 signUrl
发现:调用 ceph
的 GeneratePresignedUrlRequest
时候后,该文件会一直可被读。
- 命令:
s3cmd info s3://{object}
发现有ACL: *anon*: READ
- 这个文件是应该是私有的 (
private
)。
解决:上传时候指定权限即可。(例如:private
或 public
)
调试之后发现,原来分片上传文件都设置成 public
了。
总结命令如下:
s3cmd info s3://oceanfile/org_223c1017ad907fe8/202110/ef4cdc48-cc8a-433d-8e5f-9a5c89994f53.docx
# 设置私有
s3cmd setacl s3://oceanfile/org_223c1017ad907fe8/202110/fd512010-75b3-4c5f-84f6-ceb0b8188185.mp4 --acl-private
四、分片指令
-
查询桶下分片任务:
s3cmd multipart s3://bucketName
查询完成初始化,并未开始或者未完成的分片
$ s3cmd multipart s3://oceanfile
s3://oceanfile/
Initiated Path Id
2022-02-15T06:34:47.850Z s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/0fc66ba9-4473-4ff5-96e0-2fe58eb89c43.mp4 2~FW-hksVKeLzA4rJJ6KUoMkU8xBtO9cn
2022-02-15T06:37:36.784Z s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4 2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
2022-02-17T01:54:03.611Z s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/f0db6d2b-927b-49a7-94a8-13b6118360e8.mp4 2~RaFCeaoQMgXaOH_cDz0AGGo2lYRpcsj
- 查询分片任务信息:
s3cmd listmp s3://BUCKET/OBJECT uploadId
$ s3cmd listmp s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4 2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
LastModified PartNumber ETage Size
2022-02-15T06:37:44.312Z 1 "24771f0fefc455246dca8a7994f11ed9" 2097152
2022-02-15T06:37:44.402Z 2 "692c39e1e80f1a916e7826392d9cd106" 2097152
2022-02-15T06:37:44.481Z 3 "5cddcbcc601f65095fa185fd92c8dde5" 2097152
2022-02-15T06:37:44.554Z 4 "cd015f1112b1ae57cd4d63dccb7a0d1b" 2097152
2022-02-15T06:37:52.242Z 5 "fdd50bc58cea60a54c95122134cf4ebf" 2097152
2022-02-15T06:37:52.308Z 6 "4a89856e422d256b5a3e9b14130f77f3" 2097152
2022-02-15T06:37:52.378Z 7 "f8a0cfd027323c70b81e48239efee0a1" 2097152
2022-02-15T06:37:52.464Z 8 "bbe1466001061757c53cae98962836a0" 2097152
这样查询信息不成功:
[dolphin@bfd-yiz-1p24 ~]$ s3cmd multipart s3://oceanfile
s3://oceanfile/
Initiated Path Id
2022-02-15T06:37:36.784Z s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4 2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
2022-02-17T01:54:03.611Z s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/f0db6d2b-927b-49a7-94a8-13b6118360e8.mp4 2~RaFCeaoQMgXaOH_cDz0AGGo2lYRpcsj
[dolphin@bfd-yiz-1p24 ~]$ s3cmd info s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4
ERROR: S3 error: 404 (Not Found)
- 删除分片:
s3cmd abortmp s3://BUCKET/OBJECT uploadId
[dolphin@bfd-yiz-1p24 ~]$ s3cmd abortmp s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/0fc66ba9-4473-4ff5-96e0-2fe58eb89c43.mp4 2~FW-hksVKeLzA4rJJ6KUoMkU8xBtO9cn
s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/0fc66ba9-4473-4ff5-96e0-2fe58eb89c43.mp4
[dolphin@bfd-yiz-1p24 ~]$ s3cmd multipart s3://oceanfile
s3://oceanfile/
Initiated Path Id
2022-02-15T06:37:36.784Z s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4 2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
2022-02-17T01:54:03.611Z s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/f0db6d2b-927b-49a7-94a8-13b6118360e8.mp4 2~RaFCeaoQMgXaOH_cDz0AGGo2lYRpcsj
[dolphin@bfd-yiz-1p24 ~]$
- 上传分片:
s3cmd put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/
$ s3cmd put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/
upload: 'Camara.zip' -> 's3://some-bucket/Camara.zip' [part 1 of 16, 500MB] [1 of 1]
$ s3cmd --upload-id Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 1, skipping.
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 2, skipping.
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 3, skipping.
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 4, skipping.
upload: 'Camara.zip' -> 's3://some-bucket/Camara.zip' [part 5 of 16, 500MB] [1 of 1]
65536 of 524288000 0% in 3s 16.27 kB/s failed
WARNING: Upload failed: /Camara.zip?partNumber=5&uploadId=Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
upload: 'Camara.zip' -> 's3://some-bucket/Camara.zip' [part 5 of 16, 500MB] [1 of 1]
继续上传:
s3cmd --upload-id Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/