Ceph 之 s3cmd 命令

1,025 阅读5分钟

本文目的:记录常用 s3cmd 命令,方便以后查找使用。

一、基本使用

(1)安装

# 安装最新版 s3cmd,可通过 pip 在线安装,或从 FTP 或百度云上下载离线包离线安装
python -m pip install s3cmd==2.1.0 --upgrade -i https://pypi.douban.com/simple
# python -m pip install -r requirements.txt --no-index --find-links=/opt/software/basic/ansible/packages/
 
# 验证安装结果
s3cmd --version
  
# 为 s3 访问创建一个用户,记录 uid、access_key,secret_key
radosgw-admin user create  --uid="dolphin" --display-name="dolphin"
  
# 初始化
s3cmd --configure
# 该过程会提示输入很多设置参数,填写好 access_key 和 secret_key,其他均采用默认值
# 填写完不要执行 Test,直接保存退出
Test access with supplied credentials? [Y/n] n
Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
  
# 再次修改配置 ~/.s3cfg 中以下配置
cloudfront_host = {radosgw_ip}:{rados_port}
host_base = {radosgw_ip}:{rados_port}
host_bucket = {radosgw_ip}:{rados_port}
  
  
# 检查是否可以正常使用
s3cmd mb s3://test
s3cmd ls s3://
s3cmd rb s3://test

(2)基本使用

# 查看手册
s3cmd -h

# 查看 bucket 信息
s3cmd info s3://groupchat

# 列出所有的桶
s3cmd ls
 
# 创建一个名为 dolphin 的桶
s3cmd mb s3://dolphin
 
# 列出指定桶里所有的对
s3cmd ls s3://dolphin
 
# 存储一个名为ml_map.png的文件:
s3cmd put ml_map.png s3://dolphin
 
#上传并将权限设置为所有人可读
s3cmd put --acl-public file.txt s3://dolphin /file.txt
  
# 批量上传文件
s3cmd put ./* s3://dolphin/
 
# 获得刚刚上传的文件
s3cmd get s3://dolphin/ml_map.png ml_map.png1
 
# 删除指定对象
s3cmd del s3://dolphin/hello.txt
  
# 删除 bucket 下的文件夹
s3cmd del s3://dolphin/dolphin/ --recursive

# 删除指定桶
s3cmd rb s3://dolphin
 
# 获得对应的bucket所占用的空间大小
s3cmd du -H s3://dolphin

# 设置桶权限
s3cmd setacl s3://my-bucket-name/ --acl-public --recursive
[dolphin@bfd-yiz-1p24 ~]$ s3cmd setacl s3://oceantemplate/ --acl-public --recursive
s3://oceantemplate/tempalte.xlsx: ACL set to Public  [1 of 4]
s3://oceantemplate/template.docx: ACL set to Public  [2 of 4]
s3://oceantemplate/template.pptx: ACL set to Public  [3 of 4]
s3://oceantemplate/template.xlsx: ACL set to Public  [4 of 4]

(3)修改 Bucket 生命周期

可查看:stackoverflow.com/questions/4…

桶设置过期天数(),文件有个创建时间,

# 存在 bucket,但没有设置生命周期

# 创建文件
$ vim lifecycle_policy.xml

# 输入
# ID 不给,会随机生成
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
        <Rule>
                <Prefix/>
                <Status>Enabled</Status>
                <Expiration>
                        <Days>60</Days>
                </Expiration>
        </Rule>
</LifecycleConfiguration>


# 给 groupchat 设置生命周期
$ s3cmd setlifecycle lifecycle_policy.xml s3://groupchat


# 查看
$ s3cmd getlifecycle s3://groupchat
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<ID>7yik56gat3je2h9yexktwv75pzsxx4wzbl4mgeqdh4we6t4t</ID>
		<Prefix/>
		<Status>Enabled</Status>
		<Expiration>
			<Days>60</Days>
		</Expiration>
	</Rule>
</LifecycleConfiguration>


# 不设置过期
$ s3cmd dellifecycle s3://groupchat

# 举例如下:
[dolphin@bfd-yiz-1p24 ~]$ s3cmd dellifecycle s3://groupchat
s3://groupchat/: Lifecycle Policy deleted
$ s3cmd dellifecycle s3://chat
s3://chat/: Lifecycle Policy deleted

# 查看
$ s3cmd info s3://chat/picture/f803477a-d94d-4581-bc16-701175da5231_scaled.jpg
s3://chat/picture/f803477a-d94d-4581-bc16-701175da5231_scaled.jpg (object):
   File size: 114441
   Last mod:  Mon, 12 Apr 2021 07:00:30 GMT
   MIME type: image/jpeg
   Storage:   STANDARD
   MD5 sum:   4273cd0eafced905744498727363db57
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       *anon*: READ
   ACL:       dolphin: FULL_CONTROL
   URL:       http://172.18.1.23:7480/chat/picture/f803477a-d94d-4581-bc16-701175da5231_scaled.jpg



二、更新生命周期

测试:删除 bucket 生命周期,文件是否过期。

结果:删除 bucket 生命周期后,文件不会过期被删除。

# 创建 test-lifecycle 桶
$ s3cmd mb s3://test-lifecycle
Bucket 's3://test-lifecycle/' created

# 查看桶生命周期
$ s3cmd getlifecycle s3://test-lifecycle
ERROR: S3 error: 404 (NoSuchLifecycleConfiguration)

# 设置桶生命周期
# 1. 创建文件
$ vim lifecycle_policy.xml
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
        <Rule>
                <Prefix/>
                <Status>Enabled</Status>
                <Expiration>
                        <Days>1</Days>
                </Expiration>
        </Rule>
</LifecycleConfiguration>

# 2. 设置规则
$ s3cmd setlifecycle lifecycle_policy.xml s3://test-lifecycle
s3://test-lifecycle/: Lifecycle Policy updated

# 查看生命周期 
$ s3cmd getlifecycle s3://test-lifecycle
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<ID>iax0mb1qe44ty27e7mjh260evtmp3kxyzawx8nykxitsu3d4</ID>
		<Prefix/>
		<Status>Enabled</Status>
		<Expiration>
			<Days>1</Days>
		</Expiration>
	</Rule>
</LifecycleConfiguration>


# 上传文件
$ s3cmd put lifecycle_policy.xml s3://test-lifecycle
upload: 'lifecycle_policy.xml' -> 's3://test-lifecycle/lifecycle_policy.xml'  [1 of 1]
 295 of 295   100% in    0s    16.13 KB/s  done
 
# 列举文件
$ s3cmd ls s3://test-lifecycle
2021-06-11 08:08          295  s3://test-lifecycle/lifecycle_policy.xml

# 查看文件
$ s3cmd info s3://test-lifecycle/lifecycle_policy.xml
s3://test-lifecycle/lifecycle_policy.xml (object):
   File size: 295
   Last mod:  Fri, 11 Jun 2021 08:08:49 GMT
   MIME type: text/plain
   Storage:   STANDARD
   MD5 sum:   66a2e4e5d187b1f0e5056f82b4b0d18f
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       dolphin: FULL_CONTROL

# 删除桶生命周期
$ s3cmd dellifecycle s3://test-lifecycle
s3://test-lifecycle/: Lifecycle Policy deleted


# 一天后查看文件:
# 发现文件还在
$ s3cmd ls s3://test-lifecycle
2021-06-11 08:08          295  s3://test-lifecycle/lifecycle_policy.xml

创建时候,不设置过期时间,即可。



三、创建文件 signUrl

发现:调用 cephGeneratePresignedUrlRequest 时候后,该文件会一直可被读。

  • 命令:s3cmd info s3://{object} 发现有 ACL: *anon*: READ
  • 这个文件是应该是私有的 (private)。

2021-10-2510-06-40.png

解决:上传时候指定权限即可。(例如:privatepublic

调试之后发现,原来分片上传文件都设置成 public 了。

总结命令如下:

s3cmd info s3://oceanfile/org_223c1017ad907fe8/202110/ef4cdc48-cc8a-433d-8e5f-9a5c89994f53.docx

# 设置私有
s3cmd setacl  s3://oceanfile/org_223c1017ad907fe8/202110/fd512010-75b3-4c5f-84f6-ceb0b8188185.mp4 --acl-private



四、分片指令

  1. 查询桶下分片任务:s3cmd multipart s3://bucketName

    查询完成初始化,并未开始或者未完成的分片

$ s3cmd multipart s3://oceanfile
s3://oceanfile/
Initiated	Path	Id
2022-02-15T06:34:47.850Z	s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/0fc66ba9-4473-4ff5-96e0-2fe58eb89c43.mp4	2~FW-hksVKeLzA4rJJ6KUoMkU8xBtO9cn
2022-02-15T06:37:36.784Z	s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4	2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
2022-02-17T01:54:03.611Z	s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/f0db6d2b-927b-49a7-94a8-13b6118360e8.mp4	2~RaFCeaoQMgXaOH_cDz0AGGo2lYRpcsj

2022-02-1711-27-24.png

  1. 查询分片任务信息:s3cmd listmp s3://BUCKET/OBJECT uploadId
$ s3cmd listmp s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4 2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
LastModified             PartNumber  ETage                           Size
2022-02-15T06:37:44.312Z	1	"24771f0fefc455246dca8a7994f11ed9"	2097152
2022-02-15T06:37:44.402Z	2	"692c39e1e80f1a916e7826392d9cd106"	2097152
2022-02-15T06:37:44.481Z	3	"5cddcbcc601f65095fa185fd92c8dde5"	2097152
2022-02-15T06:37:44.554Z	4	"cd015f1112b1ae57cd4d63dccb7a0d1b"	2097152
2022-02-15T06:37:52.242Z	5	"fdd50bc58cea60a54c95122134cf4ebf"	2097152
2022-02-15T06:37:52.308Z	6	"4a89856e422d256b5a3e9b14130f77f3"	2097152
2022-02-15T06:37:52.378Z	7	"f8a0cfd027323c70b81e48239efee0a1"	2097152
2022-02-15T06:37:52.464Z	8	"bbe1466001061757c53cae98962836a0"	2097152

这样查询信息不成功:

[dolphin@bfd-yiz-1p24 ~]$ s3cmd multipart s3://oceanfile
s3://oceanfile/
Initiated	Path	Id
2022-02-15T06:37:36.784Z	s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4	2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
2022-02-17T01:54:03.611Z	s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/f0db6d2b-927b-49a7-94a8-13b6118360e8.mp4	2~RaFCeaoQMgXaOH_cDz0AGGo2lYRpcsj
[dolphin@bfd-yiz-1p24 ~]$ s3cmd info s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4
ERROR: S3 error: 404 (Not Found)
  1. 删除分片:s3cmd abortmp s3://BUCKET/OBJECT uploadId
[dolphin@bfd-yiz-1p24 ~]$ s3cmd abortmp s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/0fc66ba9-4473-4ff5-96e0-2fe58eb89c43.mp4 2~FW-hksVKeLzA4rJJ6KUoMkU8xBtO9cn
s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/0fc66ba9-4473-4ff5-96e0-2fe58eb89c43.mp4
[dolphin@bfd-yiz-1p24 ~]$ s3cmd multipart s3://oceanfile
s3://oceanfile/
Initiated	Path	Id
2022-02-15T06:37:36.784Z	s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/b1c0cda6-21da-4571-b697-822c3eae2000.mp4	2~QfTf44tuTFBadpoE4TZ7gF1echF46XA
2022-02-17T01:54:03.611Z	s3://oceanfile/org_e93791fddc0bafe0bd1f5b89/202202/f0db6d2b-927b-49a7-94a8-13b6118360e8.mp4	2~RaFCeaoQMgXaOH_cDz0AGGo2lYRpcsj
[dolphin@bfd-yiz-1p24 ~]$ 
  1. 上传分片:s3cmd put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/
$ s3cmd put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/
upload: 'Camara.zip' -> 's3://some-bucket/Camara.zip'  [part 1 of 16, 500MB] [1 of 1]

$ s3cmd --upload-id Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 1, skipping.
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 2, skipping.
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 3, skipping.
WARNING: MultiPart: size and md5sum match for s3://some-bucket/Camara.zip part 4, skipping.
upload: 'Camara.zip' -> 's3://some-bucket/Camara.zip'  [part 5 of 16, 500MB] [1 of 1]
     65536 of 524288000     0% in    3s    16.27 kB/s  failed
WARNING: Upload failed: /Camara.zip?partNumber=5&uploadId=Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
upload: 'Camara.zip' -> 's3://some-bucket/Camara.zip'  [part 5 of 16, 500MB] [1 of 1]

继续上传:

s3cmd --upload-id Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM put --multipart-chunk-size-mb=500 Camara.zip s3://some-bucket/