Eclipse 数据空间组件-实现文件传输-7

sunbin
31 阅读3分钟

操作目标

通过云存储模拟器（Azurite和MinIO）实现文件传输，提升本地机器文件传输的应用场景实用性。

graph LR
o[file] -->|存储| A{EDC}
A[Azurite] --> B{EDC 传输}
B --> C[MinIO] 
C[MinIO]-->|存储|m[file]

操作前提

确保已安装以下工具：

操作步骤

  1. 启动Docker容器

    • 先修改transfer/transfer-05-file-transfer-cloud/resources/docker-compose.yaml 将占位符 : <root-token> 更换为 root-token

    • 执行命令：

      docker compose -f transfer/transfer-05-file-transfer-cloud/resources/docker-compose.yaml up -d

    • 查看日志，确认minio、azurite和hashicorp-vault已正确启动。

  2. 在MinIO中创建存储桶

    • 访问http://localhost:9001，使用docker-compose文件（第20-21行）中的凭据登录。
    • 在“Buckets”中创建名为“src-bucket”的存储桶。

  3. 上传文件到Azurite

    • 创建容器：127.0.0.1替换为本机ip

      $conn_str="DefaultEndpointsProtocol=http;AccountName=provider;AccountKey=password;BlobEndpoint=http://127.0.0.1:10000/provider;"
az storage container create --name src-container --connection-string $conn_str

      成功后会返回{"created": true}。

    • 上传文件：

      az storage blob upload -f ./transfer/transfer-05-file-transfer-cloud/resources/test-document.txt --container-name src-container --name test-document.txt --connection-string $conn_str

    • 验证文件是否上传成功： 127.0.0.1替换为本机ip

      az storage blob list --container-name src-container --connection-string "DefaultEndpointsProtocol=http;AccountName=provider;AccountKey=password;BlobEndpoint=http://127.0.0.1:10000/provider;" --query "[].{name:name}" --output table

      应看到test-document.txt文件。

  4. 配置Vault

    • 执行以下命令查看密钥密钥： docker logs vault

      2025-04-10 11:32:55     $ export VAULT_ADDR='http://0.0.0.0:8200'
2025-04-10 11:32:55 
2025-04-10 11:32:55 The unseal key and root token are displayed below in case you want to
2025-04-10 11:32:55 seal/unseal the Vault or re-authenticate.
2025-04-10 11:32:55 
2025-04-10 11:32:55 Unseal Key: GjXnAsE+kX28esFGEwVAzKRC9guYGl4eblgNduzCa6I=
2025-04-10 11:32:55 Root Token: root-token
2025-04-10 11:32:55 
2025-04-10 11:32:55 Development mode should NOT be used in production installations!

    • 执行docker exec -it vault /bin/bash 进入到vault容器

      执行以下命令

          / # export VAULT_ADDR='http://0.0.0.0:8200'
    / # vault login  root-token
            Success! You are now authenticated. The token information displayed below
            is already stored in the token helper. You do NOT need to run "vault login"
            again. Future Vault requests will automatically use this token.

            Key                  Value
            ---                  -----
            token                root-token
            token_accessor       Ks4bzP1nqdiMie2yatLhTVft
            token_duration       ∞
            token_renewable      false
            token_policies       ["root"]
            identity_policies    []
            policies             ["root"]
    / # vault kv put secret/accessKeyId content=consumer
            ===== Secret Path =====
            secret/data/accessKeyId

            ======= Metadata =======
            Key                Value
            ---                -----
            created_time       2025-04-10T09:26:41.630422651Z
            custom_metadata    <nil>
            deletion_time      n/a
            destroyed          false
            version            1
    / # vault kv put secret/secretAccessKey content=password
            ======= Secret Path =======
            secret/data/secretAccessKey

            ======= Metadata =======
            Key                Value
            ---                -----
            created_time       2025-04-10T09:26:41.699405365Z
            custom_metadata    <nil>
            deletion_time      n/a
            destroyed          false
            version            1
    / # vault kv put secret/provider-key content=password
            ====== Secret Path ======
            secret/data/provider-key

            ======= Metadata =======
            Key                Value
            ---                -----
            created_time       2025-04-10T09:28:02.360228461Z
            custom_metadata    <nil>
            deletion_time      n/a
            destroyed          false
            version            1

      也可以在宿主机打开http://0.0.0.0:8200页面，使用root-token可以登录页面

  5. 启动连接器

    • 修改配置文件config.properties：

       edc.participant.id=provider
 edc.dsp.callback.address=http://localhost:19194/protocol
 web.http.port=19191
 web.http.path=/api
 web.http.management.port=19193
 web.http.management.path=/management
 web.http.protocol.port=19194
 web.http.protocol.path=/protocol
 edc.api.auth.key=password
 web.http.public.port=19291
 web.http.public.path=/public
 web.http.control.port=19192
 web.http.control.path=/control

 edc.vault.hashicorp.url=http://127.0.0.1:8200
 edc.vault.hashicorp.token=root-token   #修改
 edc.vault.hashicorp.api.secret.path=/v1/secret
 edc.vault.hashicorp.health.check.enabled=false

 edc.blobstore.endpoint.template=http://172.30.208.1:10000/provider/%s  #修改

 edc.aws.access.key=accessKeyId
 edc.aws.secret.access.key=secretAccessKey

    • 在一个终端窗口中执行：

      ./gradlew clean build
java -Dedc.fs.config=transfer/transfer-05-file-transfer-cloud/cloud-transfer-provider/config.properties -jar transfer/transfer-05-file-transfer-cloud/cloud-transfer-provider/build/libs/provider.jar

    • 在另一个终端窗口中执行：

      java -Dedc.fs.config=transfer/transfer-05-file-transfer-cloud/cloud-transfer-consumer/config.properties -jar transfer/transfer-05-file-transfer-cloud/cloud-transfer-consumer/build/libs/consumer.jar

  6. 获取提供者合同报价

    • 执行命令：

      curl -X POST "http://localhost:29193/management/v3/catalog/request" \
    -H 'X-Api-Key: password' -H 'Content-Type: application/json' \
    -d @transfer/transfer-05-file-transfer-cloud/resources/fetch-catalog.json -s   jq

  7. 协商合同

    • 将negotiate-contract.json文件中的{{contract-offer-id}}占位符替换为目录中dcat:dataset.odrl:hasPolicy.@id路径下找到的合同报价ID（资产@id为1）。

    • 执行命令：

      curl -d @transfer/transfer-05-file-transfer-cloud/resources/negotiate-contract.json \
  -H 'X-Api-Key: password' X POST -H 'content-type: application/json' http://localhost:29193/management/v3/contractnegotiations \
  -s   jq

    • 使用返回的UUID，在消费者端通过端点检查协商的当前状态。

  8. 获取合同协议ID

    • 执行命令：

      curl -X GET "http://localhost:29193/management/v3/contractnegotiations/{{contract-negotiation-id}}" \
    -H 'X-Api-Key: password' --header 'Content-Type: application/json' \
    -s jq

  9. 传输数据

    • 将start-transfer.json文件中的{{contract-agreement-id}}占位符替换为上一步返回的contractAgreementId。

    • 执行命令：

      curl -X POST "http://localhost:29193/management/v3/transferprocesses" \
  -H 'X-Api-Key: password' -H "Content-Type: application/json" \
  -d @transfer/transfer-05-file-transfer-cloud/resources/start-transfer.json \
  -s  jq

  10. 检查传输状态

    • 使用返回的UUID，执行命令：

      curl -H 'X-Api-Key: password' http://localhost:29193/management/v3/transferprocesses/<transfer-process-id> -s   jq

  11. 停止Docker容器

    • 执行命令：

      docker compose -f transfer/transfer-05-file-transfer-cloud/resources/docker-compose.yaml down
avatar
文章
阅读
粉丝