携手创作，共同成长！这是我参与「掘金日新计划 · 8 月更文挑战」的第5天，点击查看活动详情
1.问题描述
在学习flink的过程中，flink-playgrounds/table-walkthrough是一个很好的练习项目。这个项目的问题如下：
report(transactions).executeInsert(“spend_report”);
将 transactions 表经过 report 函数处理后写入到 spend_report 表。
每分钟（或小时）计算在五分钟（或小时）内每个账号的平均交易金额（滑动窗口）？使用分钟还是小时作为单位均可。 依赖环境：
电脑上需要安装如下环境：Java 8 或者 Java 11、Maven、Git、Docker
代码仓库：
git clone https://github.com/apache/flink-playgrounds.git
需要完成该项目中的SpendReport.java的report方法的代码：
public static Table report(Table transactions) {
return transactions;
}
2.解题过程
2.1 拉取代码：
[root@m162p201 codes]# git clone https://github.com/apache/flink-playgrounds.git
Cloning into 'flink-playgrounds'...
remote: Enumerating objects: 503, done.
remote: Counting objects: 100% (187/187), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 503 (delta 130), reused 131 (delta 107), pack-reused 316
Receiving objects: 100% (503/503), 1.15 MiB | 451.00 KiB/s, done.
Resolving deltas: 100% (220/220), done.
2.2 建立目录
代码拉取完毕之后，建立相关的目录：
[root@m162p201 codes]# cd flink-playgrounds/table-walkthrough
[root@m162p201 table-walkthrough]#
[root@m162p201 table-walkthrough]# mkdir -p /tmp/flink-checkpoints-directory
[root@m162p201 table-walkthrough]# mkdir -p /tmp/flink-savepoints-directory
[root@m162p201 table-walkthrough]# docker-compose build
-bash: docker-compose: command not found
可以看出，docker-compose没有安装。接下来下一步需要安装docker-compose.
2.3 安装docker-compose
从github中去下载docker-compose编译好的文件。github.com/docker/comp… 选择所需要的合适的版本。
[root@m162p201 codes]# ls
flink-playgrounds
[root@m162p201 codes]# wget https://github.com/docker/compose/releases/download/v2.9.0/docker-compose-linux-x86_64
--2022-08-03 23:05:16-- https://github.com/docker/compose/releases/download/v2.9.0/docker-compose-linux-x86_64
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/15045751/8a670efa-e91e-4d49-9f8b-7106ea44d055?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220803%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220803T150517Z&X-Amz-Expires=300&X-Amz-Signature=2827375cb95f6b1aefa85ef3e1f9fe1116e6c4db0b8f2d185b616e487e5239a0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=15045751&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-linux-x86_64&response-content-type=application%2Foctet-stream [following]
--2022-08-03 23:05:17-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/15045751/8a670efa-e91e-4d49-9f8b-7106ea44d055?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220803%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220803T150517Z&X-Amz-Expires=300&X-Amz-Signature=2827375cb95f6b1aefa85ef3e1f9fe1116e6c4db0b8f2d185b616e487e5239a0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=15045751&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-linux-x86_64&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25804800 (25M) [application/octet-stream]
Saving to: ‘docker-compose-linux-x86_64’
100%[=========================================================================================================================================================================>] 25,804,800 44.2KB/s in 10m 49s
2022-08-03 23:16:06 (38.8 KB/s) - ‘docker-compose-linux-x86_64’ saved [25804800/25804800]
下载之后copy到/usr/local/lib
[root@m162p201 codes]# cp ./docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@m162p201 codes]# chmod +x /usr/local/bin/docker-compose
[root@m162p201 codes]# docker-compose --version
Docker Compose version v2.9.0
docker-compose 安装成功。
2.4 通过docker-compose编译
编译过程如下：
[root@m162p201 codes]# cd flink-playgrounds/table-walkthrough
[root@m162p201 table-walkthrough]# docker-compose build
[+] Building 2014.4s (25/25) FINISHED
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.21kB 0.0s
=> [apache/data-generator:1 internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.22kB 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [apache/data-generator:1 internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 internal] load metadata for docker.io/apache/flink:1.14.4-scala_2.12-java8 2.5s
=> [apache/data-generator:1 internal] load metadata for docker.io/library/maven:3.8-jdk-8-slim 2.4s
=> [apache/data-generator:1 internal] load metadata for docker.io/library/openjdk:8-jre 3.1s
=> [apache/data-generator:1 builder 1/4] FROM docker.io/library/maven:3.8-jdk-8-slim@sha256:eefc01f3404be359f20e94eb78a28509be9ff0e2cc564b770f7701a6d56fa326 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 stage-1 1/4] FROM docker.io/apache/flink:1.14.4-scala_2.12-java8@sha256:07cd20e018e2180c90ec5cf8342f208d408fe28d871a0727c7e52099bbf80f1a 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 internal] load build context 0.0s
=> => transferring context: 22.59kB 0.0s
=> CACHED [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 stage-1 2/4] RUN wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/1.14.4/flin 0.0s
=> CACHED [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 builder 2/4] COPY ./pom.xml /opt/pom.xml 0.0s
=> CACHED [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 builder 3/4] COPY ./src /opt/src 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 builder 4/4] RUN cd /opt; mvn clean install -Dmaven.test.skip 2010.6s
=> [apache/data-generator:1 internal] load build context 0.3s
=> => transferring context: 20.29kB 0.0s
=> CACHED [apache/data-generator:1 stage-1 1/4] FROM docker.io/library/openjdk:8-jre@sha256:4549aa1d251a9a60e31a5d82fbdb9184581d57ddb3dcbf187a8765df39a7b4f8 0.0s
=> CACHED [apache/data-generator:1 builder 2/4] COPY ./src /opt/data-producer/src 0.0s
=> CACHED [apache/data-generator:1 builder 3/4] COPY ./pom.xml /opt/data-producer/pom.xml 0.0s
=> [apache/data-generator:1 builder 4/4] RUN cd /opt/data-producer; mvn clean install 1235.0s
=> [apache/data-generator:1 stage-1 2/4] COPY --from=builder /opt/data-producer/target/data-generator-*.jar /opt/data-generator.jar 0.2s
=> [apache/data-generator:1 stage-1 3/4] RUN cd /opt 0.5s
=> [apache/data-generator:1 stage-1 4/4] COPY docker-entrypoint.sh / 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12] exporting to image 0.4s
=> => exporting layers 0.1s
=> => writing image sha256:6ed6c4dc22dd24fbfa6ef97d903dc24947d6ce7a47327066a7110523b92eda22 0.0s
=> => naming to docker.io/apache/data-generator:1 0.0s
=> => writing image sha256:ed6ff618fc7ec660f99c8ee954ff9bd6a1abed9c5f5f44022a3945cb543c8f3f 0.0s
=> => naming to docker.io/apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 stage-1 3/4] COPY --from=builder /opt/target/spend-report-*.jar /opt/flink/usrlib/spend-report.jar 0.0s
=> [apache/flink-table-walkthrough:1-FLINK-1.14-scala_2.12 stage-1 4/4] RUN echo "execution.checkpointing.interval: 10s" >> /opt/flink/conf/flink-conf.yaml; echo "pipeline.object-reuse: true" >> /opt 0.5s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[root@m162p201 table-walkthrough]#
编译成功之后，可以启动服务。
2.5 启动和关闭
[root@m162p201 table-walkthrough]# docker-compose up -d
[+] Running 8/8
⠿ Network table-walkthrough_default Created 0.1s
⠿ Container table-walkthrough-mysql-1 Started 1.6s
⠿ Container table-walkthrough-zookeeper-1 Started 1.5s
⠿ Container table-walkthrough-grafana-1 Started 3.3s
⠿ Container table-walkthrough-kafka-1 Started 3.8s
⠿ Container table-walkthrough-data-generator-1 Started 5.4s
⠿ Container table-walkthrough-jobmanager-1 Started 5.4s
⠿ Container table-walkthrough-taskmanager-1 Started 6.0s
# 关闭
[root@m162p201 table-walkthrough]# docker-compose down
[+] Running 8/7
⠿ Container table-walkthrough-taskmanager-1 Removed 0.0s
⠿ Container table-walkthrough-data-generator-1 Removed 10.8s
⠿ Container table-walkthrough-grafana-1 Removed 0.3s
⠿ Container table-walkthrough-jobmanager-1 Removed 0.0s
⠿ Container table-walkthrough-mysql-1 Removed 2.5s
⠿ Container table-walkthrough-kafka-1 Removed 4.0s
⠿ Container table-walkthrough-zookeeper-1 Removed 10.4s
⠿ Network table-walkthrough_default Removed
2.6 java的实现方法
在SpendReport.java的类中，report方法：
public static Table report(Table transactions) {
Table table = transactions.window(Slide.over(lit(5).minutes())//SlideWithSize
.every(lit(1).minutes())//SlideWithSizeAndSlide
.on($("transaction_time"))//SlideWithSizeAndSlideOnTime
.as("log_ts"))
.groupBy($("account_id"), $("log_ts"))
.select($("account_id"),
$("log_ts").start().as("log_ts"),
$("amount").avg().as("amount"));
return table;
}
2.7 重新编译
[root@m162p201 table-walkthrough]# docker-compose down
[root@m162p201 table-walkthrough]# docker-compose build
[root@m162p201 table-walkthrough]# docker-compose up -d --build
2.8 登陆mysql
现在可以通过mysql查询结果了：
[root@m162p201 table-walkthrough]# docker-compose exec mysql mysql -Dsql-demo -usql-demo -pdemo-sql
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.19 MySQL Community Server - GPL
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| sql-demo |
+--------------------+
2 rows in set (0.00 sec)
mysql> use sql-demo
Database changed
mysql> show tables;
+--------------------+
| Tables_in_sql-demo |
+--------------------+
| spend_report |
+--------------------+
1 row in set (0.00 sec)
mysql> select count(1) from spend_report;
+----------+
| count(1) |
+----------+
| 26582 |
+----------+
1 row in set (0.01 sec)
mysql>
