Flink 实战1| 青训营笔记

这是我参与「第四届青训营 -大数据场」笔记创作活动的第4篇笔记

本文已参与「新人创作礼」活动， 一起开启掘金创作之路。

创建Maven工程，导入依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>FlinkProject</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_2.12</artifactId>
            <version>1.14.4</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-scala -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-scala_2.12</artifactId>
            <version>1.14.4</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>1.14.4</version>
        </dependency>




    </dependencies>

    <properties>
        <maven.compiler.source>18</maven.compiler.source>
        <maven.compiler.target>18</maven.compiler.target>
    </properties>

</project>

批处理WordCount

1. 创建执行环境

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

2. 从文件读取数据

String inputPath = "D:\Program Files\Java\project\TestCalcite1\FlinkProject\src\main\resources\hello.txt";
//批处理用DataSet
DataSet<String> inputDataSet = env.readTextFile(inputPath);

3. 这里我们关注DataSet.class里面的flatMap方法

public <R> FlatMapOperator<T, R> flatMap(FlatMapFunction<T, R> flatMapper) {
    if (flatMapper == null) {
        throw new NullPointerException("FlatMap function must not be null.");
    } else {
        String callLocation = Utils.getCallLocationName();
        TypeInformation<R> resultType = TypeExtractor.getFlatMapReturnTypes(flatMapper, this.getType(), callLocation, true);
        return new FlatMapOperator(this, resultType, (FlatMapFunction)this.clean(flatMapper), callLocation);
    }
}

public interface FlatMapFunction<T, O> extends Function, Serializable {
    void flatMap(T var1, Collector<O> var2) throws Exception;
}

这里我们发现flatMap有两个参数，第一个参数是value，第二个参数是集合，我们在实现FlatMapFunction时，重写flapMap方法，调用collector.collect(T record)

4. 重写FlatMapFunction

public static class MyFlatMapper implements FlatMapFunction<String, Tuple2<String, Integer>> {

    @Override
    public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
        String[] words = s.split(" ");

        for (String word : words) {
            collector.collect(new Tuple2<>(word, 1));

        }
    }
}

DataSet<Tuple2<String,Integer>> resultSet = inputDataSet.flatMap(new MyFlatMapper())
        .groupBy(0)//按照第一个位置的word分组
        .sum(1);//将第二个位置上的数据求和

resultSet.print();

流处理WordCount

1. 创建流处理执行环境

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

2. 从文件中读取数据

String inputPath = "D:\Program Files\Java\project\TestCalcite1\FlinkProject\src\main\resources\hello.txt";
DataStream<String> stringDataStream = env.readTextFile(inputPath);

3. 基于数据流进行转换计算

DataStream<Tuple2<String, Integer>> result = stringDataStream.flatMap(new WordCount.MyFlatMapper())
        .keyBy(0)//指定key重分区
        .sum(1);
result.print();

4. 执行任务

env.execute();