Flink 自定义sink 向 Restful 风格接口发送数据

692 阅读1分钟

添加依赖

<dependency>
	<groupId>commons-httpclient</groupId>
	<artifactId>commons-httpclient</artifactId>
	<version>3.1</version>
</dependency>

基于 Flink 服务提交任务并执行时需要的依赖包

基于 flink 服务器提交任务前,先上传依赖包到 flink 的 lib 目录下;然后重启 flink 服务,使 jar 进行加载;否则会出现 ==ClassNoFoundException== 的异常。

  • commons-codec-1.2.jar
  • commons-httpclient-3.1.jar
  • commons-logging-1.0.4.jar

构建RestfullSink参数实例

public class RestfulSink implements Serializable {

	private static final long serialVersionUID = 41026197876622956L;
	
	private String url;

	public String getUrl() {
		return url;
	}
	
	public RestfulSink(Object obj) {
		final JSONObject json = JSONObject.parseObject(obj.toString());
		this.url = json.getString("url");
	}
}

构建自定义RestfulMessageSink

基于继承 RichSinkFunction< T > 抽象类实现自定义Sink,实现方法有三个:

  • open():构建sink节点时最先执行的方法,用于实现一些初始化动作。
  • invoke():执行节点时执行,用于实现具体业务逻辑。
  • close():关闭节点回收资源时执行,用于资源的回收。
import java.io.IOException;
import java.util.Iterator;
import java.util.Map;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.sink.RichSinkFunction;

import com.ygsoft.dataprocess.exception.RestfullExcepiton;
import com.ygsoft.dataprocess.vo.sink.RestfullSink;

public class RestfulMessageSink extends RichSinkFunction<Map<String, String>> {

	private static final long serialVersionUID = -905405395688757837L;

	private RestfulSink restfulSink;
	
	private HttpClient httpClient;
	
	public RestfulMessageSink(final RestfulSink restfulSink) {
		this.restfulSink = restfulSink;
	}
	
	@Override
	public void open(final Configuration parameters) throws Exception {
		super.open(parameters);
		httpClient = new HttpClient();
	}
	
	@Override
	public void invoke(Map<String, String> value, Context context) throws Exception {
		PostMethod postMethod = new PostMethod(restfullSink.getUrl());
		postMethod.getParams().setContentCharset("UTF-8");
		Iterator<Map.Entry<String, String>> iterator = value.entrySet().iterator();
		while (iterator.hasNext()) {
			Map.Entry<String, String> next = iterator.next();
			postMethod.addParameter(next.getKey().equals("value")? "data" : next.getKey(), next.getValue().toString());
		}
		
		try {
			httpClient.executeMethod(postMethod);
			/*int code = httpClient.executeMethod(postMethod);
			if(code != 200) {
				System.out.println(postMethod.getResponseBodyAsString());
			}*/
		} catch(IOException e) {
			throw new RestfullExcepiton("restful接口调用失败");
		}
	}
	
	@Override
    public void close() throws Exception {
        super.close();
    }
}