聊聊flink的TextOutputFormat

序

本文主要研究一下flink的TextOutputFormat

DataStream.writeAsText

flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/datastream/DataStream.java

	/**
	 * Writes a DataStream to the file specified by path in text format.
	 *
	 * <p>For every element of the DataStream the result of {@link Object#toString()} is written.
	 *
	 * @param path
	 *            The path pointing to the location the text file is written to
	 * @param writeMode
	 *            Controls the behavior for existing files. Options are
	 *            NO_OVERWRITE and OVERWRITE.
	 *
	 * @return The closed DataStream.
	 */
	@PublicEvolving
	public DataStreamSink<T> writeAsText(String path, WriteMode writeMode) {
		TextOutputFormat<T> tof = new TextOutputFormat<>(new Path(path));
		tof.setWriteMode(writeMode);
		return writeUsingOutputFormat(tof);
	}

	/**
	 * Writes the dataStream into an output, described by an OutputFormat.
	 *
	 * <p>The output is not participating in Flink's checkpointing!
	 *
	 * <p>For writing to a file system periodically, the use of the "flink-connector-filesystem"
	 * is recommended.
	 *
	 * @param format The output format
	 * @return The closed DataStream
	 */
	@PublicEvolving
	public DataStreamSink<T> writeUsingOutputFormat(OutputFormat<T> format) {
		return addSink(new OutputFormatSinkFunction<>(format));
	}

• DataStream的writeAsText方法创建了TextOutputFormat，然后通过OutputFormatSinkFunction包装为sink function

TextOutputFormat

flink-java-1.7.0-sources.jar!/org/apache/flink/api/java/io/TextOutputFormat.java

/**
 * A {@link FileOutputFormat} that writes objects to a text file.
 *
 * <p>Objects are converted to Strings using either {@link Object#toString()} or a {@link TextFormatter}.
 * @param <T> type of elements
 */
@PublicEvolving
public class TextOutputFormat<T> extends FileOutputFormat<T> {

	private static final long serialVersionUID = 1L;

	private static final int NEWLINE = '\n';

	private String charsetName;

	private transient Charset charset;

	// --------------------------------------------------------------------------------------------


	/**
	 * Formatter that transforms values into their {@link String} representations.
	 * @param <IN> type of input elements
	 */
	public interface TextFormatter<IN> extends Serializable {
		String format(IN value);
	}

	public TextOutputFormat(Path outputPath) {
		this(outputPath, "UTF-8");
	}

	public TextOutputFormat(Path outputPath, String charset) {
		super(outputPath);
		this.charsetName = charset;
	}

	public String getCharsetName() {
		return charsetName;
	}

	public void setCharsetName(String charsetName) throws IllegalCharsetNameException, UnsupportedCharsetException {
		if (charsetName == null) {
			throw new NullPointerException();
		}

		if (!Charset.isSupported(charsetName)) {
			throw new UnsupportedCharsetException("The charset " + charsetName + " is not supported.");
		}

		this.charsetName = charsetName;
	}

	// --------------------------------------------------------------------------------------------

	@Override
	public void open(int taskNumber, int numTasks) throws IOException {
		super.open(taskNumber, numTasks);

		try {
			this.charset = Charset.forName(charsetName);
		}
		catch (IllegalCharsetNameException e) {
			throw new IOException("The charset " + charsetName + " is not valid.", e);
		}
		catch (UnsupportedCharsetException e) {
			throw new IOException("The charset " + charsetName + " is not supported.", e);
		}
	}

	@Override
	public void writeRecord(T record) throws IOException {
		byte[] bytes = record.toString().getBytes(charset);
		this.stream.write(bytes);
		this.stream.write(NEWLINE);
	}

	// --------------------------------------------------------------------------------------------

	@Override
	public String toString() {
		return "TextOutputFormat (" + getOutputFilePath() + ") - " + this.charsetName;
	}
}

• TextOutputFormat继承了FileOutputFormat，其open方法主要是调用FileOutputFormat的open方法，而writeRecord方法则直接往stream进行write，写完一条record之后再写一个换行(\n)

FileOutputFormat

flink-core-1.7.0-sources.jar!/org/apache/flink/api/common/io/FileOutputFormat.java

/**
 * The abstract base class for all Rich output formats that are file based. Contains the logic to
 * open/close the target
 * file streams.
 */
@Public
public abstract class FileOutputFormat<IT> extends RichOutputFormat<IT> implements InitializeOnMaster, CleanupWhenUnsuccessful {
	//......

	/**
	 * Initialization of the distributed file system if it is used.
	 *
	 * @param parallelism The task parallelism.
	 */
	@Override
	public void initializeGlobal(int parallelism) throws IOException {
		final Path path = getOutputFilePath();
		final FileSystem fs = path.getFileSystem();
		
		// only distributed file systems can be initialized at start-up time.
		if (fs.isDistributedFS()) {
			
			final WriteMode writeMode = getWriteMode();
			final OutputDirectoryMode outDirMode = getOutputDirectoryMode();

			if (parallelism == 1 && outDirMode == OutputDirectoryMode.PARONLY) {
				// output is not written in parallel and should be written to a single file.
				// prepare distributed output path
				if(!fs.initOutPathDistFS(path, writeMode, false)) {
					// output preparation failed! Cancel task.
					throw new IOException("Output path could not be initialized.");
				}

			} else {
				// output should be written to a directory

				// only distributed file systems can be initialized at start-up time.
				if(!fs.initOutPathDistFS(path, writeMode, true)) {
					throw new IOException("Output directory could not be created.");
				}
			}
		}
	}
	
	@Override
	public void tryCleanupOnError() {
		if (this.fileCreated) {
			this.fileCreated = false;
			
			try {
				close();
			} catch (IOException e) {
				LOG.error("Could not properly close FileOutputFormat.", e);
			}

			try {
				FileSystem.get(this.actualFilePath.toUri()).delete(actualFilePath, false);
			} catch (FileNotFoundException e) {
				// ignore, may not be visible yet or may be already removed
			} catch (Throwable t) {
				LOG.error("Could not remove the incomplete file " + actualFilePath + '.', t);
			}
		}
	}

	@Override
	public void configure(Configuration parameters) {
		// get the output file path, if it was not yet set
		if (this.outputFilePath == null) {
			// get the file parameter
			String filePath = parameters.getString(FILE_PARAMETER_KEY, null);
			if (filePath == null) {
				throw new IllegalArgumentException("The output path has been specified neither via constructor/setters" +
						", nor via the Configuration.");
			}
			
			try {
				this.outputFilePath = new Path(filePath);
			}
			catch (RuntimeException rex) {
				throw new RuntimeException("Could not create a valid URI from the given file path name: " + rex.getMessage()); 
			}
		}
		
		// check if have not been set and use the defaults in that case
		if (this.writeMode == null) {
			this.writeMode = DEFAULT_WRITE_MODE;
		}
		
		if (this.outputDirectoryMode == null) {
			this.outputDirectoryMode = DEFAULT_OUTPUT_DIRECTORY_MODE;
		}
	}

	@Override
	public void open(int taskNumber, int numTasks) throws IOException {
		if (taskNumber < 0 || numTasks < 1) {
			throw new IllegalArgumentException("TaskNumber: " + taskNumber + ", numTasks: " + numTasks);
		}
		
		if (LOG.isDebugEnabled()) {
			LOG.debug("Opening stream for output (" + (taskNumber+1) + "/" + numTasks + "). WriteMode=" + writeMode +
					", OutputDirectoryMode=" + outputDirectoryMode);
		}
		
		Path p = this.outputFilePath;
		if (p == null) {
			throw new IOException("The file path is null.");
		}
		
		final FileSystem fs = p.getFileSystem();

		// if this is a local file system, we need to initialize the local output directory here
		if (!fs.isDistributedFS()) {
			
			if (numTasks == 1 && outputDirectoryMode == OutputDirectoryMode.PARONLY) {
				// output should go to a single file
				
				// prepare local output path. checks for write mode and removes existing files in case of OVERWRITE mode
				if(!fs.initOutPathLocalFS(p, writeMode, false)) {
					// output preparation failed! Cancel task.
					throw new IOException("Output path '" + p.toString() + "' could not be initialized. Canceling task...");
				}
			}
			else {
				// numTasks > 1 || outDirMode == OutputDirectoryMode.ALWAYS
				
				if(!fs.initOutPathLocalFS(p, writeMode, true)) {
					// output preparation failed! Cancel task.
					throw new IOException("Output directory '" + p.toString() + "' could not be created. Canceling task...");
				}
			}
		}



		// Suffix the path with the parallel instance index, if needed
		this.actualFilePath = (numTasks > 1 || outputDirectoryMode == OutputDirectoryMode.ALWAYS) ? p.suffix("/" + getDirectoryFileName(taskNumber)) : p;

		// create output file
		this.stream = fs.create(this.actualFilePath, writeMode);
		
		// at this point, the file creation must have succeeded, or an exception has been thrown
		this.fileCreated = true;
	}

	@Override
	public void close() throws IOException {
		final FSDataOutputStream s = this.stream;
		if (s != null) {
			this.stream = null;
			s.close();
		}
	}
}

• FileOutputFormat继承了RichOutputFormat，实现了InitializeOnMaster(initializeGlobal方法)、CleanupWhenUnsuccessful(tryCleanupOnError方法)接口
• initializeGlobal主要是判断，如果文件是分布式系统文件，那么就在启动的时候全局初始化一下；tryCleanupOnError方法先close，然后再delete文件
• FileOutputFormat还实现了OutputFormat接口的configure、open、close方法，而writeRecord方法由子类来实现；configure方法主要是配置outputFilePath、writeMode、outputDirectoryMode这几个属性；open方法则根据taskNumber来获取actualFilePath(对于numTasks大于1的，则根据tasknumber在配置的outputFilePath目录下新增文件，文件名为tasknumber对应的数值+1)，然后创建stream；close方法只要是关闭stream

RichOutputFormat

flink-core-1.7.0-sources.jar!/org/apache/flink/api/common/io/RichOutputFormat.java

/**
 * An abstract stub implementation for Rich output formats.
 * Rich formats have access to their runtime execution context via {@link #getRuntimeContext()}.
 */
@Public
public abstract class RichOutputFormat<IT> implements OutputFormat<IT> {
	
	private static final long serialVersionUID = 1L;
	
	// --------------------------------------------------------------------------------------------
	//  Runtime context access
	// --------------------------------------------------------------------------------------------
	
	private transient RuntimeContext runtimeContext;

	public void setRuntimeContext(RuntimeContext t) {
		this.runtimeContext = t;
	}
	
	public RuntimeContext getRuntimeContext() {
		if (this.runtimeContext != null) {
			return this.runtimeContext;
		} else {
			throw new IllegalStateException("The runtime context has not been initialized yet. Try accessing " +
					"it in one of the other life cycle methods.");
		}
	}
}

• RichOutputFormat声明实现OutputFormat接口，它主要是增加了RuntimeContext属性

OutputFormat

flink-core-1.7.0-sources.jar!/org/apache/flink/api/common/io/OutputFormat.java

/**
 * The base interface for outputs that consumes records. The output format
 * describes how to store the final records, for example in a file.
 * <p>
 * The life cycle of an output format is the following:
 * <ol>
 *   <li>configure() is invoked a single time. The method can be used to implement initialization from
 *       the parameters (configuration) that may be attached upon instantiation.</li>
 *   <li>Each parallel output task creates an instance, configures it and opens it.</li>
 *   <li>All records of its parallel instance are handed to the output format.</li>
 *   <li>The output format is closed</li>
 * </ol>
 * 
 * @param <IT> The type of the consumed records. 
 */
@Public
public interface OutputFormat<IT> extends Serializable {
	
	/**
	 * Configures this output format. Since output formats are instantiated generically and hence parameterless, 
	 * this method is the place where the output formats set their basic fields based on configuration values.
	 * <p>
	 * This method is always called first on a newly instantiated output format. 
	 *  
	 * @param parameters The configuration with all parameters.
	 */
	void configure(Configuration parameters);
	
	/**
	 * Opens a parallel instance of the output format to store the result of its parallel instance.
	 * <p>
	 * When this method is called, the output format it guaranteed to be configured.
	 * 
	 * @param taskNumber The number of the parallel instance.
	 * @param numTasks The number of parallel tasks.
	 * @throws IOException Thrown, if the output could not be opened due to an I/O problem.
	 */
	void open(int taskNumber, int numTasks) throws IOException;
	
	
	/**
	 * Adds a record to the output.
	 * <p>
	 * When this method is called, the output format it guaranteed to be opened.
	 * 
	 * @param record The records to add to the output.
	 * @throws IOException Thrown, if the records could not be added to to an I/O problem.
	 */
	void writeRecord(IT record) throws IOException;
	
	/**
	 * Method that marks the end of the life-cycle of parallel output instance. Should be used to close
	 * channels and streams and release resources.
	 * After this method returns without an error, the output is assumed to be correct.
	 * <p>
	 * When this method is called, the output format it guaranteed to be opened.
	 *  
	 * @throws IOException Thrown, if the input could not be closed properly.
	 */
	void close() throws IOException;
}

• OutputFormat接口定义了configure、open、writeRecord、close方法

小结

• DataStream的writeAsText方法创建了TextOutputFormat，然后通过OutputFormatSinkFunction包装为sink function
• TextOutputFormat继承了FileOutputFormat，其open方法主要是调用FileOutputFormat的open方法，而writeRecord方法则直接往stream进行write，写完一条record之后再写一个换行(\n)
• FileOutputFormat继承了RichOutputFormat，实现了InitializeOnMaster(initializeGlobal方法)、CleanupWhenUnsuccessful(tryCleanupOnError方法)接口，以及OutputFormat接口的configure、open、close方法，而writeRecord方法由子类来实现；
• FileOutputFormat的open方法则根据taskNumber来获取actualFilePath(对于numTasks大于1的，则根据tasknumber在配置的outputFilePath目录下新增文件，文件名为tasknumber对应的数值+1)，然后创建stream
• RichOutputFormat声明实现OutputFormat接口，它主要是增加了RuntimeContext属性；OutputFormat接口则定义了configure、open、writeRecord、close方法

doc

• TextOutputFormat