持续创作，加速成长！这是我参与「掘金日新计划 · 10 月更文挑战」的第15天，点击查看活动详情

flume-ftp-sink

背景

因为项目需要，需要写一个flume-ftp-sink，而flume官方不支持ftp sink所以，我要先研究一下原来架构使用的flume-hdfs-sink，在自定义的写一个flume-ftp-sink。具体请看

自定义flumesink

flume-ftp-sink

源码

核心代码：

@Override
    public synchronized void start() {
        sinkCounter.start();
        connectServer();
        super.start();
        if (rollInterval > 0) {

            rollService = Executors.newScheduledThreadPool(
                    1,
                    new ThreadFactoryBuilder().setNameFormat(
                            "rollingFileSink-roller-" +
                                    Thread.currentThread().getId() + "-%d").build());

            /*
             * Every N seconds, mark that it's time to rotate. We purposefully do NOT
             * touch anything other than the indicator flag to avoid error handling
             * issues (e.g. IO exceptions occuring in two different threads.
             * Resist the urge to actually perform rotation in a separate thread!
             */
            rollService.scheduleAtFixedRate(new Runnable() {

                @Override
                public void run() {
                    logger.debug("Marking time to rotate file {}",
                            pathController.getCurrentFile());
                    shouldRotate = true;
                }

            }, rollInterval, rollInterval, TimeUnit.SECONDS);
        } else {
            logger.info("RollInterval is not valid, file rolling will not happen.");
        }
        logger.info("RollingFileSink {} started.", getName());
    }

    @Override
    public synchronized void stop() {
        disConnectServer();
        sinkCounter.stop();
        super.stop();
        if (rollInterval > 0) {
            rollService.shutdown();

            while (!rollService.isTerminated()) {
                try {
                    rollService.awaitTermination(1, TimeUnit.SECONDS);
                } catch (InterruptedException e) {
                    logger.debug("Interrupted while waiting for roll service to stop. " +
                            "Please report this.", e);
                }
            }
        }
        logger.info("RollingFile sink {} stopped. Event metrics: {}",
                getName(), sinkCounter);
    }

    @Override
    public Status process() throws EventDeliveryException {
        Status status = Status.READY;
        Channel channel = getChannel();
        Transaction transaction = channel.getTransaction();
        try {
            transaction.begin();
            verifyConnection();

            List<Event> batch = new ArrayList<>();

            for (int i = 0; i < batchSize; i++) {
                Event event = channel.take();
                if (event == null) {
                    break;
                }
                realDirPath = BucketPath.escapeString(ftpDirPath, event.getHeaders(),
                        timeZone, needRounding, roundUnit, roundValue, useLocalTime);
                realprefix = BucketPath.escapeString(prefix, event.getHeaders(),
                        timeZone, needRounding, roundUnit, roundValue, useLocalTime);
                batch.add(event);
            }

            if (batch.size() == 0) {
                sinkCounter.incrementBatchEmptyCount();
                status = Status.BACKOFF;
            } else {
                if (batch.size() < batchSize) {
                    sinkCounter.incrementBatchUnderflowCount();
                } else {
                    sinkCounter.incrementBatchCompleteCount();
                }
                sinkCounter.addToEventDrainAttemptCount(batch.size());
                dealEventList(batch);
            }

            sinkCounter.addToEventDrainSuccessCount(batch.size());
        } catch (Exception ex) {
            logger.error(ex.getMessage(), ex);
            sinkCounter.incrementEventWriteOrChannelFail(ex);
            transaction.rollback();
            throw new EventDeliveryException("Failed to send events", ex);
        } finally {
            transaction.commit();
            transaction.close();
        }

        return status;
    }

配置参数

Property Name	Default	Description
type	-	need to be com.jzy.flume.FlumeFtpSink
host	-	host
port	21	port
user	-	user
password	-	password
ftpDirPath	/	ftp upload dir
prefix	DATA-	file prefix
suffix	.dat	file prefix
tempSuffix	.tmp	temp suffix when uploading
batchSize	1000	max records of a file
rollInterval	30	Number of seconds to wait before rolling current file (0 = never roll based on time interval)