epoll I/O复用分析

epoll I/O复用分析

typedef union epoll_data {
        void *ptr;
        int fd;
        uint32_t u32;
        uint64_t u64;
} epoll_data_t;

struct epoll_event {
        uint32_t     events;      /* Epoll events */
        epoll_data_t data;        /* User data variable */
};

int epoll_create(int size)；
int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event)；
int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);

• events则是分配好的 epoll_event结构体数组，epoll将会把发生的事件复制到 events数组中（events不可以是空指针，内核只负责把数据复制到这个 events数组中，不会去帮助我们在用户态中分配内存。内核这种做法效率很高）

  • 数组名也是一个指针。数组名是常量指针，存放的是首元素地址，在方法入参中，数组名指针会被强转为一个普通指针

      struct epoll_event eventItems[EPOLL_MAX_EVENTS];
      int eventCount = epoll_wait(mEpollFd.get(), eventItems, EPOLL_MAX_EVENTS, timeoutMillis);
  
  

Message

looper对外提供的，应用可自定义的消息。looper负责message的接收和转发，但消息的处理，由应用指定的Handler处理

struct Message {
    Message() : what(0) { }
    Message(int w) : what(w) { }

    /* The message type. (interpretation is left up to the handler) */
    int what;
};

MessageHandler

MessageHandler是一个接口，需要应用继承并实现

class MessageHandler : public virtual RefBase {
protected:
    virtual ~MessageHandler();

public:
    /**
     * Handles a message.
     */
    virtual void handleMessage(const Message& message) = 0;
};

LooperCallback

LooperCallback和Looper_callbackFunc目的：事件发生时，应用指定的callback方法。一个是类，一个是函数指针

class LooperCallback : public virtual RefBase {
protected:
    virtual ~LooperCallback();

public:
    /**
     * Handles a poll event for the given file descriptor.
     * It is given the file descriptor it is associated with,
     * a bitmask of the poll events that were triggered (typically EVENT_INPUT),
     * and the data pointer that was originally supplied.
     *
     * Implementations should return 1 to continue receiving callbacks, or 0
     * to have this file descriptor and callback unregistered from the looper.
     */
    virtual int handleEvent(int fd, int events, void* data) = 0;
};

typedef int (*Looper_callbackFunc)(int fd, int events, void* data);

Looper

总结

• one looper one thread，通过TLS实现
• looper可接收和转发应用自定义消息

class Looper : public RefBase {

• POLL_WAKE 被唤醒，没有callback被执行，没有文件fd准备

• POLL_CALLBACK 执行callback，包含message或文件fd

• POLL_TIMEOUT 超时返回，没有callback被执行，没有文件fd准备

• EVENT_INPUT fd监听输入事件，即read

• EVENT_OUTPUT fd监听输出事件，即write

• pollOnce 在指定time监听事件并返回

    int pollOnce(int timeoutMillis, int* outFd, int* outEvents, void** outData);
    inline int pollOnce(int timeoutMillis) {
        return pollOnce(timeoutMillis, nullptr, nullptr, nullptr);
    }

addFd

• ident和callback，if callback else ident

    int addFd(int fd, int ident, int events, Looper_callbackFunc callback, void* data);
    int addFd(int fd, int ident, int events, const sp<LooperCallback>& callback, void* data);

sendMessage

入队消息，该消息是一次性消费的

    void sendMessage(const sp<MessageHandler>& handler, const Message& message);

setForThread

一个looper可以不与特定线程bind，但：

• 调用pollOnce的线程，与创建looper的线程，可能并不是同一个，looper便不是线程安全的，变量访问需要锁机制

将looper与特定线程bind（一般是创建looper的线程，但looper可在线程间转移），即 one loop one thread

• 多个线程可拥有同一个looper，即多个thread可共享同一个looper
• 一个looper不可有多个线程
• looper的事件监听及其处理是在同一个线程的。但其实也可以looper1对事件进行预处理，入队looper2，wake looper2对事件进行处理，looper1返回继续监听事件

    static sp<Looper> prepare(int opts);
	static void setForThread(const sp<Looper>& looper);
    static sp<Looper> getForThread();

looper对处理对象的封装

• Request 对文件fd及其callback封装
• Response epoll对相应request的返回事件封装
• MessageEnvelope 应用自定义消息封装

    struct Request {
        int fd;
        int ident;
        int events;
        int seq;
        sp<LooperCallback> callback;
        void* data;

        void initEventItem(struct epoll_event* eventItem) const;
    };

    struct Response {
        int events;
        Request request;
    };

    struct MessageEnvelope {
        MessageEnvelope() : uptime(0) { }

        MessageEnvelope(nsecs_t u, const sp<MessageHandler> h,
                const Message& m) : uptime(u), handler(h), message(m) {
        }

        nsecs_t uptime;
        sp<MessageHandler> handler;
        Message message;
    };

• mMessageEnvelopes mRequests mResponses

    Vector<MessageEnvelope> mMessageEnvelopes; // guarded by mLock
    // Locked list of file descriptor monitoring requests.
    KeyedVector<int, Request> mRequests;  // guarded by mLock
    // This state is only used privately by pollOnce and does not require a lock since
    // it runs on a single thread.
    Vector<Response> mResponses;

• mEpollFd epoll_create返回的epoll文件描述符
• mWakeEventFd 唤醒epoll_wait的文件描述符

    android::base::unique_fd mEpollFd;  // guarded by mLock but only modified on the looper thread
    android::base::unique_fd mWakeEventFd;  // immutable

具体分析

addFd

• 检查，如果没有callback，需要检查mAllowNonCallbacks，若无则报错；或者其ident小于0

int Looper::addFd(int fd, int ident, int events, const sp<LooperCallback>& callback, void* data) {
    

• 将此次addFd，翻译为一个request，并根据fd，检查是否已在mRequests队列中

  • 不在队列中，说明是一个新的请求：

    • int epollResult = epoll_ctl(mEpollFd.get(), EPOLL_CTL_ADD, fd, &eventItem);
      mRequests.add(fd, request);
      

    • 在队列中，说明这次请求是修改感兴趣的事件event

      int epollResult = epoll_ctl(mEpollFd.get(), EPOLL_CTL_MOD, fd, &eventItem);
      mRequests.replaceValueAt(requestIndex, request);
      

sendMessageAtTime

• while，根据uptime(消息处理时间)在队列中查找index，即消息按时间大小排序
• 将handler及其对应的message装入信封MessageEnvelope

        size_t messageCount = mMessageEnvelopes.size();
        while (i < messageCount && uptime >= mMessageEnvelopes.itemAt(i).uptime) {//mMessageEnvelopes按时间长短排序
            i += 1;
        }
        MessageEnvelope messageEnvelope(uptime, handler, message);
        mMessageEnvelopes.insertAt(messageEnvelope, i, 1);

pollInner

• mResponses仅looper所在线程访问，故不用加锁

    // Poll.
    int result = POLL_WAKE;
    mResponses.clear();
    mResponseIndex = 0;

• epoll_wait阻塞，等待事件返回
  • mPolling 表示当前looper空闲，可添加消息 wake 并处理

    // We are about to idle.
    mPolling = true;

    struct epoll_event eventItems[EPOLL_MAX_EVENTS];
    int eventCount = epoll_wait(mEpollFd.get(), eventItems, EPOLL_MAX_EVENTS, timeoutMillis);

    // No longer idling.
    mPolling = false;

• 无文件fd事件就绪，超时返回，可能接下来需要处理应用自定义消息

    // Check for poll timeout.
    if (eventCount == 0) {
        result = POLL_TIMEOUT;
        goto Done;
    }

• 有文件fd事件就绪
  • mWakeEventFd如果是wake事件
  • mRequests.indexOfKey(fd) 已注册fd的事件回调，初步加工为response

    for (int i = 0; i < eventCount; i++) {
        int fd = eventItems[i].data.fd;
        uint32_t epollEvents = eventItems[i].events;
        if (fd == mWakeEventFd.get()) {
            if (epollEvents & EPOLLIN) {
                awoken();
            } else {
                ALOGW("Ignoring unexpected epoll events 0x%x on wake event fd.", epollEvents);
            }
        } else {
            ssize_t requestIndex = mRequests.indexOfKey(fd);
            if (requestIndex >= 0) {
                int events = 0;
                if (epollEvents & EPOLLIN) events |= EVENT_INPUT;
                if (epollEvents & EPOLLOUT) events |= EVENT_OUTPUT;
                if (epollEvents & EPOLLERR) events |= EVENT_ERROR;
                if (epollEvents & EPOLLHUP) events |= EVENT_HANGUP;
                pushResponse(events, mRequests.valueAt(requestIndex));
            } else {
                ALOGW("Ignoring unexpected epoll events 0x%x on fd %d that is "
                        "no longer registered.", epollEvents, fd);
            }
        }
    }

• 处理到时的message

  handler->handleMessage(message)

      // Invoke pending message callbacks.
      mNextMessageUptime = LLONG_MAX;
      while (mMessageEnvelopes.size() != 0) {
          nsecs_t now = systemTime(SYSTEM_TIME_MONOTONIC);
          const MessageEnvelope& messageEnvelope = mMessageEnvelopes.itemAt(0);
          if (messageEnvelope.uptime <= now) {
              // Remove the envelope from the list.
              // We keep a strong reference to the handler until the call to handleMessage
              // finishes.  Then we drop it so that the handler can be deleted *before*
              // we reacquire our lock.
              { // obtain handler
                  sp<MessageHandler> handler = messageEnvelope.handler;
                  Message message = messageEnvelope.message;
                  mMessageEnvelopes.removeAt(0);
                  mSendingMessage = true;
                  mLock.unlock();
  
  #if DEBUG_POLL_AND_WAKE || DEBUG_CALLBACKS
                  ALOGD("%p ~ pollOnce - sending message: handler=%p, what=%d",
                          this, handler.get(), message.what);
  #endif
                  handler->handleMessage(message);
              } // release handler
  
              mLock.lock();
              mSendingMessage = false;
              result = POLL_CALLBACK;
          } else {
              // The last message left at the head of the queue determines the next wakeup time.
              mNextMessageUptime = messageEnvelope.uptime;
              break;
          }
      }
  

• 最后处理response·

  response.request.callback->handleEvent(fd, events, data)

    // Invoke all response callbacks.先把response放入列表，延时处理
    for (size_t i = 0; i < mResponses.size(); i++) {
        Response& response = mResponses.editItemAt(i);
        if (response.request.ident == POLL_CALLBACK) {
            int fd = response.request.fd;
            int events = response.events;
            void* data = response.request.data;
#if DEBUG_POLL_AND_WAKE || DEBUG_CALLBACKS
            ALOGD("%p ~ pollOnce - invoking fd event callback %p: fd=%d, events=0x%x, data=%p",
                    this, response.request.callback.get(), fd, events, data);
#endif
            // Invoke the callback.  Note that the file descriptor may be closed by
            // the callback (and potentially even reused) before the function returns so
            // we need to be a little careful when removing the file descriptor afterwards.
            int callbackResult = response.request.callback->handleEvent(fd, events, data);
            if (callbackResult == 0) {
                removeFd(fd, response.request.seq);
            }

            // Clear the callback reference in the response structure promptly because we
            // will not clear the response vector itself until the next poll.
            response.request.callback.clear();
            result = POLL_CALLBACK;
        }
    }

• pthread_once_t 仅调用一次
• pthread_key_t 线程局部对象，在不同的线程有不同的含义

static pthread_once_t gTLSOnce = PTHREAD_ONCE_INIT;
static pthread_key_t gTLSKey = 0;

• 在获取线程对象时才通过pthread_once创建initTLSKey

static pthread_once_t gTLSOnce = PTHREAD_ONCE_INIT;

sp<Looper> Looper::getForThread() {
    int result = pthread_once(& gTLSOnce, initTLSKey);
    return (Looper*)pthread_getspecific(gTLSKey);
}

void Looper::initTLSKey() {
    int error = pthread_key_create(&gTLSKey, threadDestructor);
}

messageQueue

上述讲解Looper时，应用自定义消息为message。那runnable或callable等task此类消息如何利用message封装呢？

• class Task : public MessageHandler

  • 继承MessageHandler，说明这是一个可被Looper接收的消息

  • packaged_task<T()> 将入参的可调用对象封装为一个promise

template <typename F>
class Task : public MessageHandler {
    template <typename G>
    friend auto makeTask(G&&);

    explicit Task(F&& f) : mTask(std::move(f)) {}

    void handleMessage(const Message&) override { mTask(); }

    using T = std::invoke_result_t<F>;
    std::packaged_task<T()> mTask;
};

makeTask

makeTask将一个普通的可调用对象封装为一个promise

template <typename F>
inline auto makeTask(F&& f) {
    sp<Task<F>> task = new Task<F>(std::move(f));
    return std::make_pair(task, task->mTask.get_future());
}

• SurfaceFlinger::schedule

  语义：将一个可调用对象封装为一个可被looper接收的消息及其一个feature，即在子线程调用schedule，该消息会在looper线程执行，子线程阻塞并等待结果返回

template <typename F, typename T>
inline std::future<T> SurfaceFlinger::schedule(F&& f) {
    auto [task, future] = makeTask(std::move(f));
    mEventQueue->postMessage(std::move(task));
    return std::move(future);
}

• 将一个可调用对象封装为一个looper可接收的消息及其feature(任务完成可能有返回值)，即pair[Task,feature]，再将task传给looper处理，将feature传给调用者