掌握Android图像显示原理下(二)

4,075 阅读18分钟

接着上一篇文章掌握Android图像显示原理下(一)

BufferQueue的创建

在前面分析Layer的创建时,有提到会在Layer的初始化函数onFirstRef中会创建BufferQueue,onFirstRef函数实现如下

/frameworks/native/services/surfaceflinger/Layer.cpp

void Layer::onFirstRef() {
    // Creates a custom BufferQueue for SurfaceFlingerConsumer to use
    sp<IGraphicBufferProducer> producer;
    sp<IGraphicBufferConsumer> consumer;
    BufferQueue::createBufferQueue(&producer, &consumer, true);
    mProducer = new MonitoredProducer(producer, mFlinger, this);
    mSurfaceFlingerConsumer = new SurfaceFlingerConsumer(consumer, mTextureName, this);
    mSurfaceFlingerConsumer->setConsumerUsageBits(getEffectiveUsage(0));
    mSurfaceFlingerConsumer->setContentsChangedListener(this);
    mSurfaceFlingerConsumer->setName(mName);

    if (mFlinger->isLayerTripleBufferingDisabled()) {
        mProducer->setMaxDequeuedBufferCount(2);
    }

    const sp<const DisplayDevice> hw(mFlinger->getDefaultDisplayDevice());
    updateTransformHint(hw);
}

onFirstRef函数中关键的一步就是调用createBufferQueue创建BufferQueue,它的实现如下

/frameworks/native/libs/gui/BufferQueue.cpp

void BufferQueue::createBufferQueue(sp<IGraphicBufferProducer>* outProducer,
        sp<IGraphicBufferConsumer>* outConsumer,
        bool consumerIsSurfaceFlinger) {

    sp<BufferQueueCore> core(new BufferQueueCore());    
    sp<IGraphicBufferProducer> producer(new BufferQueueProducer(core, consumerIsSurfaceFlinger));
    sp<IGraphicBufferConsumer> consumer(new BufferQueueConsumer(core));
  
    *outProducer = producer;
    *outConsumer = consumer;
}

createBufferQueue做了三件事情

  1. 创建BufferQueueCore
  2. 创建BufferQueueProducer
  3. 创建BufferQueueConsumer

下面详细讲一下这三件事情

BufferQueueCore

在概述里面提到过,BufferQueueCore拥有一个slots数组用来存储GraphicBuffer,并且最多可能存放64个GraphicBuffer。这里我们会进一步了解BufferQueueCore。先看看BufferQueueCore的头文件 /frameworks/native/include/gui/BufferQueueCore.h

namespace android {

class BufferQueueCore : public virtual RefBase {

    friend class BufferQueueProducer;
    friend class BufferQueueConsumer;

……

private:
    
    ……
    BufferQueueDefs::SlotsType mSlots;
    Fifo mQueue;
    std::set<int> mFreeSlots;
    std::list<int> mFreeBuffers;
    std::list<int> mUnusedSlots;
    std::set<int> mActiveBuffers;
    ……

}; 

} 

#endif

从BufferQueueCore的头文件可以看到,BufferQueueCore除了持有BufferQueueProducer,BufferQueueConsumer和mSlots,还持有很多其他的数据,如mQueue,mFreeSlots,mFreeBuffers等。下面介绍一下关键的几个成员变量。

  • mSlots就是一个BufferSlot数组,大小为NUM_BUFFER_SLOTS,也就是64个,BufferSlot主要是用来绑定GraphicBuffer的,一个BufferSlot绑定一个GraphicBuffer
//文件-->/frameworks/native/include/gui/BufferQueueDefs.h
namespace android {
    class BufferQueueCore;
    namespace BufferQueueDefs {
        typedef BufferSlot SlotsType[NUM_BUFFER_SLOTS];
    } 
} 

//文件-->/frameworks/native/include/ui/BufferQueueDefs.h
namespace android {
    namespace BufferQueueDefs {        
        static constexpr int NUM_BUFFER_SLOTS = 64;
    } 
} 
  • mQueue:以队列的方式存放图像消费者提交的GraphicBuffer
  • mFreeSlots:代表所有没有绑定GraphicBuffer的BufferSlot集合
  • mActiveBuffers:代表所有绑定了GraphicBuffer的BufferSlot集合,并且BufferSlot状态不为FREE
  • mUnusedSlots:代表当前没有使用的BufferSlot集合。

这里介绍一下BufferSlot,它的数据结构如下。可以看到BufferSlot持有一个GraphicBuffer,以及BufferState,还有一些其他的数据。

/frameworks/native/include/gui/BufferSlot.h

struct BufferSlot {

    BufferSlot()
    : mGraphicBuffer(nullptr),
      mBufferState(),
      ……{
    }

    sp<GraphicBuffer> mGraphicBuffer;
    BufferState mBufferState;
    ……

};

BufferState又分为五种状态,FREE,DEQUEUED,QUEUED,ACQUIRED,SHARED。了解了 BufferSlot和BufferState,我们也就能理解BufferQueueCore为什么会有这么多存储GraphicBuffer的数据结构,如slots,mFreeSlots,mFreeBuffers等等,这些数据结构都是为了给BufferSlot分类,以便获取GraphicBuffer时更加高效。

BufferQueueProducer

了解了BufferQueueCore,我们接着看BufferQueueProducer,在前面概述中提到,Surface会中持BufferQueueProducer的BP代理。它的构造函数如下。

/frameworks/native/libs/gui/BufferQueueProducer.cpp

BufferQueueProducer::BufferQueueProducer(const sp<BufferQueueCore>& core,
        bool consumerIsSurfaceFlinger) :
    mCore(core),
    mSlots(core->mSlots),
    …… {}

BufferQueueProducer的构造函数是空方法,在初始化成员变量时,会直接将前面创建好的BufferQueueCore和mSlots赋值到BufferQueueProducer的成员变量中。

我们已经知道,图像生产者通过dequeue函数来获取一块可用的GraphicBuffer,并通过queue来归还绘制了数据的Graphicbuffer。 在这里插入图片描述

那么我们在BufferQueueProducer中看看这两个函数的具体实现。

dequeueBuffer

/frameworks/native/libs/gui/BufferQueueProducer.cpp

status_t BufferQueueProducer::dequeueBuffer(int *outSlot,
        sp<android::Fence> *outFence, uint32_t width, uint32_t height,
        PixelFormat format, uint32_t usage,
        FrameEventHistoryDelta* outTimestamps) {
    
    ……


        int found = BufferItem::INVALID_BUFFER_SLOT;
    while (found == BufferItem::INVALID_BUFFER_SLOT) {
        //1,寻找Free状态的BufferSlot
        status_t status = waitForFreeSlotThenRelock(FreeSlotCaller::Dequeue,&found);
    }



    //2,把找到的slot放到mActiveBuffers中管理
    if (mCore->mSharedBufferSlot != found) {
        mCore->mActiveBuffers.insert(found);
    }
    *outSlot = found;
    ATRACE_BUFFER_INDEX(found);

    attachedByConsumer = mSlots[found].mNeedsReallocation;
    mSlots[found].mNeedsReallocation = false;

    //修改mBufferState状态为DEQUEUE状态
    mSlots[found].mBufferState.dequeue();
    //3,如果找到的GraphicBuffer是空的,或者需要重新申请,则把设置到BufferSlot的参数全部初始化
    if ((buffer == NULL) ||
        buffer->needsReallocation(width, height, format, BQ_LAYER_COUNT, usage))
    {
        mSlots[found].mAcquireCalled = false;
        mSlots[found].mGraphicBuffer = NULL;
        mSlots[found].mRequestBufferCalled = false;
        mSlots[found].mEglDisplay = EGL_NO_DISPLAY;
        mSlots[found].mEglFence = EGL_NO_SYNC_KHR;
        mSlots[found].mFence = Fence::NO_FENCE;
        mCore->mBufferAge = 0;
        mCore->mIsAllocating = true;

        returnFlags |= BUFFER_NEEDS_REALLOCATION;
    } else {
        // We add 1 because that will be the frame number when this buffer
        // is queued
        mCore->mBufferAge =
            mCore->mFrameCounter + 1 - mSlots[found].mFrameNumber;
    }
    ……

   if (returnFlags & BUFFER_NEEDS_REALLOCATION) {
       //4,如果GraphicBuffer为空,则重新创建GraphicBuffer,并放入对应的slot中
        sp<GraphicBuffer> graphicBuffer = new GraphicBuffer(
                width, height, format, BQ_LAYER_COUNT, usage,
                {mConsumerName.string(), mConsumerName.size()});

        status_t error = graphicBuffer->initCheck();

        { 
            if (error == NO_ERROR && !mCore->mIsAbandoned) {
                //将新创建的GraphicBuffer放入对应的BufferSLot中
                graphicBuffer->setGenerationNumber(mCore->mGenerationNumber);
                mSlots[*outSlot].mGraphicBuffer = graphicBuffer;
            }
			……
        } 
    }
    ……
   

    return returnFlags;
}

dequeueBuffer函数做的事情主要如下

  1. 执行waitForFreeSlotThenRelock ,查找BufferQueueCore的Slot中空闲的插槽位置,并且取出BufferSlot中的GraphicBuffer的index,判断当前这个图元是否宽高,像素格式是否和请求一样,不一样则需要重新请求。

  2. 先把found的index添加到mActiveBuffer集合中,标示为活跃状态,并且设置为dequeue状态。

  3. 如果找到的BufferSlot的GraphicBuffer为空或者需要重新申请,则把设置到BufferSlot的参数全部初始化。

  4. 如果需要重新创建GraphicBuffer,则创建一个新的GraphicBuffer,接着调用GraphicBuffer的initCheck进行校验,并把新的GraphicBuffer 加入到mSlot。

queueBuffer

了解了dequeueBuffer的流程,在接着看queueBuffer,queuebuffer是生产者使用完GraphicBuffer后把GraphicBuffer放回BufferQueue,并把BufferState修改成QUEUE状态的流程,我们看一下源码

/frameworks/native/libs/gui/BufferQueueProducer.cpp

status_t BufferQueueProducer::queueBuffer(int slot,
        const QueueBufferInput &input, QueueBufferOutput *output) {
  

    ……
        
    BufferItem item;
    { 

        //错误处理
        ……

        ……

        mSlots[slot].mFence = acquireFence;
        //1,修改mBufferState状态为QUEUE状态
        mSlots[slot].mBufferState.queue();

        // Increment the frame counter and store a local version of it
        // for use outside the lock on mCore->mMutex.
        ++mCore->mFrameCounter;
        currentFrameNumber = mCore->mFrameCounter;
        mSlots[slot].mFrameNumber = currentFrameNumber;
		//2,BufferItem复制
        item.mAcquireCalled = mSlots[slot].mAcquireCalled;
        item.mGraphicBuffer = mSlots[slot].mGraphicBuffer;
        item.mCrop = crop;
        item.mTransform = transform &
                ~static_cast<uint32_t>(NATIVE_WINDOW_TRANSFORM_INVERSE_DISPLAY);
        item.mTransformToDisplayInverse =
                (transform & NATIVE_WINDOW_TRANSFORM_INVERSE_DISPLAY) != 0;
        item.mScalingMode = static_cast<uint32_t>(scalingMode);
        item.mTimestamp = requestedPresentTimestamp;
        item.mIsAutoTimestamp = isAutoTimestamp;
        item.mDataSpace = dataSpace;
        item.mFrameNumber = currentFrameNumber;
        item.mSlot = slot;
        item.mFence = acquireFence;
        item.mFenceTime = acquireFenceTime;
        item.mIsDroppable = mCore->mAsyncMode ||
                mCore->mDequeueBufferCannotBlock ||
                (mCore->mSharedBufferMode && mCore->mSharedBufferSlot == slot);
        item.mSurfaceDamage = surfaceDamage;
        item.mQueuedBuffer = true;
        item.mAutoRefresh = mCore->mSharedBufferMode && mCore->mAutoRefresh;

        mStickyTransform = stickyTransform;

       

        output->bufferReplaced = false;
        if (mCore->mQueue.empty()) {
            // BufferQueueCore的BufferSlot队列为空时,直接push到队尾
            mCore->mQueue.push_back(item);
            frameAvailableListener = mCore->mConsumerListener;
        } else {
            // 队列不为空,需要判断一下last BufferItem是否被替换,如果可以替换就替换,如果不可以替换就直接把BufferItem放到mQueue尾部
            const BufferItem& last = mCore->mQueue.itemAt(
                    mCore->mQueue.size() - 1);
            if (last.mIsDroppable) {

                if (!last.mIsStale) {
                    mSlots[last.mSlot].mBufferState.freeQueued();

                    if (!mCore->mSharedBufferMode &&
                            mSlots[last.mSlot].mBufferState.isFree()) {
                        mSlots[last.mSlot].mBufferState.mShared = false;
                    }            
                    if (!mSlots[last.mSlot].mBufferState.isShared()) {
                        mCore->mActiveBuffers.erase(last.mSlot);
                        mCore->mFreeBuffers.push_back(last.mSlot);
                        output->bufferReplaced = true;
                    }
                }

                // Overwrite the droppable buffer with the incoming one
                mCore->mQueue.editItemAt(mCore->mQueue.size() - 1) = item;
                frameReplacedListener = mCore->mConsumerListener;
            } else {
                mCore->mQueue.push_back(item);
                frameAvailableListener = mCore->mConsumerListener;
            }
        }

       ……
    } 

    return NO_ERROR;
}

queueBuffer的流程主要有这两件事情:

  1. 将对应BufferSlot状态设置成QUEUED
  2. 创建BufferItem对象,并将GraphicBuffer的等数据复制给BufferItem,并入队到BufferQueueCore的mQueue队列中,这样可以方便图像消费者直接按先进先出的顺序从mQueue队列取出GraphicBuffer使用
BufferQueueConsumer

了解了BufferQueueCore和BufferQueueProducer,接着看BufferQueue的最后一个元素:BufferQueueConsumer,在概览中讲到过,图像消费者SurfceFlinger通过acquire来获取图像缓冲区,通过release来释放该缓冲区。下面就看看BufferQueueConsumer中acquire和release两个操作的具体流程。

acquireBuffer

先看acquireBuffer的过程

/frameworks/native/libs/gui/BufferQueueConsumer.cpp

status_t BufferQueueConsumer::acquireBuffer(BufferItem* outBuffer,
        nsecs_t expectedPresent, uint64_t maxFrameNumber) {
 
    int numDroppedBuffers = 0;
    sp<IProducerListener> listener;
    {
        //1,检查acquire的buffer的数量是否超出了限制
        ……

        //2.检查BufferQueueCore中的mQueue队列是否为空
        if (mCore->mQueue.empty() && !sharedBufferAvailable) {
            return NO_BUFFER_AVAILABLE;
        }

        //获取BufferQueueCore中的mQueue队列的迭代器
        BufferQueueCore::Fifo::iterator front(mCore->mQueue.begin());


        int slot = BufferQueueCore::INVALID_BUFFER_SLOT;

        if (sharedBufferAvailable && mCore->mQueue.empty()) {
            //共享Buffer模式
        } else {
            //从front获取对应的slot
            slot = front->mSlot;
            *outBuffer = *front;
        }

       

        if (!outBuffer->mIsStale) {
            mSlots[slot].mAcquireCalled = true;           
            if (mCore->mQueue.empty()) {
                mSlots[slot].mBufferState.acquireNotInQueue();
            } else {
                //将BufferState状态改为acquire
                mSlots[slot].mBufferState.acquire();
            }
            mSlots[slot].mFence = Fence::NO_FENCE;
        }

        if (outBuffer->mAcquireCalled) {
            outBuffer->mGraphicBuffer = NULL;
        }
		//将该Buffer从mQueue中移除
        mCore->mQueue.erase(front);

        
    }

    //回调
	……

    return NO_ERROR;
}

acquireBuffer函数中的逻辑也非常的清晰,主要就是这几件事情

  1. 判断BufferQueueCore中的mQueue是否为空,mQueue就是前面BufferQueueProducer调用queueBuffer函数时,将缓冲区入队的容器。
  2. 取出对应的BufferSlot
  3. 将BufferState改为acquire状态
  4. 将该Buffer从mQueue中移除
releaseBuffer

再接着看releaseBuffer的流程

/frameworks/native/libs/gui/BufferQueueConsumer.cpp

status_t BufferQueueConsumer::releaseBuffer(int slot, uint64_t frameNumber,
        const sp<Fence>& releaseFence, EGLDisplay eglDisplay,
        EGLSyncKHR eglFence) {
  
    //slot合法性校验
    if (slot < 0 || slot >= BufferQueueDefs::NUM_BUFFER_SLOTS ||
            releaseFence == NULL) {      
        return BAD_VALUE;
    }


    {       
		//slot状态校验
        if (!mSlots[slot].mBufferState.isAcquired()) {         
            return BAD_VALUE;
        }

        
        ……
        //将BufferState转为release
        mSlots[slot].mBufferState.release();

        ……
        // 将slot放在FreeBuffers中
        if (!mSlots[slot].mBufferState.isShared()) {
            mCore->mActiveBuffers.erase(slot);
            mCore->mFreeBuffers.push_back(slot);
        }
    } 

    // 回调
    if (listener != NULL) {
        listener->onBufferReleased();
    }

    return NO_ERROR;
}

releaseBuffer的流程也比较简单,主要是将BufferState改为release状态,并将该BufferlSlot放在FreeBuffers这个容器中。

GraphicBuffer的创建

GraphicBuffer是图像缓冲区的最后一个组成部分,也是最基本最重要的一个单元,在前面讲到BufferQueueProucer通过dequeueBuffer获取BufferSlot时,会判断BufferSlot中的GraphicBuffer是否为空,如果为空就会创建新的GraphicBuffer。

status_t BufferQueueProducer::dequeueBuffer(int *outSlot,
        sp<android::Fence> *outFence, uint32_t width, uint32_t height,
        PixelFormat format, uint32_t usage,
        FrameEventHistoryDelta* outTimestamps) {
    
    ……

   if (returnFlags & BUFFER_NEEDS_REALLOCATION) {
       //如果GraphicBuffer为空,则重新创建GraphicBuffer,并放入对应的slot中
        sp<GraphicBuffer> graphicBuffer = new GraphicBuffer(
                width, height, format, BQ_LAYER_COUNT, usage,
                {mConsumerName.string(), mConsumerName.size()});

        status_t error = graphicBuffer->initCheck();

        { 
            if (error == NO_ERROR && !mCore->mIsAbandoned) {
                //将新创建的GraphicBuffer放入对应的BufferSLot中
                graphicBuffer->setGenerationNumber(mCore->mGenerationNumber);
                mSlots[*outSlot].mGraphicBuffer = graphicBuffer;
            }
			……
        } 
    }
    ……
   

    return returnFlags;
}

可以看到在dequeueBuffer会创建GraphicBuffer,GraphicBuffer是什么呢?我们先看看它的构造函数

/frameworks/native/libs/ui/GraphicBuffer.cpp


GraphicBuffer::GraphicBuffer(uint32_t inWidth, uint32_t inHeight,
        PixelFormat inFormat, uint32_t inLayerCount, uint64_t usage,
        std::string requestorName)
    : GraphicBuffer()
{
    mInitCheck = initWithSize(inWidth, inHeight, inFormat, inLayerCount,
            usage, std::move(requestorName));
}

status_t GraphicBuffer::initWithSize(uint32_t inWidth, uint32_t inHeight,
        PixelFormat inFormat, uint32_t inLayerCount, uint64_t inUsage,
        std::string requestorName)
{
    GraphicBufferAllocator& allocator = GraphicBufferAllocator::get();
    uint32_t outStride = 0;
    status_t err = allocator.allocate(inWidth, inHeight, inFormat, inLayerCount,
            inUsage, &handle, &outStride, mId,
            std::move(requestorName));
    if (err == NO_ERROR) {
        width = static_cast<int>(inWidth);
        height = static_cast<int>(inHeight);
        format = inFormat;
        layerCount = inLayerCount;
        usage = static_cast<int>(inUsage);
        stride = static_cast<int>(outStride);
    }
    return err;
}

GraphicBuffer的构造函数调用initWithSize方法创建指定大小的Buffer,流程如下

  1. 获取GraphicBufferAllocator,它是图像缓冲区的内存分配器
  2. 调用GraphicBufferAllocator.allocate分配缓冲内存

/frameworks/native/libs/ui/GraphicBufferAllocator.cpp

GraphicBufferAllocator::GraphicBufferAllocator()
  : mMapper(GraphicBufferMapper::getInstance()),
    mAllocator(std::make_unique<Gralloc2::Allocator>(
                mMapper.getGrallocMapper()))
{
}

从GraphicBufferAllocator的构造函数中可以看到,真正的缓冲区分配器是Gralloc2::Allocator模块。由于篇幅原因,在这里就不在深入讲解Gralloc了,我们只需要了解Gralloc是位于HAL(硬件抽象层)中的模块,封装了对帧缓冲区的所有访问创建等操作。

图像生产者使用缓冲区

我们已经对图像缓冲区有了一定的了解了,包括Surface,Layer,BufferQueue的创建和使用也都熟悉了。那么接下来看看图像消费者在Android中使用图像缓冲区的场景:软件绘制、硬件加速

软件绘制

图像生产者中介绍Skia时讲到过,软件绘制的入口函数drawSoftware的三步操作。

/frameworks/base/core/java/android/view/ViewRootImpl.java

private boolean drawSoftware(Surface surface, AttachInfo attachInfo, int xoff, int yoff,
        boolean scalingRequired, Rect dirty) {


    final Canvas canvas;

    canvas = mSurface.lockCanvas(dirty);
	……
    mView.draw(canvas);
	……
    surface.unlockCanvasAndPost(canvas);
	……
    return true;
}
  1. 通过mSurface.lockCanvas获取Canvas
  2. 通过draw方法,将根View及其子View遍历绘制到Canvas上
  3. 通过surface.unlockCanvasAndPost将绘制内容提交给surfaceFlinger进行合成

第一步会创建或获取图像缓冲区,第三步会提交图像缓冲区。

获取图像缓冲区

先看第一步的详细步骤

/frameworks/base/core/java/android/view/Surface.java

public Canvas lockCanvas(Rect inOutDirty)
    throws Surface.OutOfResourcesException, IllegalArgumentException {
    synchronized (mLock) {        
        mLockedObject = nativeLockCanvas(mNativeObject, mCanvas, inOutDirty);
        return mCanvas;
    }
}

lockCanvas会调用native方法nativeLockCanvas

/frameworks/base/core/jni/android_view_Surface.cpp

static jlong nativeLockCanvas(JNIEnv* env, jclass clazz,
        jlong nativeObject, jobject canvasObj, jobject dirtyRectObj) {
    sp<Surface> surface(reinterpret_cast<Surface *>(nativeObject));

    if (!isSurfaceValid(surface)) {
        doThrowIAE(env);
        return 0;
    }

    Rect dirtyRect(Rect::EMPTY_RECT);
    Rect* dirtyRectPtr = NULL;

    if (dirtyRectObj) {
        dirtyRect.left   = env->GetIntField(dirtyRectObj, gRectClassInfo.left);
        dirtyRect.top    = env->GetIntField(dirtyRectObj, gRectClassInfo.top);
        dirtyRect.right  = env->GetIntField(dirtyRectObj, gRectClassInfo.right);
        dirtyRect.bottom = env->GetIntField(dirtyRectObj, gRectClassInfo.bottom);
        dirtyRectPtr = &dirtyRect;
    }

    ANativeWindow_Buffer outBuffer;
    //1,获取用来存储图形绘制的buffer
    status_t err = surface->lock(&outBuffer, dirtyRectPtr);
    

    SkImageInfo info = SkImageInfo::Make(outBuffer.width, outBuffer.height,
                                         convertPixelFormat(outBuffer.format),
                                         outBuffer.format == PIXEL_FORMAT_RGBX_8888
                                                 ? kOpaque_SkAlphaType : kPremul_SkAlphaType,
                                         GraphicsJNI::defaultColorSpace());

    SkBitmap bitmap;
    ssize_t bpr = outBuffer.stride * bytesPerPixel(outBuffer.format);
    bitmap.setInfo(info, bpr);
   
    if (outBuffer.width > 0 && outBuffer.height > 0) {
         //将上一个buffer里的图形数据复制到当前bitmap中
        bitmap.setPixels(outBuffer.bits);
    } else {
        // be safe with an empty bitmap.
        bitmap.setPixels(NULL);
    }

    //2,创建一个SKCanvas
    Canvas* nativeCanvas = GraphicsJNI::getNativeCanvas(env, canvasObj);
    //3,给SKCanvas设置Bitmap
    nativeCanvas->setBitmap(bitmap);
    //如果指定了脏区,则设定脏区的区域
    if (dirtyRectPtr) {
        nativeCanvas->clipRect(dirtyRect.left, dirtyRect.top,
                dirtyRect.right, dirtyRect.bottom, SkClipOp::kIntersect);
    }

    if (dirtyRectObj) {
        env->SetIntField(dirtyRectObj, gRectClassInfo.left,   dirtyRect.left);
        env->SetIntField(dirtyRectObj, gRectClassInfo.top,    dirtyRect.top);
        env->SetIntField(dirtyRectObj, gRectClassInfo.right,  dirtyRect.right);
        env->SetIntField(dirtyRectObj, gRectClassInfo.bottom, dirtyRect.bottom);
    }

    sp<Surface> lockedSurface(surface);
    lockedSurface->incStrong(&sRefBaseOwner);
    return (jlong) lockedSurface.get();
}

nativeLockCanvas函数中会调用了surface->lock函数来获取一个供SKCanvas使用的缓冲区,看一下具体实现。

/frameworks/native/libs/gui/Surface.cpp


status_t Surface::lock(
        ANativeWindow_Buffer* outBuffer, ARect* inOutDirtyBounds)
{

	……
    ANativeWindowBuffer* out;
    int fenceFd = -1;
    //1.调用dequeueBuffer获取图像缓冲区
    status_t err = dequeueBuffer(&out, &fenceFd);
   
    if (err == NO_ERROR) {
        sp<GraphicBuffer> backBuffer(GraphicBuffer::getSelf(out));
        const Rect bounds(backBuffer->width, backBuffer->height);

        Region newDirtyRegion;
        if (inOutDirtyBounds) {
            newDirtyRegion.set(static_cast<Rect const&>(*inOutDirtyBounds));
            newDirtyRegion.andSelf(bounds);
        } else {
            newDirtyRegion.set(bounds);
        }

        
        const sp<GraphicBuffer>& frontBuffer(mPostedBuffer);
        const bool canCopyBack = (frontBuffer != 0 &&
                backBuffer->width  == frontBuffer->width &&
                backBuffer->height == frontBuffer->height &&
                backBuffer->format == frontBuffer->format);

        if (canCopyBack) {
            // 2,如果前一帧的缓冲区大小和格式和当前一样,则将前一帧的缓冲区数据拷贝到当前获取的缓冲区来,这样能复用数据,节约性能。
            const Region copyback(mDirtyRegion.subtract(newDirtyRegion));
            if (!copyback.isEmpty()) {
                copyBlt(backBuffer, frontBuffer, copyback, &fenceFd);
            }
        } else {
           ……
        }


       ……

        void* vaddr;
        status_t res = backBuffer->lockAsync(
                GRALLOC_USAGE_SW_READ_OFTEN | GRALLOC_USAGE_SW_WRITE_OFTEN,
                newDirtyRegion.bounds(), &vaddr, fenceFd);

        ……
    }
    return err;
}

我们直接看dequeueBuffer是如何获取缓冲区的

/frameworks/native/libs/gui/Surface.cpp

int Surface::dequeueBuffer(android_native_buffer_t** buffer, int* fenceFd) {
   
   ……
    //创建缓冲区
   status_t result = mGraphicBufferProducer->dequeueBuffer(&buf, &fence,
            reqWidth, reqHeight, reqFormat, reqUsage,
            enableFrameTimestamps ? &frameTimestamps : nullptr);
   

    if ((result & IGraphicBufferProducer::BUFFER_NEEDS_REALLOCATION) || gbuf == nullptr) {
        //映射缓冲区
        result = mGraphicBufferProducer->requestBuffer(buf, &gbuf);
        
    }

    return OK;
}

dequeueBuffer关键的步骤只有两步

  1. 通过GraphicBufferProducer的dequeueBuffer获取缓冲区
  2. 通过GraphicBufferProducer的requestBuffer映射缓冲区

dequeueBuffer已经讲过了,就不重复讲了,我们看一下requestBuffer,当我们获取了缓冲区内存后,图像生产者和图像消费者都需要将缓冲区映射到他们的进程中才能使用。

/frameworks/native/libs/gui/IGraphicBufferProducer.cpp

virtual status_t requestBuffer(int bufferIdx, sp<GraphicBuffer>* buf) {
    Parcel data, reply;
    data.writeInterfaceToken(IGraphicBufferProducer::getInterfaceDescriptor());
    data.writeInt32(bufferIdx);
    status_t result =remote()->transact(REQUEST_BUFFER, data, &reply);
    if (result != NO_ERROR) {
        return result;
    }
    bool nonNull = reply.readInt32();
    if (nonNull) {
        //在用户进程创建空的GraphicBuffer
        *buf = new GraphicBuffer();
        //通过共享内存的方式,映射GraphicBufferProducer中的GraphicBuffer
        result = reply.read(**buf);
        if(result != NO_ERROR) {
            (*buf).clear();
            return result;
        }
    }
    result = reply.readInt32();
    return result;
}

可以看到,requestBuffer函数中创建了GraphicBuffer对象,这个对应位于用户进程,并且没有申请内存。接着调用了Parcel的read的方法,将GraphicBufferProducer中的GraphicBuffer映射过来。共享内存是最高效也是传输数据量最大的IPC通信机制,关于Android IPC通信的更多知识,可以看看我的这篇文章《深入理解Android进程间通信机制》

提交图像缓冲区

接着看软绘是如何提交缓冲区,它的实现在nativeUnlockCanvasAndPost这个native函数中

/frameworks/base/core/jni/android_view_Surface.cpp

static void nativeUnlockCanvasAndPost(JNIEnv* env, jclass clazz,
        jlong nativeObject, jobject canvasObj) {
    sp<Surface> surface(reinterpret_cast<Surface *>(nativeObject));
    if (!isSurfaceValid(surface)) {
        return;
    }

    // detach the canvas from the surface
    Canvas* nativeCanvas = GraphicsJNI::getNativeCanvas(env, canvasObj);
    nativeCanvas->setBitmap(SkBitmap());

    // unlock surface
    status_t err = surface->unlockAndPost();
    if (err < 0) {
        doThrowIAE(env);
    }
}

nativeUnlockCanvasAndPost调用了surface的unlockAndPost函数

/frameworks/native/libs/gui/Surface.cpp

status_t Surface::unlockAndPost()
{
    if (mLockedBuffer == 0) {
        return INVALID_OPERATION;
    }

    int fd = -1;
    status_t err = mLockedBuffer->unlockAsync(&fd);

    //图像缓冲区的入队操作
    err = queueBuffer(mLockedBuffer.get(), fd);
   

    mPostedBuffer = mLockedBuffer;
    mLockedBuffer = 0;
    return err;
}

在着看queueBuffer的实现

/frameworks/native/libs/gui/Surface.cpp

int Surface::queueBuffer(android_native_buffer_t* buffer, int fenceFd) {
   
    Mutex::Autolock lock(mMutex);
    int64_t timestamp;
    bool isAutoTimestamp = false;

    //1,获取Buffer在slots中的index
    int i = getSlotFromBufferLocked(buffer);
    if (i < 0) {
        if (fenceFd >= 0) {
            close(fenceFd);
        }
        return i;
    }
    if (mSharedBufferSlot == i && mSharedBufferHasBeenQueued) {
        if (fenceFd >= 0) {
            close(fenceFd);
        }
        return OK;
    }

	……


    //2,调用GraphicBufferProducer的queueBuffer函数,进行入队操作
    status_t err = mGraphicBufferProducer->queueBuffer(i, input, &output);
    mLastQueueDuration = systemTime() - now;
    if (err != OK)  {
        ALOGE("queueBuffer: error queuing buffer to SurfaceTexture, %d", err);
    }

    ……

    return err;
}

可以看到surface.unlockCanvasAndPost最终也是通过queueBuffer函数来提交缓冲区的。

硬件加速

了解了软件绘制如何使用缓冲区,我们在看看硬件加速中是如何使用缓冲区的,硬件加速的绘制入口在CanvasContext的draw函数中,对硬件绘制流程不熟的,可以看图像生产者中使用OpenGL进行硬件绘制的部分。

/frameworks/base/libs/hwui/renderthread/CanvasContext.cpp

void CanvasContext::draw() {
    SkRect dirty;
    mDamageAccumulator.finish(&dirty);

    mCurrentFrameInfo->markIssueDrawCommandsStart();
	//1,获取缓冲区
    Frame frame = mRenderPipeline->getFrame();

    SkRect windowDirty = computeDirtyRect(frame, &dirty);
	//2,调用OpenGL的draw函数进行绘制
    bool drew = mRenderPipeline->draw(frame, windowDirty, dirty, mLightGeometry, &mLayerUpdateQueue,
            mContentDrawBounds, mOpaque, mLightInfo, mRenderNodes, &(profiler()));

    waitOnFences();

    bool requireSwap = false;
    //3,提交缓冲区
    bool didSwap = mRenderPipeline->swapBuffers(frame, drew, windowDirty, mCurrentFrameInfo,
            &requireSwap);

    mIsDirty = false;

	//……

}

硬件加速和软件绘制的流程都是分为三步

  1. 获取缓冲区
  2. 绘制
  3. 提交缓冲区

获取图像缓冲区

我们先看第一步:获取缓冲区。它的是在RenderPipeline的getFrame函数中,这里的RenderPipeline以OpenGLPipeline为例讲解。

/frameworks/base/libs/hwui/renderthread/OpenGLPipeline.cpp

Frame OpenGLPipeline::getFrame() {
    return mEglManager.beginFrame(mEglSurface);
}

getFrame调用了EglManager的beginFrame函数

Frame EglManager::beginFrame(EGLSurface surface) {    
    makeCurrent(surface);
    Frame frame;
    frame.mSurface = surface;
    eglQuerySurface(mEglDisplay, surface, EGL_WIDTH, &frame.mWidth);
    eglQuerySurface(mEglDisplay, surface, EGL_HEIGHT, &frame.mHeight);
    frame.mBufferAge = queryBufferAge(surface);
    eglBeginFrame(mEglDisplay, surface);
    return frame;
}

beginFrame中的流程有这几步

  1. 调用makeCurrent获取缓冲区
  2. 调用eglQuerySurface获取缓冲区的宽高
  3. 调用eglBeginFrame进行参数合法性校验

关键函数是makeCurrent,直接看它的实现

/frameworks/base/libs/hwui/renderthread/EglManager.cpp

bool EglManager::makeCurrent(EGLSurface surface, EGLint* errOut) {
    
    if (!eglMakeCurrent(mEglDisplay, surface, surface, mEglContext)) {
        if (errOut) {
            *errOut = eglGetError();          
        } else {
           
        }
    }
    mCurrentSurface = surface;
    if (Properties::disableVsync) {
        eglSwapInterval(mEglDisplay, 0);
    }
    return true;
}

makeCurrent实际调用的是eglMakeCurrent,这部分的实现在EGL里面

/frameworks/native/opengl/libs/EGL/egl.cpp

EGLBoolean eglMakeCurrent(  EGLDisplay dpy, EGLSurface draw,
                            EGLSurface read, EGLContext ctx)
{
    
    //draw和read和合法校验
    ……

    if (ctx == EGL_NO_CONTEXT) {      
        current_ctx = (EGLContext)getGlThreadSpecific();
    } else {
        egl_context_t* c = egl_context_t::context(ctx);
        //将EGLSurface转换成egl_surface_t
        egl_surface_t* d = (egl_surface_t*)draw;
        egl_surface_t* r = (egl_surface_t*)read;

    }

    ogles_context_t* gl = (ogles_context_t*)ctx;
    if (makeCurrent(gl) == 0) {
        if (ctx) {
            
            egl_context_t* c = egl_context_t::context(ctx);
            egl_surface_t* d = (egl_surface_t*)draw;
            egl_surface_t* r = (egl_surface_t*)read;
            
            if (c->draw) {
                egl_surface_t* s = reinterpret_cast<egl_surface_t*>(c->draw);
                s->disconnect();
                s->ctx = EGL_NO_CONTEXT;
                if (s->zombie)
                    delete s;
            }
            if (c->read) {
                
            }
            
            c->draw = draw;
            c->read = read;

            if (c->flags & egl_context_t::NEVER_CURRENT) {
                c->flags &= ~egl_context_t::NEVER_CURRENT;
                GLint w = 0;
                GLint h = 0;
                if (draw) {
                    w = d->getWidth();
                    h = d->getHeight();
                }
                ogles_surfaceport(gl, 0, 0);
                ogles_viewport(gl, 0, 0, w, h);
                ogles_scissor(gl, 0, 0, w, h);
            }
            if (d) {
                //1,调用egl_surface_t的connect函数创建缓冲区
                if (d->connect() == EGL_FALSE) {
                    return EGL_FALSE;
                }
                d->ctx = ctx;                
                d->bindDrawSurface(gl);
            }
            if (r) {               
                r->ctx = ctx;
                r->bindReadSurface(gl);
            }
        } else {
            //异常处理
            ……
        }
        return EGL_TRUE;
    }
    return setError(EGL_BAD_ACCESS, EGL_FALSE);
}

eglMakeCurrent函数中我们只关心一件事情:执行d->connect()创建缓冲区

/frameworks/native/opengl/libs/EGL/egl.cpp

EGLBoolean egl_window_surface_v2_t::connect() 
{
    //获取图像缓冲区
    if (nativeWindow->dequeueBuffer(nativeWindow, &buffer,
            &fenceFd) != NO_ERROR) {
        return setError(EGL_BAD_ALLOC, EGL_FALSE);
    }
   	……
    // 绑定缓冲区
    if (lock(buffer, GRALLOC_USAGE_SW_READ_OFTEN | 
            GRALLOC_USAGE_SW_WRITE_OFTEN, &bits) != NO_ERROR) {
        return setError(EGL_BAD_ACCESS, EGL_FALSE);
    }
    return EGL_TRUE;
}

可以看到connect函数中最终依然是调用了nativeWindow的dequeueBuffer函数,前面提到过Surface继承自nativeWindow,所以这里实际就是调用了Surface的dequeueBuffer函数,这个函数的实现在前面详细讲了,这儿就不再讲了。但是这里和软件绘制有点区别的地方在于缓冲区的绑定,软件绘制中是调用的requestBuffer进行共享内存的绑定,这里是调用lock函数,它的实现如下

/frameworks/native/opengl/libs/EGL/egl.cpp

status_t egl_window_surface_v2_t::lock(
        ANativeWindowBuffer* buf, int usage, void** vaddr)
{
    auto& mapper = GraphicBufferMapper::get();
    return mapper.lock(buf->handle, usage,
            android::Rect(buf->width, buf->height), vaddr);
}

lock函数里面其实调用GraphicBufferMapper进行了一个内存绑定的操作,GraphicBufferMapper属于Galloc模块,它是单例的,在requesetBuffer函数中进行内存共享绑定操作时,也会用到这个GraphicBufferMapper,他们是同一个GraphicBufferMapper。

提交图像缓冲区

我们接着看硬件加速是如何提交缓冲区的,它的实现在swapBuffers函数中。

/frameworks/native/opengl/libagl/egl.cpp

EGLBoolean egl_window_surface_v2_t::swapBuffers()
{
    ……
    
    unlock(buffer);
    previousBuffer = buffer;
    //1,提交缓冲区
    nativeWindow->queueBuffer(nativeWindow, buffer, -1);
    buffer = 0;

    int fenceFd = -1;
    //2,获取新的缓冲区
    if (nativeWindow->dequeueBuffer(nativeWindow, &buffer, &fenceFd) == NO_ERROR) {
        ……
    } else {
        return setError(EGL_BAD_CURRENT_SURFACE, EGL_FALSE);
    }

    return EGL_TRUE;
}

swapBuffers中的逻辑很清晰,主要就两步操作:queueBuffer提交缓冲区,dequeueBuffer获取新的缓冲区。

关于缓冲区的使用实例这里就讲完了,可以看到,不管是软件绘制,还是硬件加速,使用的方式都是一致的,都为下面几步。

  1. dequeueBuffer获取缓冲区
  2. 进行缓冲区内存绑定操作
  3. 使用缓冲区存储图像
  4. queueBuffer提交缓冲区

这些步骤是通用的流程,不仅适用在软件绘制或者硬件绘制,还包括webview,或者是Flutter,或者我们自己开发渲染引擎,都需要使用同样的步骤。

下面接着看图像消费使用缓冲区的实例。

图像消费者使用缓冲区

图像消费者这一篇中讲到,SurfaceFlinger会在onMessageReceived收到VSync的通知

/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

void SurfaceFlinger::onMessageReceived(int32_t what) {
    ATRACE_CALL();
    switch (what) {
        case MessageQueue::INVALIDATE: {
            bool frameMissed = !mHadClientComposition &&
                    mPreviousPresentFence != Fence::NO_FENCE &&
                    (mPreviousPresentFence->getSignalTime() ==
                            Fence::SIGNAL_TIME_PENDING);
            ATRACE_INT("FrameMissed", static_cast<int>(frameMissed));
            if (mPropagateBackpressure && frameMissed) {
                ALOGD("Backpressure trigger, skipping transaction & refresh!");
                //如果掉帧则请求下一次VSync,跳过这一次请求
                signalLayerUpdate();
                break;
            }

            //更新VR模式的Flinger
            updateVrFlinger();

            bool refreshNeeded = handleMessageTransaction();
            //获取缓冲区
            refreshNeeded |= handleMessageInvalidate();
            refreshNeeded |= mRepaintEverything;
            if (refreshNeeded) {
                //判断是否要做刷新
                signalRefresh();
            }
            break;
        }
        case MessageQueue::REFRESH: {
            handleMessageRefresh();
            break;
        }
    }
}

INVALIDATE的流程如下:

  1. 判断是否掉帧
  2. 通过handleMessageTransaction函数来判断是否要处理这一次的Vsync
  3. 调用handleMessageInvalidate函数获取Layer中的缓冲区
  4. 执行signalRefresh函数

这些流程在图像消费者都有详细讲过,我们直接看handleMessageInvalidate函数的实现

/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

bool SurfaceFlinger::handleMessageInvalidate() {
    return handlePageFlip();
}


bool SurfaceFlinger::handlePageFlip()
{

    bool visibleRegions = false;
    bool frameQueued = false;
    bool newDataLatched = false;

   ……

    for (auto& layer : mLayersWithQueuedFrames) {
        //遍历所有Layer,获取缓冲区
        const Region dirty(layer->latchBuffer(visibleRegions, latchTime));
        layer->useSurfaceDamage();
        invalidateLayerStack(layer->getLayerStack(), dirty);
        if (!dirty.isEmpty()) {
            newDataLatched = true;
        }
    }

    mVisibleRegionsDirty |= visibleRegions;

    if (frameQueued && mLayersWithQueuedFrames.empty()) {
        signalLayerUpdate();
    }

    return !mLayersWithQueuedFrames.empty() && newDataLatched;
}

handlePageFlip函数中调用layer->latchBuffer()函数来获取缓冲区

/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

Region Layer::latchBuffer(bool& recomputeVisibleRegions, nsecs_t latchTime)
{
  
  
    status_t updateResult = mSurfaceFlingerConsumer->updateTexImage(&r,
            mFlinger->mPrimaryDispSync, &mAutoRefresh, &queuedBuffer,
            mLastFrameNumberReceived);
   
   	……
        
    return outDirtyRegion;
}

latchBuffer函数非常的长,流程也很多,但我们只关心它获取图像缓冲区的部分:调用SurfaceFlingerConsumer的updateTexImage函数。SurfaceFlingerConsumer是对BufferQueueConsumer的封装,我们看看updateTexImage的实现。

/frameworks/native/services/surfaceflinger/SurfaceFlingerConsumer.cpp

status_t SurfaceFlingerConsumer::updateTexImage(BufferRejecter* rejecter,
        const DispSync& dispSync, bool* autoRefresh, bool* queuedBuffer,
        uint64_t maxFrameNumber)
{
    

    BufferItem item;

  	//获取缓冲区
    err = acquireBufferLocked(&item, computeExpectedPresent(dispSync),
            maxFrameNumber);
   ……

    // 释放前一个缓冲区
#ifdef USE_HWC2
    err = updateAndReleaseLocked(item, &mPendingRelease);
#else
    err = updateAndReleaseLocked(item);
#endif
    if (err != NO_ERROR) {
        return err;
    }

    ……

    return err;
}

updateTexImage函数中主要做了两件事情:

  1. 调用acquireBufferLocked函数获取缓冲区
  2. 调用releaseBufferLocked释放上一个缓冲区

下面分别看一下这两件事情的实现。

acquireBufferLocked函数的实现如下:

/frameworks/native/services/surfaceflinger/SurfaceFlingerConsumer.cpp

status_t SurfaceFlingerConsumer::acquireBufferLocked(BufferItem* item,
        nsecs_t presentWhen, uint64_t maxFrameNumber) {
    status_t result = GLConsumer::acquireBufferLocked(item, presentWhen,
            maxFrameNumber);
    if (result == NO_ERROR) {
        mTransformToDisplayInverse = item->mTransformToDisplayInverse;
        mSurfaceDamage = item->mSurfaceDamage;
    }
    return result;
}

这里调用了GLConsumer的acquireBufferLocked,SurfaceFlingerConsumer是继承自GLConsumer,所以其实是调用父类方法。

/frameworks/native/libs/gui/ConsumerBase.cpp

status_t ConsumerBase::acquireBufferLocked(BufferItem *item,
        nsecs_t presentWhen, uint64_t maxFrameNumber) {
    if (mAbandoned) {
        CB_LOGE("acquireBufferLocked: ConsumerBase is abandoned!");
        return NO_INIT;
    }
	//获取缓冲区
    status_t err = mConsumer->acquireBuffer(item, presentWhen, maxFrameNumber);
    if (err != NO_ERROR) {
        return err;
    }

    
    if (item->mGraphicBuffer != NULL) {
        if (mSlots[item->mSlot].mGraphicBuffer != NULL) {
            freeBufferLocked(item->mSlot);
        }
        mSlots[item->mSlot].mGraphicBuffer = item->mGraphicBuffer;
    }

    mSlots[item->mSlot].mFrameNumber = item->mFrameNumber;
    mSlots[item->mSlot].mFence = item->mFence;


    return OK;
}


releaseBufferLocked函数的实现如下

/frameworks/native/libs/gui/ConsumerBase.cpp

status_t ConsumerBase::releaseBufferLocked(
        int slot, const sp<GraphicBuffer> graphicBuffer,
        EGLDisplay display, EGLSyncKHR eglFence) {
    if (mAbandoned) {
        CB_LOGE("releaseBufferLocked: ConsumerBase is abandoned!");
        return NO_INIT;
    }
    //释放缓冲区
    status_t err = mConsumer->releaseBuffer(slot, mSlots[slot].mFrameNumber,
            display, eglFence, mSlots[slot].mFence);
    if (err == IGraphicBufferConsumer::STALE_BUFFER_SLOT) {
        freeBufferLocked(slot);
    }

    mPrevFinalReleaseFence = mSlots[slot].mFence;
    mSlots[slot].mFence = Fence::NO_FENCE;

    return err;
}

可以看到acquireBufferLockedreleaseBufferLocked最终调用的依然是BufferQueueConsumer的acuqireBuffer和releaseBuffer函数

结尾

到这里,图像显示原理的知识就都讲完了。技术飞速发展,Android中不断有新的图像显示系统出现,如RN,小程序,以及现在很火的Flutter等等,当面对层出不穷的新技术时,我们只有掌握底层的本质原理和组成,才能快速的理解新的技术。比如我在学习Webview的时候,就将我对Android显示一个界面的基本组成(如界面的布局,测量,绘制,图像的生产者,图像的缓冲区,图像的消费)的了解迁移到对Webview的学习上,去寻找Webview在这些基本组成部分的异同点,这样很快就能熟悉和上手Webview的开发了,所以,在最后希望大家能更有耐心的深入到底层原理的学习中,而不要潜伏于技术的表面。