本文基于Android平台简单梳理了Chromium在打开一个视频资源时背后所发生的逻辑与源码调用流程
1. Video标签初始化视频源
当浏览器打开一个视频资源链接时，会默认为视频资源创建video标签，帮将视频源链接设置给video标签的src属性，当video标签解析到视频源链接后，会触发以下流程进行视频资源的加载
################ 消息边界
++++++++++++++++ 区域边界
$$$$$$$$$$$$$$$$ 线程边界
~~~~~~~~~~~~~~~~ 进程边界
---------------- 代码块功能模块分隔
html_media_element.SourceWasAdded
|-->InvokeResourceSelectionAlgorithm
|-->ScheduleNextSourceChild
|-->load_timer_.StartOnShot-------|
|
html_media_element.LoadTimerFired<------|
|-->LoadInternal
|-->SelectMediaResource
|-->LoadNextSourceChild
|-->LoadResource
2. 创建播放器WebMediaPlayer
video标签加载视频资源时会创建相应的播放器用于播放视频，当播放器创建完成后，通过播放器加载视频资源
html_media_element.LoadResource
|-->StartPlayerLoad
|-->frame->Client()->CreateWebMediaPlayer------------|
[frame: local_frame Client:local_frame_client] |
|-->local_frame_client_impl.CreateWebMediaPlayer<----|
|-->core_initializer.CreateWebMediaPlayer--------|
extends |
|-->modules_initializer.CreateWebMediaPlayer<----|
|-->web_local_frame_client.CreateMediaPlayer-----|
extends |
|-->render_frame_impl.CreateMediaPlayer<---------|
|-->media_factory.CreateMediaPlayer [ DeferLoadCB=RenderFrameImpl:DeferMediaLoad ]
|-->web_media_player_builder.Build
|-->new WebMediaPlayerImpl
|-->new PipelineImpl [ create_renderer_cb_=WebMediaPlayerImpl:CreateRenderer ]
|-->web_media_player_impl.Load [通过播放器加载资源]
创建播放器时，也会一并初始化PipelineImpl，传入renderer视频渲染器的构造回调函数
3. 播放器初始化
播放器创建完成后，首先将播放器相关的数据源对象初始化
web_media_player_impl.Load
|-->defer_load_cb_.Run(cb=WebMediaPlayerImpl::DoLoad)------|
media_factory构造WebMediaPlayerImpl时传入 |
render_frame_impl:DoLoad<------------------------------------|
|-->GetContentClient().renderer().DeferMediaLoad
|-->no_state_prefetch_utils.DeferMediaLoad-------|
|
web_media_player_impl.DoLoad<------------------------|
|-->multi_buffer_data_source.Initialize(cb=WebMediaPlayerImpl::MultiBufferDataSourceInitialized)-----|
|
web_media_player_impl.MultiBufferDataSourceInitialized<------------------------------------------------|
|-->DataSourceInitialized
数据源初始化成功后，启动Pipeline流程，web_media_player_impl.StartPipeline
web_media_player_impl.DataSourceInitialized
|-->StartPipeline
|-->post runnable SetOnNewProcessedFrameCallback [ 设置守帧回调 ]
|-->demuxer_manager.CreateDemuxer(cb=WebMediaPlayerImpl::OnDemuxerCreated)--------|
[ 创建视频解封装器 ] |
web_media_player_impl.OnDemuxerCreated<-----------------------------------------------|
|-->pipeline_controller.Start
|-->pipeline_impl.Start
|-->create_renderer_cb_.Run [ WebMediaPlayerImpl:CreateRenderer ]
|-->按需执行初始化任务--------------------------------------------------------|
|-->1.PipelineImpl::RendererWrapper::InitializeDemuxer |
|-->2.PipelineImpl::RendererWrapper::ReportMetadata |
|-->3.PipelineImpl::RendererWrapper::CreateRenderer |
|-->4.PipelineImpl::RendererWrapper::InitializeRenderer |
|-->final callback PipelineImpl::RendererWrapper::CompleteSeek<----------|
|
PipelineImpl::RendererWrapper::CompleteSeek<------------------------|
pipeline_impl.Start首先通过create_renderer_cb_创建媒体渲染器，创建完成后逐步执行代码流程中所述的初始化流程
create_renderer_cb_ 创建媒体渲染器此处就是我们的视频播放器，在Android平台上视频渲染器分为2种:
1. 主场景为codec解码器方式，即Render进程负责调度视频的解封装以及声画同步逻辑，而Android平台只提供基本的硬解单元
2. 另外一种则是完全复用Android平台的视频播放器，就是我们平时开发所使用的MediaPlayer
这里我们只详细分析第一种播放器初始化流程，从create_renderer_cb_.Run开始
pipeline_impl.Start
|-->create_renderer_cb_.Run--------------------------------------------------------|
|
web_media_player_impl.CreateRenderer<------------------------------------------------|
|-->renderer_factory_selector_->GetCurrentFactory()->CreateRenderer [ media_factory.CreateRendererFactorySelector ]
|-->renderer_impl_factory.CreateRenderer
|-->new AudioRendererImpl [ CreateAudioDecodersCB=RendererImplFactory:CreateAudioDecoders ]
|-->new VideoRendererImpl [ CreateVideoDecodersCB=RendererImplFactory:CreateVideoDecoders ]
|-->new RendererImpl
create_renderer_cb_创建完成后，逐步执行以下初始化任务
1.InitializeDemuxer
2.ReportMetadata
3.CreateRenderer
4.InitializeRenderer
这里我们先关注最后一步初始化媒体渲染器，首先先创建render进程的视频解码器，这里是因为render是sandbox进程没有权限访问Android平台的解码器资源，因此render进程无法访问到解码器资源需要通过跨进程的方式使用Android平台的硬解单元
PipelineImpl::RendererWrapper::InitializeRenderer
|-->renderer_impl.Initialize
|-->InitializeAudioRenderer
|-->audio_renderer_impl.Initialize(cb=RendererImpl::OnAudioRendererInitializeDone)-------|
|
renderer_impl.OnAudioRendererInitializeDone<---------------------------------------------------|
|-->InitializeVideoRenderer
|-->video_renderer_impl.Initialize(init_cb_=RendererImpl::OnVideoRendererInitializeDone)
|-->decoder_stream.Initialize(init_cb_=VideoRendererImpl::OnVideoDecoderStreamInitialized)
|-->BeginDecoderSelection
|-->decoder_selector.BeginDecoderSelection
|-select_decoder_cb=DecoderStream::OnDecoderSelected
|-output_cb=DecoderStream::OnDecodeOutputReady
|-->SelectDecoderInternal
|-->CreateDecoders
|-->create_decoders_cb_.Run
|
|-->RendererImplFactory:CreateVideoDecoders
|-->mojo_decoder_factory.CreateVideoDecoders
|-->new MojoVideoDecoder [ mojo 视频解码器 ]
render进程硬解对象创建好后开始创建browser进程的硬解对象，browser进程的硬解对象才是真正的硬解单元
decoder_selector.SelectDecoderInternal
|-->CreateDecoders
|-->GetAndInitializeNextDecoder
|-->decoder_stream_traits.InitializeDecoder [ init_cb=DecoderSelector::OnDecoderInitializeDone ]
|-->mojo_video_decoder.Initialize [ init_cb=DecoderStreamTraits::OnDecoderInitialized ]
|-->InitAndBindRemoteDecoder [ cb=MojoVideoDecoder::InitializeRemoteDecoder ]
|-->GpuVideoAcceleratorFactories [ cb=MojoVideoDecoder::OnChannelTokenReady ]--------------------|
[ complete_cb= MojoVideoDecoder::InitializeRemoteDecoder ] |
mojo_video_decoder.OnChannelTokenReady<--------------------------------------------------------------------|
|-->InitAndConstructRemoteDecoder [ complete_cb= MojoVideoDecoder::InitializeRemoteDecoder ]
|-->mojom::VideoDecoder.Construct----------------------------------------------------------------------|
|-->complete_cb.Run |
| 顺序执行 |
|-->mojo_video_decoder.InitializeRemoteDecoder |
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Browser~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
mojo_video_decoder_service.Construct<----------------------------------------------------------------------|
|-->mojo_media_client.CreateVideoDecoder
| extends
|-->gpu_mojo_media_client.CreateVideoDecoder
|-->CreatePlatformVideoDecoder
| extends
|-->gpu_mojo_media_client_android.CreatePlatformVideoDecoder
|-codec_allocator.GetInstance
|-->new CodecAllocator [ factory_cb=MediaCodecBridgeImpl::CreateVideoDecoder ]
|-media_codec_video_decoder.Create
|-->new MediaCodecVideoDecoder
|-->android_video_surface_chooser_impl.SetClientCallbacks [ cb=MediaCodecVideoDecoder::OnSurfaceChosen ]
|-->new AsyncDestroyVideoDecoder(MediaCodecVideoDecoder)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Renderer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mojo_video_decoder.InitializeRemoteDecoder
|-->mojom::VideoDecoder.Initialize [ init_cb_=MojoVideoDecoder::OnInitializeDone ]-----------------------|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Browser~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
mojo_video_decoder_service.Initialize [ cb=MojoVideoDecoder::OnInitializeDone ]<---------------------------|
|-->media_codec_video_decoder.Initialize [ init_cb=MojoVideoDecoderService::OnDecoderInitialized ]
[ output_cb=MojoVideoDecoderService::OnDecoderOutput ]
|-->set decoder_config
|-->init_cb.Run
|
|-->mojo_video_decoder_service.OnDecoderInitialized
|-->init_cb_.Run-----------------------------------------------------------------------------------|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Renderer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
mojo_video_decoder.OnInitializeDone<---------------------------------------------------------------------|
4. 解码器初始化
当mojo_video_decoder_service完成browser、render解码器初步的创建后，会触发一系列的回调，最终到PipelineImpl::RendererWrapper::CompleteSeek，具体调用流程如下
mojo_video_decoder.OnInitializeDone
|-->init_cb_.Run [ DecoderStreamTraits::OnDecoderInitialized ]------------------------|
(decoder_stream_traits.InitializeDecoder bind the callback) |
decoder_stream_traits.OnDecoderInitialized<---------------------------------------------|
|-->cb.Run [ DecoderSelector::OnDecoderInitializeDone ]-------------------------------|
(decoder_selector.GetAndInitializeNextDecoder bind the callback) |
decoder_selector.OnDecoderInitializeDone<-----------------------------------------------|
|-->RunSelectDecoderCB
|-->select_decoder_cb_.Run----------------------------------------------------------|
(decoder_stream.BeginDecoderSelection bind the callback) |
decoder_stream.OnDecoderSelected<-------------------------------------------------------|
|-->init_cb_.Run----------------------------------------------------------------------|
(video_renderer_impl.Initialize bind the callback) |
video_renderer_impl.OnVideoDecoderStreamInitialized<------------------------------------|
|-->FinishInitialization
|-->init_cb_.Run--------------------------------------------------------------------|
(renderer_impl.InitializeVideoRenderer bind the callback) |
renderer_impl.OnVideoRendererInitializeDone<--------------------------------------------|
|-->FinishInitialization
|-->init_cb_.Run--------------------------------------------------------------------|
(PipelineImpl::RendererWrapper::InitializeRenderer bind done_cb) |
(doen_cb=SerialRunner.RunNextInSeries) |
serial_runner.RunNextInSeries<----------------------------------------------------------|
|-->bound_fns is empty
|-->done_cb_.Run ---------------------------------------------------------------------|
| PipelineImpl::RendererWrapper::Start bind the callback |
PipelineImpl::RendererWrapper::CompleteSeek<--------------------------------------------|
PipelineImpl::RendererWrapper::CompleteSeek方法里通知到PipelineImpl解码器对象已准备好，此时PipelineImpl会通知视频渲染对象尝试从数据源中读取数据，读取数据的方式则是从视频解封装器读取相应的音频流和视频流
视频渲染对象将读取到的数据发送到browser端的解码器对象进行解码，由于browser端的解码器对象刚准备好，还没有进行解码器的初始化，所以在收到第一次数据buffer时会进行解码器对象的prepare操作，具体调用流程如下
PipelineImpl::RendererWrapper::CompleteSeek
|-->renderer_impl.StartPlayingFrom
|-->video_renderer_impl.StartPlayingFrom
|-->AttemptRead_Locked
|-->decoder_stream->Read [ read_cb_=VideoRendererImpl::FrameReady ]
|-->ReadFromDemuxerStream
|-->demuxer_stream.Read [ cb=DecoderStream:OnBuffersReady ]--------|
|
decoder_stream.OnBuffersReady<-------------------------------------------------|
|-->Decode
|-->mojo_video_decoder.Decode [ decode_cb=DecoderStream::OnDecodeDone ]
|-->mojo::VideoDecoder.Decode [ cb=MojoVideoDecoder::OnDecodeDone ]----------|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Browser~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
mojo_video_decoder_service.Decode [ callback=MojoVideoDecoder::OnDecodeDone ]<-----|
|-->mojo_decoder_buffer_reader.ReadDecodeBuffer [ callback=MojoVideoDecoderService::OnReaderRead ]------|
[ bind callback MojoVideoDecoder::OnDecodeDone] |
mojo_video_decoder_service.OnReaderRead<------------------------------------------------------------------|
|-->media_codec_video_decoder.Decode [ callback=MojoVideoDecoderService::OnDecoderDecoded ]
[ bind callback MojoVideoDecoder::OnDecodeDone]
|-->pending_decodes_ 先记录读取到的数据Buffer
|-->StartLazyInit
MediaCodecVideoDecoder.StartLazyInit开始初始化Android平台的硬件解码单元
media_codec_video_decoder.StartLazyInit
|-->video_frame_factory.Initialize [ callback=MediaCodecVideoDecoder::OnVideoFrameFactoryInitialized ]-------|
[ 生成硬件解码器的texture，用于构建surface用于解码数据的输出 ] |
media_codec_video_decoder.OnVideoFrameFactoryInitialized<------------------------------------------------------|
|-->request_overlay_info_cb_.Run [ callback=MediaCodecVideoDecoder::OnOverlayInfoChanged ]
| mojo_video_decoder_service.Construct set the request_overlay_info_cb
| MojoVideoDecoderService::OnDecoderRequestedOverlayInfo
|-->mojo_video_decoder_service.OnDecoderRequestedOverlayInfo [ callback=MediaCodecVideoDecoder::OnOverlayInfoChanged ]
|-->provide_overlay_info_cb_ = callback
|-->mojom::VideoDecoderClient.RequestOverlayInfo----------------------------|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Render~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
mojo_video_decoder.RequestOverlayInfo<------------------------------------------|
|-->request_overlay_info_cb_.Run [ callback=MojoVideoDecoder::OnOverlayInfoChanged ]
| web_media_player_impl.CreateRenderer set the request_overlay_info_cb
| WebMediaPlayerImpl::OnOverlayInfoRequested
|-->web_media_player_impl.OnOverlayInfoRequested [ callback=MojoVideoDecoder::OnOverlayInfoChanged ]
|-->provide_overlay_info_cb_ = callback
|-->MaybeSendOverlayInfoToDecoder
|-->provide_overlay_info_cb_.Run----------|
|
mojo_video_decoder.OnOverlayInfoChanged<--------|
|-->mojo::VideoDecoder.OnOverlayInfoChanged--------------------------------|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Browser~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
mojo_video_decoder_service.OnOverlayInfoChanged<-----------------------------|
|-->provide_overlay_info_cb_.Run
| MediaCodecVideoDecoder::OnOverlayInfoChanged
|-->media_codec_video_decoder.OnOverlayInfoChanged
|-->surface_chooser_helper.UpdateChooserState
|-->android_video_surface_chooser_impl.UpdateState
|-->Choose
|-->SwitchToTextureOwner
|-->use_texture_owner_cb_.Run-------------------------------------|
[ MediaCodecVideoDecoder constructor set the callback ] |
[ MediaCodecVideoDecoder::OnSurfaceChosen ] |
|
media_codec_video_decoder.OnSurfaceChosen<------------------------------------|
|-->CreateCodec
|-->codec_allocator.CreateMediaCodecAsync [ callback=MediaCodecVideoDecoder::OnCodecConfiguredInternal ]
|-->CreateMediaCodecInternal [ return value as CodecAllocator::OnCodecCreated ]
[ bind param callback MediaCodecVideoDecoder::OnCodecConfiguredInternal ]
|-->factory_cb.Run
| MediaCodecBridgeImpl::CreateVideoDecoder
|-->media_codec_bridge_impl.CreateVideoDecoder
|-->Java_MediaCodecBridgeBuilder_createVideoDecoder
|-->new MediaCodecBridgeImpl(java object)
|-->codec_allocator.OnCodecCreated
|-->codec_created_cb.Run
| MediaCodecVideoDecoder::OnCodecConfiguredInternal
|-->media_codec_video_decoder.OnCodecConfiguredInternal
至此Android平台的硬件解码单元已真正创建成功
5. 解码视频帧上屏
硬件解码单元(Android java mediacodec 对象)创建完成后，通过回调通知到media_codec_video_decoder使用此解码单元对从Render接收到的视频流数据进行解码，media_codec_video_decoder.Decode方法触发时会先将数据流记录到pending_decodes_，之后无论是创建解码器还是使用解码器解码数据都是从pending_decodes_提取数据进行解码，解码视频后则将视频帧数据封装为VideoFrame帧数据送到Render的video_compositor进行视频内容的真正上屏，具体流程如下
media_codec_video_decoder.OnCodecConfiguredInternal
|-->OnCodecConfigured
|-->StartTimerOrPumpCodec
|-->PumpCodec
|-->QueueInput [ 向解码器输入视频流 ]
|-->DequeueOutput [ 从解码器或者解码数据 ]
|-->video_frame_factory.CreateVideoFrame [ callback=MediaCodecVideoDecoder::ForwardVideoFrame ]------|
[ 通过解码数据生成视频帧 ] |
media_codec_video_decoder.ForwardVideoFrame<-------------------------------------------------------------------|
|-->output_cb_.Run------------------------------------------------------|
| media_video_decoder_service.Initialize set the callback |
| MojoVideoDecoderService::OnDecoderOutput |
|-->mojo_video_decoder_service.OnDecoderOutput<-------------------------|
|-->mojom::VideoDecoderClient.OnVideoFrameDecoded-----------------------------------------------------|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Render~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
mojo_video_decoder.OnVideoFrameDecoded<-------------------------------------------------------------------|
|-->output_cb_.Run
| decoder_stream.BeginDecoderSelection set to decoder_selector output_cb=DecoderStream::OnDecodeOutputReady
| decoder_selector.output_cb_ = DecoderStream::OnDecodeOutputReady |
| decoder_selector.GetAndInitializeNextDecoder set output_cb to decoder_stream_traits |
| decoder_selector.InitializeDecoder set output_cb to mojo_video_decoder |
| mojo_video_decoder.Initialize record the output_cb_--------------------------------------|
|-->decoder_stream.OnDecodeOutputReady
|-->SatisfyRead
|-->read_cb_.Run
| video_renderer_impl.AttemptRead_Locked bind the callback VideoRendererImpl::FrameReady
|-->video_renderer_impl.FrameReady
|-->PaintFirstFrame
|-->VideoRendererSink.PaintSingleFrame
| extends
|-->video_frame_compositor.PaintSingleFrame
|-->ProcessNewFrame
|-->new_processed_frame_cb_.Run
| web_media_player_impl.StartPipeline set the callback=WebMediaPlayerImpl::OnFirstFrame
|-->web_media_player_impl.OnFirstFrame
|-->通知首帧
|-->client_[cc::VideoFrameProvider::Client].DidReceiveFrame
| extends
|-->web_video_frame_submitter.DidReceiveFrame
| extends
|-->video_frame_submitter.DidReceiveFrame
|-->SubmitSingleFrame
|-->SubmitFrame
|-->viz::mojom::blink::CompositorFrameSink.SubmitCompositorFrame-----------------------|
|-->painted_first_frame_=false [下次 不再触发首帧] |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Browser~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
compositor_frame_sink_impl.SubmitCompositorFrame<--------------------------------------------------------|
|-->SubmitCompositorFrameInternal
|-->compositor_frame_sink_support.MaybeSubmitCompositorFrame
|-->SetNeedsBeginFrame(true)
|-->触发VSync绘制上屏
video_compositor收到browser端传入的视频帧数据时，将该帧数据通过video_frame_submitter将数据通过compositor_frame_sink将数据送到browser端处理数据compositor_frame_sink_impl上，最终通过Android平台的VSync一系列绘制流程，将视频帧数据绘制到设备屏幕上
video_frame_compositor.client_就是上述中video_frame_submitter，它是在PipelineImpl::RendererWrapper::Start 4 步初始化流程中的第 2 步 ReportMetadata 设置的，具体调用流程如下
PipelineImpl::RendererWrapper::ReportMetadata
|-->pipeline_impl::OnMetadata
|-->client_[Pipeline::Client].OnMetadata
| web_media_player_impl.Start set the client, the client is WebMediaPlayerImpl
|-->web_media_player_impl.OnMetadata
|-->ActivateSurfaceLayerForVideo
|-->video_frame_compositor.EnableSubmission
|-->client_=submitter_
| client_[cc::VideoFrameProvider::Client]
| submitter_[WebVideoFrameSubmitter]
| WebVideoFrameSubmitter extends cc::VideoFrameProvider::Client
| video_frame_submitter extends WebVideoFrameSubmitter
至此Chromium打开一个视频资源时内部的源码流程基本已梳理完成，主流程上的分支细节后续系列文章会一一补充完善
