Media playback encompasses a large swatch of technologies, so by necessity this
will provide only a brief outline. Inside this directory you'll find components
for media demuxing, software and hardware video decode, audio output, as well as
audio and video rendering.
Specifically under the playback heading, media/ contains the implementations of
components required for HTML media elements and extensions:
* [HTML5 Audio & Video](https://dev.w3.org/html5/spec-author-view/video.html)
* [Media Source Extensions](https://www.w3.org/TR/media-source/)
* [Encrypted Media Extensions](https://www.w3.org/TR/encrypted-media/)
As a case study we'll consider the playback of a video through the `<video>` tag.
`<video>` (and `<audio>`) starts in `blink::HTMLMediaElement` in
third_party/WebKit/ and reaches media/blink in `media::WebMediaPlayerImpl`
after a brief hop through `content::MediaFactory`. Each
`blink::HTMLMediaElement` owns a `media::WebMediaPlayerImpl` for handling
things like play, pause, seeks, and volume changes (among other things).
`media::WebMediaPlayerImpl` handles or delegates media loading over the network
as well as demuxer and pipeline initialization. `media::WebMediaPlayerImpl`
owns a `media::PipelineController` which manages the coordination of a
`media::DataSource`, `media::Demuxer`, and `media::Renderer` during playback.
During a normal playback, the `media::Demuxer` owned by WebMediaPlayerImpl may
be either `media::FFmpegDemuxer` or `media::ChunkDemuxer`. The ffmpeg variant
is used for standard src= playback where WebMediaPlayerImpl is responsible for
loading bytes over the network. `media::ChunkDemuxer` is used with Media Source
Extensions (MSE), where JavaScript code provides the muxed bytes.
The media::Renderer is typically `media::RendererImpl` which owns and
coordinates `media::AudioRenderer` and `media::VideoRenderer` instances. Each
of these in turn own a set of `media::AudioDecoder` and `media::VideoDecoder`
implementations. Each issues an async read to a `media::DemuxerStream` exposed
by the `media::Demuxer` which is routed to the right decoder by
`media::DecoderStream`. Decoding is again async, so decoded frames are
delivered at some later time to each renderer.
The media/ library contains hardware decoder implementations in media/gpu for
all supported Chromium platforms, as well as software decoding implementations
in media/filters backed by FFmpeg and libvpx. Decoders are attempted in the
order provided via the `media::RendererFactory`; the first one which reports
success will be used for playback (typically the hardware decoder for video).
Each renderer manages timing and rendering of audio and video via the event-
driven `media::AudioRendererSink` and `media::VideoRendererSink` interfaces
respectively. These interfaces both accept a callback that they will issue
periodically when new audio or video frames are required.
On the audio side, again in the normal case, the `media::AudioRendererSink` is
driven via a `base::SyncSocket` and shared memory segment owned by the browser
process. This socket is ticked periodically by a platform level implementation
of `media::AudioOutputStream` within media/audio.
On the video side, the `media::VideoRendererSink` is driven by async callbacks
issued by the compositor to `media::VideoFrameCompositor`. The
`media::VideoRenderer` will talk to the `media::AudioRenderer` through a
`media::TimeSource` for coordinating audio and video sync.
With that we've covered the basic flow of a typical playback. When debugging
issues, it's helpful to review the internal logs at chrome://media-internals.
The internals page contains information about active
`media::WebMediaPlayerImpl`, `media::AudioInputController`,
`media::AudioOutputController`, and `media::AudioOutputStream` instances.
