ExoPlayer 漫谈之Extractor

2,244 阅读3分钟

一个正常的url设置到播放器中。播放器一般会经历如下的流程:

  • 请求url
  • 解封装
  • 分离音频、视频、字幕轨道
  • 起一个线程解码音频,同时起一个线程解码视频,然后做好音视频同步
  • 音视频使用AudioTrack或者OpenSL ES播放,视频使用TextureView或者SurfaceView渲染

上一篇文章已经分析了ExoPlayer是如何请求url的,我们请求得到了一定的数据,就要对源数据进行解封装。解封装的前提要知道视频是什么封装格式的? 探知视频封装格式的过程就是Extractor。本文主要分析ExoPlayer的Extractor模块。

1. ExoPlayer支持的封装格式

主要看DefaultExtractorsFactory.java类中的createExtractors方法:

  public synchronized Extractor[] createExtractors() {
    Extractor[] extractors = new Extractor[14];
    extractors[0] = new MatroskaExtractor(matroskaFlags);
    extractors[1] = new FragmentedMp4Extractor(fragmentedMp4Flags);
    extractors[2] = new Mp4Extractor(mp4Flags);
    extractors[3] =
        new Mp3Extractor(
            mp3Flags
                | (constantBitrateSeekingEnabled
                    ? Mp3Extractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING
                    : 0));
    extractors[4] =
        new AdtsExtractor(
            adtsFlags
                | (constantBitrateSeekingEnabled
                    ? AdtsExtractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING
                    : 0));
    extractors[5] = new Ac3Extractor();
    extractors[6] = new TsExtractor(tsMode, tsFlags);
    extractors[7] = new FlvExtractor();
    extractors[8] = new OggExtractor();
    extractors[9] = new PsExtractor();
    extractors[10] = new WavExtractor();
    extractors[11] =
        new AmrExtractor(
            amrFlags
                | (constantBitrateSeekingEnabled
                    ? AmrExtractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING
                    : 0));
    extractors[12] = new Ac4Extractor();
    if (FLAC_EXTENSION_EXTRACTOR_CONSTRUCTOR != null) {
      try {
        extractors[13] = FLAC_EXTENSION_EXTRACTOR_CONSTRUCTOR.newInstance();
      } catch (Exception e) {
        // Should never happen.
        throw new IllegalStateException("Unexpected error creating FLAC extractor", e);
      }
    } else {
      extractors[13] = new FlacExtractor();
    }
    return extractors;
  }
  • MatroskaExtractor:Extracts data from the Matroska and WebM container formats
  • FragmentedMp4Extractor:Extracts data from the FMP4 container format
  • Mp4Extractor:Extracts data from the MP4 container format
  • Mp3Extractor:Extracts data from the MP3 container format
  • AdtsExtractor:Extracts data from AAC bit streams with ADTS framing
  • Ac3Extractor:Extracts data from (E-)AC-3 bitstreams
  • TsExtractor:Extracts data from the MPEG-2 TS container format
  • FlvExtractor:Extracts data from the FLV container format
  • OggExtractor:Extracts data from the Ogg container format
  • PsExtractor:Extracts data from the MPEG-2 PS container format
  • WavExtractor:Extracts data from WAV byte streams
  • AmrExtractor:Extracts data from the AMR containers format (either AMR or AMR-WB)
  • Ac4Extractor:Extracts data from AC-4 bitstreams
  • FlacExtractor:Extracts data from FLAC container format

2.Extractor过程

Extractor的接口有如下方法:

public interface Extractor {
  int RESULT_CONTINUE = 0;
  int RESULT_SEEK = 1;
  int RESULT_END_OF_INPUT = C.RESULT_END_OF_INPUT;

  boolean sniff(ExtractorInput input) throws IOException, InterruptedException;

  void init(ExtractorOutput output);

  int read(ExtractorInput input, PositionHolder seekPosition)
      throws IOException, InterruptedException;

  void seek(long position, long timeUs);

  void release();
}
  • sniff函数:嗅探视频封装格式的函数,这个函数主要读取视频文件中的二进制数据,和封装格式的协议比较。
  • init函数:已经知道了封装格式,开始初始化具体的封装Extractor,因为接下来要准备读取视频中的各个轨道数据了。
  • read函数:读取具体的视频数据。

探测视频封装格式的地方在ProgressiveMediaPeriod.java中的ExtractorHolder内部类。

    public Extractor selectExtractor(ExtractorInput input, ExtractorOutput output, Uri uri)
        throws IOException, InterruptedException {
      if (extractor != null) {
        return extractor;
      }
      if (extractors.length == 1) {
        this.extractor = extractors[0];
      } else {
        for (Extractor extractor : extractors) {
          try {
            if (extractor.sniff(input)) {
              this.extractor = extractor;
              break;
            }
          } catch (EOFException e) {
            // Do nothing.
          } finally {
            input.resetPeekPosition();
          }
        }
        if (extractor == null) {
          throw new UnrecognizedInputFormatException(
              "None of the available extractors ("
                  + Util.getCommaDelimitedSimpleClassNames(extractors)
                  + ") could read the stream.",
              uri);
        }
      }
      extractor.init(output);
      return extractor;
    }

遍历extractors数组,发现符合特定格式的规则之后,就认为它是这种封装格式。

以Mp4Extractor为例分析一下ExoPlayer内部如何识别特定的封装格式。

  • extractor.sniff 调用到 Mp4Extractor.sniff中
  • 然后调用到Sniffer.sniffUnfragmented中,Sniffer.sniffInternal

在Sniffer.sniffInternal中有探测是否是MP4视频的代码。 这儿有一个问题:MP4视频是有排列顺序的,主要是moov和mdat,moov表示的视频相关的属性集合,mdat是具体的音视频数据。必须要先读到moov数据才能正确解析mdat数据。否则无法正常播放MP4视频 正常情况下moov在mdat之前。加入moov就在mdat之后,播放器很容易看到解析的过程。 在Sniffer.sniffInternal函数中: 这就是双IO检测机制。对解决MP4视频的moov位置很有帮组。

获取了正确的Extractor,可以开始读取视频数据了,现在我们回到ProgressiveMediaPeriod.load方法:

          while (result == Extractor.RESULT_CONTINUE && !loadCanceled) {
            loadCondition.block();
            result = extractor.read(input, positionHolder);
            if (input.getPosition() > position + continueLoadingCheckIntervalBytes) {
              position = input.getPosition();
              loadCondition.close();
              handler.post(onContinueLoadingRequestedRunnable);
            }
          }
    onContinueLoadingRequestedRunnable =
        () -> {
          if (!released) {
            Assertions.checkNotNull(callback)
                .onContinueLoadingRequested(ProgressiveMediaPeriod.this);
          }
        };

加载数据都放在子线程中完成。

这个extractor.read操作,对应Mp4Extractor.read

  public int read(ExtractorInput input, PositionHolder seekPosition)
      throws IOException, InterruptedException {
    while (true) {
      switch (parserState) {
        case STATE_READING_ATOM_HEADER:
          if (!readAtomHeader(input)) {
            return RESULT_END_OF_INPUT;
          }
          break;
        case STATE_READING_ATOM_PAYLOAD:
          if (readAtomPayload(input, seekPosition)) {
            return RESULT_SEEK;
          }
          break;
        case STATE_READING_SAMPLE:
          return readSample(input, seekPosition);
        default:
          throw new IllegalStateException();
      }
    }
  }
  • STATE_READING_ATOM_HEADER 开始读atom header信息
  • STATE_READING_ATOM_PAYLOAD 如果atom没有读完全,播放器会将parserState设置为STATE_READING_ATOM_PAYLOAD
  • STATE_READING_SAMPLE atom读完,开始读媒体文件原始数据

readAtomHeader : 读取MP4文件的头部

readAtomPayload:atom没有读完,继续请求匹配atom header

readSample:开始读取moov的解析出的track信息