Media Source Extensions踩坑记录

Chak
1,783 阅读4分钟

Media Source Extensions MDN

背景

项目里有这样一个场景，用户输入文字，请求接口获取视频并播放，由于Demo阶段接口返回的是完整视频的MP4链接，导致每次切换视频源都会闪烁，体验极差，想着优化这个视频切换的过程，接近无感。

方案

利用Media Source Extensions这个特性，初始化一个blob链接，分段请求MP4流，喂给这个MSE下的SourceBuffer，每次切换视频则清空当前SourceBuffer，重新填充。

缺陷

MSE兼容性不太好，主要是IOS，目前只有IPadOS Safari 13部分支持，其他浏览器均不支持，所以在IOS目前还没有优化方案，大佬们麻烦支个招。
（2022.08.10 项目中暂时采用重叠video的方式，每次切换视频源待视频加载完毕后切换到对应的DOM，避开加载视频期间的黑屏）

实现

话不多说，记录一下实现过程
主要有三个步骤（如果你的视频资源是FMP4，那么前两步是不需要的）

  • 分段请求MP4资源，利用Range
  • 将MP4资源转化为FMP4（MSE不支持unfragment MP4）
  • 创建MSE，初始化SourceBuffer，将FMP4 append 到SourceBuffer上

分段请求MP4

声明一个Reader类，用于分段请求，提供一个stop和resume方法，因为后续要利用MP4Box来转化FMP4，需要先读取部分MP4配置做初始化，而后resume继续请求后续片段资源，Reader部分代码如下

export enum STATUS {
  READING = 'reading',
  PAUSE = 'pause',
  STOP = 'stop',
  NONE = 'none',
}

export interface ExArrayBuffer extends ArrayBuffer {
  fileStart?: number
}

export interface ReaderOption {
  chunkStart?: number
  chunkSize?: number
  total?: number
  onChunk?: (e: { bytes: ExArrayBuffer; isEof: boolean; total: number }) => void
}

type SuccessCB = (end: number, bytes: ExArrayBuffer, total: number) => void

// TODO 是否做缓存
export class Reader {
  url: string
  status: string
  fetcher: AbortController | null

  constructor(public onChunk: Function, public chunkStart = 100 * 1024, public chunkSize = 300 * 1024, public total = 0) {
    this.status = STATUS.NONE
    this.fetcher = null
    this.url = ''
  }
  // 开始请求
  start(url: string) {
    this.url = url
    this.chunkStart = 0
    this.total = this.chunkStart + this.chunkSize - 1
    this.fetcher = null
    this.status = STATUS.READING
    this.loadchunk()
  }
  // 重启请求
  resume() {
    if (this.status === STATUS.READING) return
    this.status = STATUS.READING
    this.loadchunk()
  }
  // 中止请求
  stop() {
    this.status = STATUS.STOP
    if (!this.fetcher) return
    this.fetcher.abort()
    this.fetcher = null
  }
  // 读取片段
  loadchunk() {
    if (this.fetcher) return
    if (this.chunkStart === this.total - 1) return
    this.fetcher = new AbortController()
    this.read((end: number, bytes: ExArrayBuffer, total: number) => {
      const chunkStart_cache = this.chunkStart
      this.chunkStart = end
      this.total = total
      this.fetcher = null
      bytes.fileStart = chunkStart_cache
      const isEof = end === total - 1
      if (isEof) this.status = STATUS.STOP
      this.onChunk({ bytes, isEof, total })

      if (this.status === STATUS.READING) {
        this.loadchunk()
      }
    })
  }

  // 发起请求
  read(onSuccess: SuccessCB) {
    const end = this.chunkStart + this.chunkSize - 1 > this.total ? this.total : this.chunkStart + this.chunkSize - 1
    if (this.chunkStart === this.total) return
    fetch(this.url, {
      headers: { Range: `bytes=${this.chunkStart}-${end}` },
      signal: this.fetcher?.signal,
    })
      .then(async (res) => {
        const [, , end, total] = res.headers.get('Content-Range')?.match(/(\d+)-(\d+)\/(\d+)/) as RegExpMatchArray
        onSuccess(parseInt(end, 10), await res.arrayBuffer(), parseInt(total, 10))
      })
      .catch((e) => {
        console.log(e)
      })
  }
}

封装MP4Box

MP4Box可以接受分片的MP4资源，组装成Fragment，组装完成后喂给SourceBuffer

import MP4Box from 'mp4box'
import { Reader } from './reader'
import type { ExArrayBuffer } from './reader'

export class FetchFMp4 {
  MF: any
  Mp4Info: any
  reader: Reader
  SourceBuffers: Array<{ codec: string; sourceBuffer: SourceBuffer }>

  constructor(public MS: MediaSource, public el: HTMLVideoElement, public onComplete?: Function) {
    this.SourceBuffers = []
    this.Mp4Info = null
    this.reader = new Reader(this.onChunk.bind(this))
  }

  // 初始化MF 开始拉取文件
  fetchFile(url: string) {
    try {
      this.el.pause()
      this.SourceBuffers.forEach((s) => {
        s?.sourceBuffer.remove(0, this.el.duration)
      })
    } catch (error) {
      console.log('sourceBuffer is empty')
    }
    this.MF = MP4Box.createFile()

    this.MF.onReady = this.onReady.bind(this)

    this.MF.onSegment = this.onSegment.bind(this)

    this.MF.reader = this.reader

    this.MF.reader.start(url)
  }

  // 读取文件配置完毕，开始组装
  onReady(info: any) {
    this.Mp4Info = info
    this.MS.duration = info.duration / info.timescale

    this.initSourceBuffers()
    this.initTracks()
    this.initializeSegmentation()
  }

  // 每组装一个，触发该方法，装进MSE
  onSegment(id: string, ctx: any, buffer: ExArrayBuffer, sampleNum: number, isEnd: boolean) {
    ctx.pending.push({
      id: id,
      buffer: buffer,
      sampleNum: sampleNum,
      isEnd: isEnd,
    })
    this.updateEnd(ctx)
  }

  // 读取每一个chunk完毕，如果还未结束，则继续拉取
  onChunk(chunk: { bytes: ExArrayBuffer; isEof: boolean }) {
    const next = this.MF.appendBuffer(chunk.bytes, chunk.isEof)

    if (chunk.isEof) {
      this.MF.flush()
    } else {
      this.MF.reader.chunkStart = next
    }
  }

  // 初始化sourceBuffer，正常为视频和音频各一个
  initSourceBuffers() {
    if (!this.SourceBuffers.length) {
      this.Mp4Info.tracks.forEach((track: any) => {
        const mime = `video/mp4; codecs="${track.codec}"`
        if (!MediaSource.isTypeSupported(mime)) {
          throw new Error('MSE does not support: ' + mime)
        }
        const sourceBuffer = this.MS.addSourceBuffer(mime)

        this.SourceBuffers.push({
          codec: track.codec,
          sourceBuffer,
        })
      })
    }
  }

  // 初始化轨道
  initTracks() {
    const trackLen = this.Mp4Info.tracks.length
    const shared = {
      loading: false,
      notEndCnt: trackLen,
      pendingInitCnt: trackLen,
      reader: this.MF.reader,
      isMseEnd: false,
    }

    this.Mp4Info.tracks.forEach((track: any) => {
      this.setSegmentOptions(track, shared)
    })
  }

  // 初始化MF segment
  initializeSegmentation() {
    this.MF.initializeSegmentation().forEach((seg: any) => {
      const ctx = seg.user
      if (ctx.sourceBuffer.updating) {
        ctx.pending.push({ isInit: true, buffer: seg.buffer })
      } else {
        ctx.sourceBuffer.appendBuffer(seg.buffer)
        ctx.pending.push({ isInit: true })
      }
    })
  }

  // 设置轨道segment
  setSegmentOptions(track: any, shared: any) {
    const sourceBuffer = this.SourceBuffers.find((item) => item.codec === track.codec)!.sourceBuffer
    const ctx = {
      sourceBuffer,
      id: track.id,
      pending: [],
      shared,
    }
    sourceBuffer.onerror = (e) => console.error(e)
    sourceBuffer.onupdateend = () => this.updateEnd(ctx)
    this.MF.setSegmentOptions(track.id, ctx)
  }

  // 注入sourceBuffer完成
  updateEnd(ctx: any) {
    if (ctx.sourceBuffer.updating || this.MS.readyState !== 'open') return

    const seg = ctx.pending.shift()

    if (seg && seg.isInit) {
      ctx.shared.pendingInitCnt--
      if (seg.buffer) {
        ctx.sourceBuffer.appendBuffer(seg.buffer)
      }
    }

    if (ctx.shared.pendingInitCnt === 0 && !ctx.shared.loading) {
      ctx.shared.loading = true
      this.loadMediaData()
      return
    }

    if (ctx.isEof) {
      ctx.shared.notEndCnt--
    }

    if (ctx.shared.notEndCnt === 0 && !ctx.shared.isMSEnd) {
      if (ctx.sampleNum) {
        this.MF.releaseUsedSamples(ctx.id, ctx.sampleNum)
        ctx.sampleNum = null
      }

      ctx.shared.isMSEnd = true
      this.onComplete && this.onComplete()
      // if (this.SourceBuffers.every((item) => !item.sourceBuffer.updating)) {
      //   this.MS.endOfStream()
      //   this.onComplete && this.onComplete()
      // }

      this.el.currentTime = 0
      this.el.play()
    }

    if (seg && !seg.isInit) {
      ctx.sampleNum = seg.sampleNum
      ctx.isEof = seg.isEnd
      ctx.sourceBuffer.appendBuffer(seg.buffer)
    }
  }

  // 读取mp4文件配置
  loadMediaData() {
    this.MF.reader.chunkStart = this.MF.seek(0, true).offset
    this.MF.start()
    this.MF.reader.resume()
  }
}

MSE

最后一步，按照官方的Demo做MSE的初始化，接收ArrayBuffer即可

import { FetchFMp4 } from './transform'
const wURL = window.URL || webkitURL

export default class MSE {
  MS: MediaSource
  MF: FetchFMp4 | null

  constructor(
    public onComplete?: Function,
    public el = document.querySelector('video'),
    public mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"'
  ) {
    if (!this.el) throw new Error('error')
    this.MS = new MediaSource()
    this.el.src = wURL.createObjectURL(this.MS)
    this.MS.onsourceopen = () => this.onSourceOpen()
    this.MF = null
  }

  static isSupport(mimeCodec?: string): boolean {
    if (!mimeCodec) return !!window.MediaSource
    return window.MediaSource && MediaSource.isTypeSupported(mimeCodec)
  }

  onSourceOpen() {
    wURL.revokeObjectURL(this.el!.src)
    this.MS.onsourceopen = null
    // 配置MP4Box
    this.MF = new FetchFMp4(this.MS, this.el!, this.onComplete)
  }

  // 注入资源
  async append(file: string) {
    this.MF && this.MF.fetchFile(file)
  }
}

总结

MSE的使用相对简单，就是文档比较少，兼容性也比较差。
比较难的部分还是在MP4 -> FMP4。
由于接触音视频的场景比较少，对音视频编解码相关的知识比较薄弱，相关的文档也比较少。
总结一下，如果你也遇到类似的问题，可以考虑这个方案。
目前只是实现功能，还没有做相应优化，比如缓存、SourceBuffer append的位置、支持不同codec类型的音视频等。

non-switch-video

avatar
文章
阅读
粉丝