Android mirrorSurface镜像分屏

前言

在之前的一篇文章中，我们介绍了Surface的截屏方法，但是在文章的后半部分，我们引入了一个新的问题——副屏录制。众所周知，Android系统默认投影主屏的，这部分逻辑可以查看DisplayContent、VirtualDeviceAdapter、OverlayDisplayAdapter等相关实现。当然，Android 11开始，理论上是可以选择投影副屏的，这点我们后续有时间再做介绍。

// com.android.server.display.DisplayDevice#getDisplayIdToMirrorLocked
/**
 * Gets the id of the display to mirror.
 */
public int getDisplayIdToMirrorLocked() {
    return Display.DEFAULT_DISPLAY;
}

本篇原理

SurfaceControl reparent 机制

在Android 10系统开始，官方为了解决MediaCodec#setOutputSurface稳定性问题，引入了SurfaceControl#reparent机制，简单的说是离屏渲染机制。通过SurfaceControl创建一个离屏的Surface，传入到MediaCodec中，从而避免调用MediaCodec#setOuputSurface。当然，这种方案目前无法兼容到很早的版本，因此Android 10之前还得通过open gl es或者TextureView复用机制来解决此问题。

通过下面方法，我们就能实现SurfaceView的随意切换了，当然，这部分具体底层原理没有研究过，涉及到的是framework部分代码，关联也比较多。

private static void reparent(@Nullable SurfaceView surfaceView) {
  SurfaceControl surfaceControl = Assertions.checkNotNull(MirrorMainActivity.surfaceControl);
  if (surfaceView == null) {
    //不渲染SurfaceView时
    new SurfaceControl.Transaction()
        .reparent(surfaceControl, /* newParent= */ null)
        .setBufferSize(surfaceControl, /* w= */ 0, /* h= */ 0)
        .setVisibility(surfaceControl, /* visible= */ true)
        .apply();
  } else {
  //需要渲染指定的SurfaceView时
    SurfaceControl newParentSurfaceControl = surfaceView.getSurfaceControl();
    new SurfaceControl.Transaction()
        .reparent(surfaceControl, newParentSurfaceControl)
        .setBufferSize(surfaceControl, surfaceView.getWidth(), surfaceView.getHeight())
        .setVisibility(surfaceControl, /* visible= */ true)
        .apply();
  }
}

SurfaceControl mirrorSurface机制

mirrorSurface是Android 11引入的镜像机制，通过这个机制，就能实现将上面的离屏Surface镜像到其他所有View。当然，这个由于是系统隐藏类，因此我们需要借助一下非常手段。

反射api

由于类似freeflection项目，由于版本限制，一些方法的反射难度大幅提高，相比而言hiddenapibypass的反射成功率还是相当不错的，因此这里使用hiddenapibypass，其本质是借助了MethodHandle来实现反射的。

implementation 'org.lsposed.hiddenapibypass:hiddenapibypass:4.3'

权限问题

本质上，在Android官方服务com.android.server.wm.WindowManagerService#mirrorDisplay中，是需要权限的，但是这个显然在普通app中是可以绕过去的，我们无需理会。

public boolean mirrorDisplay(int displayId, SurfaceControl outSurfaceControl) {
    if (!checkCallingPermission(READ_FRAME_BUFFER, "mirrorDisplay()")) {
        throw new SecurityException("Requires READ_FRAME_BUFFER permission");
    }

    final SurfaceControl displaySc;
    synchronized (mGlobalLock) {
        DisplayContent displayContent = mRoot.getDisplayContent(displayId);
        if (displayContent == null) {
            Slog.e(TAG, "Invalid displayId " + displayId + " for mirrorDisplay");
            return false;
        }

        displaySc = displayContent.getWindowingLayer();
    }

    final SurfaceControl mirror = SurfaceControl.mirrorSurface(displaySc);
    outSurfaceControl.copyFrom(mirror, "WMS.mirrorDisplay");

    return true;
}

分屏过程

在系统代码中，Android官方对于mirrorSurface有使用，我们参考即可，上面的代码就是一个例子。

分屏过程：

• 1.创建一个SurfaceControl
• 2.通过mirrorSurface将原来的SurfaceControl镜像到（1）创建的SurfaceControl
• 3.利用reparent机制实现分屏渲染

分屏实现

以上我们了解实现原理和实现过程，下面我们来实现一下

获取SurfaceControl镜像

下面是获取镜像的视线核心代码，下面的代码是将inputSurfaceControl镜像出一个outSurfaceControl

public SurfaceControl mirrorSurfaceControl(SurfaceControl inputSurfaceControl) {
  try {
    if(mirrorSurfaceMethod == null) {
      mirrorSurfaceMethod = HiddenApiBypass.getDeclaredMethod(SurfaceControl.class, "mirrorSurface",SurfaceControl.class);
      mirrorSurfaceMethod.setAccessible(true);
    }
    SurfaceControl outSurfaceControl = (SurfaceControl) mirrorSurfaceMethod.invoke(SurfaceControl.class,inputSurfaceControl); 
    return outSurfaceControl;
  } catch (Exception e) {
    e.printStackTrace();
  }
  return null;
}

当然SurfaceControl资源是比较昂贵的，因此这里我们有必要进行保存一下，使得SurfaceView和SurfaceControl建立映射关系

private final Map<SurfaceView,SurfaceControl> viewSurfaceControlMap = new HashMap<>();

reparent实现渲染

拿到SurfaceControl我们便能实现镜像功能，核心代码如下

SurfaceControl mirrorSurfaceControl = viewSurfaceControlMap.get(surfaceView);
if(mirrorSurfaceControl == null || !mirrorSurfaceControl.isValid()) {
  mirrorSurfaceControl = mirrorSurfaceControl(MirrorMainActivity.surfaceControl);
  viewSurfaceControlMap.put(surfaceView,mirrorSurfaceControl);
}
new SurfaceControl.Transaction()
    .reparent(mirrorSurfaceControl, surfaceView.getSurfaceControl())
    .setBufferSize(mirrorSurfaceControl, surfaceView.getWidth(), surfaceView.getHeight())
    .setVisibility(mirrorSurfaceControl, /* visible= */ true)
    .apply();

看着是不是很简单，核心代码就此完成，但是你可能会疑惑SurfaceControl.Transaction是干啥的。其本身是一个类似FragmentTransaction的事务，很显然，他不允许动态修改，只认可最后一次提交。

BLASTBufferQueue 问题

问题是，上面的代码在一些系统中仍然无法工作，具体占比多少，或者是不是与系统有关，暂时没有明确的数据。

对比SurfaceControl和Surface之间的关系可知，可能原因是部分系统需要有一个与BLASTBufferQueue关联的SurfaceControl作为缓冲，其他Surface才能被渲染，这部分其实在SurfaceView中也是这样做的。但是BLASTBufferQueue也属于隐藏的类，仍然需要反射。不过在本篇内容中，这部分当然也没多仔细研究过，就不在赘述了，但理论上应该就是这里有问题。

另外，使用BLASTBufferQueue比较繁琐。本篇这里使用了另一种方式————先reparent一个SurfaceView到离屏的SurfaceControl,然后再将其他SurfaceView reparent到 Mirror的SurfaceControl上便能解决此问题。

private void mirrorSurface(SurfaceView surfaceView,int index) {
  if(index == 0){
    reparent(surfaceView);
    return;
  }
  SurfaceControl mirrorSurfaceControl = viewSurfaceControlMap.get(surfaceView);
  if(mirrorSurfaceControl == null || !mirrorSurfaceControl.isValid()) {
    mirrorSurfaceControl = mirrorSurfaceControl(MirrorMainActivity.surfaceControl);
    viewSurfaceControlMap.put(surfaceView,mirrorSurfaceControl);
  }
  new SurfaceControl.Transaction()
      .reparent(mirrorSurfaceControl, surfaceView.getSurfaceControl())
      .setBufferSize(mirrorSurfaceControl, surfaceView.getWidth(), surfaceView.getHeight())
      .setVisibility(mirrorSurfaceControl, /* visible= */ true)
      .apply();

}

最终效果

下面是最终实现的效果，最终我们利用reparent和mirrorSurface实现了分屏效果

TextureView问题

从本篇的代码片段我们知道，reparent机制比较依赖SurfaceControl，然而TextureView内部并没有SurfaceControl，这显然是一个问题，本篇的方案目前仅仅探索到SurfaceView，对于TextureView其实目前仍然只能借助open gl es，似乎没有其他更好的方法。

总结

在本篇我们了解SurfaceControl的mirrorSurface机制和reparent机制的用法，其实SurfaceControl很强大，比如genymotion公司著名的开源项目scrcpy大量使用SurfaceControl，不过由于需要录屏权限和SurfaceFlinger访问权限，导致scrcpy只能在adb模式下使用。

不过话说回来，SurfaceControl对于Android开发者既熟悉又陌生，由于底层设计复杂，且其用法也让人难以理解，目前而言很少有SurfaceControl的相关源码分析，因此而言，这个工具类仍然需要大家多学习。

好了，本篇就到这里，希望对你有所帮助。

附1：源码

下面是核心代码逻辑，这里主要使用了ExoPlayer播放器

public final class MirrorMainActivity extends Activity {

  private static final String DEFAULT_MEDIA_URI =
      "https://storage.googleapis.com/exoplayer-test-media-1/mkv/android-screens-lavf-56.36.100-aac-avc-main-1280x720.mkv";
  private static final String SURFACE_CONTROL_NAME = "surfacedemo";

  private static final String ACTION_VIEW = "com.google.android.exoplayer.surfacedemo.action.VIEW";
  private static final String EXTENSION_EXTRA = "extension";
  private static final String DRM_SCHEME_EXTRA = "drm_scheme";
  private static final String DRM_LICENSE_URL_EXTRA = "drm_license_url";
  private static final String OWNER_EXTRA = "owner";
  private final Map<SurfaceView,SurfaceControl> viewSurfaceControlMap = new HashMap<>();

  @Nullable private PlayerControlView playerControlView;
  @Nullable private SurfaceView fullScreenView;
  @Nullable private SurfaceView nonFullScreenView;
  @Nullable private SurfaceView currentOutputView;

  @Nullable private static ExoPlayer player;
  @Nullable private static SurfaceControl surfaceControl;
  @Nullable private static Surface videoSurface;
  private Method mirrorSurfaceMethod = null;
  @Override
  protected void onCreate(@Nullable Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.mirror_main_activity);

    playerControlView = findViewById(R.id.player_control_view);
    fullScreenView = findViewById(R.id.full_screen_view);
    fullScreenView.setOnClickListener(
        v -> {
          setCurrentOutputView(nonFullScreenView);
          Assertions.checkNotNull(fullScreenView).setVisibility(View.GONE);
        });
    attachSurfaceListener(fullScreenView);
    GridLayout gridLayout = findViewById(R.id.grid_layout);
    for (int i = 0; i < 9; i++) {
      View view;
      if (i == 0) {
        Button button = new Button(/* context= */ this);
        view = button;
        button.setText(getString(R.string.no_output_label));
        button.setOnClickListener(v -> reparent(/* surfaceView= */ null));
      } else if (i == 1) {
        Button button = new Button(/* context= */ this);
        view = button;
        button.setText(getString(R.string.full_screen_label));
        button.setOnClickListener(
            v -> {
              setCurrentOutputView(fullScreenView);
              Assertions.checkNotNull(fullScreenView).setVisibility(View.VISIBLE);
            });
      } else if (i == 2) {
        Button button = new Button(/* context= */ this);
        view = button;
        button.setText(getString(R.string.mirror_surface_label));
        button.setOnClickListener(v -> {
          reparent(null);
          int childCount = gridLayout.getChildCount();
          int index = 0;
          for (int c = 0; c < childCount; c++) {
            View child = gridLayout.getChildAt(c);
            if (child instanceof SurfaceView) {
              mirrorSurface((SurfaceView) child,index);
              index++;
            }
          }
        });
      } else {
        SurfaceView surfaceView = new SurfaceView(this);
        view = surfaceView;
        attachSurfaceListener(surfaceView);
        if (nonFullScreenView == null) {
          nonFullScreenView = surfaceView;
        }
      }
      gridLayout.addView(view);
      GridLayout.LayoutParams layoutParams = new GridLayout.LayoutParams();
      layoutParams.width = 0;
      layoutParams.height = 0;
      layoutParams.columnSpec = GridLayout.spec(i % 3, 1f);
      layoutParams.rowSpec = GridLayout.spec(i / 3, 1f);
      layoutParams.bottomMargin = 10;
      layoutParams.leftMargin = 10;
      layoutParams.topMargin = 10;
      layoutParams.rightMargin = 10;
      view.setLayoutParams(layoutParams);
    }
  }

  @Override
  public void onResume() {
    super.onResume();

    if (player == null) {
      initializePlayer();
    }

    setCurrentOutputView(nonFullScreenView);

    PlayerControlView playerControlView = Assertions.checkNotNull(this.playerControlView);
    playerControlView.setPlayer(player);
    playerControlView.show();
  }

  @Override
  public void onPause() {
    super.onPause();

    Assertions.checkNotNull(playerControlView).setPlayer(null);
  }

  @Override
  public void onDestroy() {
    super.onDestroy();
    if (isFinishing()) {
      if (surfaceControl != null) {
        surfaceControl.release();
        surfaceControl = null;
      }
      if (videoSurface != null) {
        videoSurface.release();
        videoSurface = null;
      }
      for (Map.Entry<SurfaceView,SurfaceControl> entry : viewSurfaceControlMap.entrySet()) {
        entry.getValue().release();
      }
      viewSurfaceControlMap.clear();
      if (player != null) {
        player.release();
        player = null;
      }
    }
  }

  private void initializePlayer() {
    Intent intent = getIntent();
    String action = intent.getAction();
    Uri uri =
        ACTION_VIEW.equals(action)
            ? Assertions.checkNotNull(intent.getData())
            : Uri.parse(DEFAULT_MEDIA_URI);
    DrmSessionManager drmSessionManager;
    if (intent.hasExtra(DRM_SCHEME_EXTRA)) {
      String drmScheme = Assertions.checkNotNull(intent.getStringExtra(DRM_SCHEME_EXTRA));
      String drmLicenseUrl = Assertions.checkNotNull(intent.getStringExtra(DRM_LICENSE_URL_EXTRA));
      UUID drmSchemeUuid = Assertions.checkNotNull(Util.getDrmUuid(drmScheme));
      DataSource.Factory licenseDataSourceFactory = new DefaultHttpDataSource.Factory();
      HttpMediaDrmCallback drmCallback =
          new HttpMediaDrmCallback(drmLicenseUrl, licenseDataSourceFactory);
      drmSessionManager =
          new DefaultDrmSessionManager.Builder()
              .setUuidAndExoMediaDrmProvider(drmSchemeUuid, FrameworkMediaDrm.DEFAULT_PROVIDER)
              .build(drmCallback);
    } else {
      drmSessionManager = DrmSessionManager.DRM_UNSUPPORTED;
    }

    DataSource.Factory dataSourceFactory = new DefaultDataSource.Factory(this);
    MediaSource mediaSource;
    @Nullable String fileExtension = intent.getStringExtra(EXTENSION_EXTRA);
    @C.ContentType
    int type =
        TextUtils.isEmpty(fileExtension)
            ? Util.inferContentType(uri)
            : Util.inferContentTypeForExtension(fileExtension);
    if (type == C.CONTENT_TYPE_DASH) {
      mediaSource =
          new DashMediaSource.Factory(dataSourceFactory)
              .setDrmSessionManagerProvider(unusedMediaItem -> drmSessionManager)
              .createMediaSource(MediaItem.fromUri(uri));
    } else if (type == C.CONTENT_TYPE_OTHER) {
      mediaSource =
          new ProgressiveMediaSource.Factory(dataSourceFactory)
              .setDrmSessionManagerProvider(unusedMediaItem -> drmSessionManager)
              .createMediaSource(MediaItem.fromUri(uri));
    } else {
      throw new IllegalStateException();
    }
    ExoPlayer player = new ExoPlayer.Builder(getApplicationContext()).build();
    player.setMediaSource(mediaSource);
    player.prepare();
    player.play();
    player.setRepeatMode(Player.REPEAT_MODE_ALL);

    SurfaceControl.Builder builder = new SurfaceControl.Builder()
        .setName(SURFACE_CONTROL_NAME)
        .setBufferSize(/* width= */ 0, /* height= */ 0);

    surfaceControl = builder.build();
    videoSurface = new Surface(surfaceControl);
    player.setVideoSurface(videoSurface);
    MirrorMainActivity.player = player;
  }

  private void setCurrentOutputView(@Nullable SurfaceView surfaceView) {
    currentOutputView = surfaceView;
    if (surfaceView != null && surfaceView.getHolder().getSurface() != null) {
      reparent(surfaceView);
    }
  }

  private void attachSurfaceListener(SurfaceView surfaceView) {
    surfaceView
        .getHolder()
        .addCallback(
            new SurfaceHolder.Callback() {
              @Override
              public void surfaceCreated(SurfaceHolder surfaceHolder) {
                if (surfaceView == currentOutputView) {
                  reparent(surfaceView);
                }
              }

              @Override
              public void surfaceChanged(
                  SurfaceHolder surfaceHolder, int format, int width, int height) {}

              @Override
              public void surfaceDestroyed(SurfaceHolder surfaceHolder) {}
            });
  }
  private static void reparent(@Nullable SurfaceView surfaceView) {
    SurfaceControl surfaceControl = Assertions.checkNotNull(MirrorMainActivity.surfaceControl);
    if (surfaceView == null) {
      new SurfaceControl.Transaction()
          .reparent(surfaceControl, /* newParent= */ null)
          .setBufferSize(surfaceControl, /* w= */ 0, /* h= */ 0)
          .setVisibility(surfaceControl, /* visible= */ true)
          .apply();
    } else {
      SurfaceControl newParentSurfaceControl = surfaceView.getSurfaceControl();
      new SurfaceControl.Transaction()
          .reparent(surfaceControl, newParentSurfaceControl)
          .setBufferSize(surfaceControl, surfaceView.getWidth(), surfaceView.getHeight())
          .setVisibility(surfaceControl, /* visible= */ true)
          .apply();
    }
  }

  private void mirrorSurface(SurfaceView surfaceView,int index) {
    if(index == 0){
      reparent(surfaceView);
      return;
    }
    SurfaceControl mirrorSurfaceControl = viewSurfaceControlMap.get(surfaceView);
    if(mirrorSurfaceControl == null || !mirrorSurfaceControl.isValid()) {
      mirrorSurfaceControl = mirrorSurfaceControl(MirrorMainActivity.surfaceControl);
      viewSurfaceControlMap.put(surfaceView,mirrorSurfaceControl);
    }
    new SurfaceControl.Transaction()
        .reparent(mirrorSurfaceControl, surfaceView.getSurfaceControl())
        .setBufferSize(mirrorSurfaceControl, surfaceView.getWidth(), surfaceView.getHeight())
        .setVisibility(mirrorSurfaceControl, /* visible= */ true)
        .apply();

  }
  public SurfaceControl mirrorSurfaceControl(SurfaceControl inputSurfaceControl) {
    try {
      if(mirrorSurfaceMethod == null) {
        mirrorSurfaceMethod = HiddenApiBypass.getDeclaredMethod(SurfaceControl.class, "mirrorSurface",SurfaceControl.class);
        mirrorSurfaceMethod.setAccessible(true);
      }
      SurfaceControl outSurfaceControl = (SurfaceControl) mirrorSurfaceMethod.invoke(SurfaceControl.class,inputSurfaceControl);
      return outSurfaceControl;
    } catch (Exception e) {
      e.printStackTrace();
    }
    return null;
  }
}

xml 布局文件如下

<FrameLayout
  xmlns:android="http://schemas.android.com/apk/res/android"
  xmlns:app="http://schemas.android.com/apk/res-auto"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:keepScreenOn="true">

  <LinearLayout
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical">

    <GridLayout
      android:id="@+id/grid_layout"
      android:layout_width="match_parent"
      android:layout_height="0dp"
      android:layout_weight="1"
      android:columnCount="3"/>

    <com.google.android.exoplayer2.ui.PlayerControlView
      android:id="@+id/player_control_view"
      android:layout_width="match_parent"
      android:layout_height="wrap_content"
      android:layout_gravity="bottom"
      app:show_timeout="0"/>

  </LinearLayout>

  <SurfaceView
    android:id="@+id/full_screen_view"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:visibility="gone"/>

</FrameLayout>

附2：多屏镜像

实际上，mirrorSurface不仅仅能实现SurfaceView之间的分屏，还能实现Display之间的分屏，核心源码参考官方的demo 《官方案例MirrorSuraceTest》

核心代码如下

private void createMirror(Rect displayFrame, float scale) {
    boolean success = false;
    try {
        success = WindowManagerGlobal.getWindowManagerService().mirrorDisplay(mDisplayId, mSurfaceControl);  //获取WMS Binder
    
    } catch (RemoteException e) {
    }
    if (!success) {
        return;
    }
    if (!mSurfaceControl.isValid()) {
        return;
    }
    mHasMirror = true;
    mBorderSc = new SurfaceControl.Builder()  // 边框，非核心逻辑
            .setName("Mirror Border")
            .setBufferSize(1, 1)
            .setFormat(PixelFormat.TRANSLUCENT)
            .build();
    updateMirror(displayFrame, scale);  //更新尺寸
    mTransaction
            .setVisibility(mSurfaceControl,true)
            .reparent(mSurfaceControl, mOverlayView.getViewRootImpl().getSurfaceControl())
            .setLayer(mBorderSc, 1)
            .setVisibility(mBorderSc,true)
            .reparent(mBorderSc, mSurfaceControl)
            .apply();
}
private void updateMirror(Rect displayFrame, float scale) {
    if (displayFrame.isEmpty()) {
        Rect bounds = mWindowBounds;
        int defaultCropW = Math.round(bounds.width() / 2);
        int defaultCropH = Math.round(bounds.height() / 2);
        displayFrame.set(0, 0, defaultCropW, defaultCropH);
    }
    if (scale <= 0) {
        scale = DEFAULT_SCALE;
    }
    mCurrFrame.set(displayFrame);
    mCurrScale = scale;
    int width = (int) Math.ceil(displayFrame.width() / scale);
    int height = (int) Math.ceil(displayFrame.height() / scale);
    Rect sourceBounds = getSourceBounds(displayFrame, scale);
    mTransaction.setGeometry(mSurfaceControl, sourceBounds, displayFrame, Surface.ROTATION_0)
            .setPosition(mBorderSc, sourceBounds.left, sourceBounds.top)
            .setBufferSize(mBorderSc, width, height)
            .apply();
    drawBorder(mBorderSc, width, height, (int) Math.ceil(BORDER_SIZE / scale));
    mSourcePositionText.setText(sourceBounds.left + ", " + sourceBounds.top);
    mDisplayFrameText.setText(
            String.format("%s, %s, %s, %s", mCurrFrame.left, mCurrFrame.top, mCurrFrame.right,
                    mCurrFrame.bottom));
    mScaleText.setText(String.valueOf(mCurrScale));
}
//绘制边框
private void drawBorder(SurfaceControl borderSc, int width, int height, int borderSize) {
    mTmpSurface.copyFrom(borderSc);
    Canvas c = null;
    try {
        c = mTmpSurface.lockCanvas(null);
    } catch (IllegalArgumentException | Surface.OutOfResourcesException e) {
    }
    if (c == null) {
        return;
    }
    // Top
    c.save();
    c.clipRect(new Rect(0, 0, width, borderSize));
    c.drawColor(DEFAULT_BORDER_COLOR);
    c.restore();
    // Left
    c.save();
    c.clipRect(new Rect(0, 0, borderSize, height));
    c.drawColor(DEFAULT_BORDER_COLOR);
    c.restore();
    // Right
    c.save();
    c.clipRect(new Rect(width - borderSize, 0, width, height));
    c.drawColor(DEFAULT_BORDER_COLOR);
    c.restore();
    // Bottom
    c.save();
    c.clipRect(new Rect(0, height - borderSize, width, height));
    c.drawColor(DEFAULT_BORDER_COLOR);
    c.restore();
    mTmpSurface.unlockCanvasAndPost(c);
}