Replaykit 介绍
在之前的 iOS 版本中，iOS 开发者只能拿到编码后的数据，拿不到原始的 PCM 和 YUV，到 iOS 10 之后，开发者可以拿到原始数据，但是只能录制 App 内的内容，如果切到后台，将停止录制，直到 iOS 11，苹果对屏幕共享进行了升级并开放了权限，既可以拿到原始数据，又可以录制整个系统，以下我们重点来说 iOS 11 之后的屏幕共享功能。
系统屏幕共享
- (void)initMode_1 { self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, ScreenWidth, 80)]; self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.replaytest.Recoder"; self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0]; self.systemBroadcastPickerView.showsMicrophoneButton = NO; [self.view addSubview:self.systemBroadcastPickerView];}
在 iOS 11 创建一个 Extension 之后，调用上面的代码就可以开启屏幕共享了，然后系统会为我们生成一个 SampleHandler 的类，在这个方法中，苹果会根据 RPSampleBufferType 上报不同类型的数据。
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType
那怎么通过融云的 RongRTCLib 将屏幕共享数据发送出去呢？
1. 基于 Socket 的逼格玩法
1.1. Replaykit 框架启动和创建 Socket
//// ViewController.m// Socket_Replykit//// Created by Sun on 2020/5/19.// Copyright © 2020 RongCloud. All rights reserved.//#import "ViewController.h"#import <ReplayKit/ReplayKit.h>#import "RongRTCServerSocket.h"@interface ViewController ()<RongRTCServerSocketProtocol>@property (nonatomic, strong) RPSystemBroadcastPickerView *systemBroadcastPickerView;/** server socket */@property(nonatomic , strong)RongRTCServerSocket *serverSocket;@end@implementation ViewController- (void)viewDidLoad { [super viewDidLoad]; self.view.backgroundColor = [UIColor whiteColor]; // Do any additional setup after loading the view. [self.serverSocket createServerSocket]; self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, [UIScreen mainScreen].bounds.size.width, 80)]; self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.sealrtc.RongRTCRP"; self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0]; self.systemBroadcastPickerView.showsMicrophoneButton = NO; [self.view addSubview:self.systemBroadcastPickerView];}- (RongRTCServerSocket *)serverSocket { if (!_serverSocket) { RongRTCServerSocket *socket = [[RongRTCServerSocket alloc] init]; socket.delegate = self; _serverSocket = socket; } return _serverSocket;}- (void)didProcessSampleBuffer:(CMSampleBufferRef)sampleBuffer { // 这里拿到了最终的数据，比如最后可以使用融云的音视频SDK RTCLib 进行传输就可以了}@end
其中，包括了创建 Server Socket 的步骤，我们把主 App 当做 Server，然后屏幕共享 Extension 当做 Client ，通过 Socket 向我们的主 APP 发送数据。
在 Extension 里面，我们拿到 ReplayKit 框架上报的屏幕视频数据后：
//// SampleHandler.m// SocketReply//// Created by Sun on 2020/5/19.// Copyright © 2020 RongCloud. All rights reserved.//#import "SampleHandler.h"#import "RongRTCClientSocket.h"@interface SampleHandler()/** Client Socket */@property (nonatomic, strong) RongRTCClientSocket *clientSocket;@end@implementation SampleHandler- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo { // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional. self.clientSocket = [[RongRTCClientSocket alloc] init]; [self.clientSocket createCliectSocket];}- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType { switch (sampleBufferType) { case RPSampleBufferTypeVideo: // Handle video sample buffer [self sendData:sampleBuffer]; break; case RPSampleBufferTypeAudioApp: // Handle audio sample buffer for app audio break; case RPSampleBufferTypeAudioMic: // Handle audio sample buffer for mic audio break; default: break; }}- (void)sendData:(CMSampleBufferRef)sampleBuffer { [self.clientSocket encodeBuffer:sampleBuffer];}@end
可见 ，这里我们创建了一个 Client Socket，然后拿到屏幕共享的视频 sampleBuffer 之后，通过 Socket 发给我们的主 App，这就是屏幕共享的流程。
1.2 Local Socket 的使用
//// RongRTCSocket.m// SealRTC//// Created by Sun on 2020/5/7.// Copyright © 2020 RongCloud. All rights reserved.//#import "RongRTCSocket.h"#import <arpa/inet.h>#import <netdb.h>#import <sys/types.h>#import <sys/socket.h>#import <ifaddrs.h>#import "RongRTCThread.h"@interface RongRTCSocket()/** receive thread */@property (nonatomic, strong) RongRTCThread *receiveThread;@end@implementation RongRTCSocket- (int)createSocket { int socket = socket(AF_INET, SOCK_STREAM, 0); self.socket = socket; if (self.socket == -1) { close(self.socket); NSLog(@"socket error : %d", self.socket); } self.receiveThread = [[RongRTCThread alloc] init]; [self.receiveThread run]; return socket;}- (void)setSendBuffer { int optVal = 1024 * 1024 * 2; int optLen = sizeof(int); int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDBUF, (char *)&optVal,optLen); NSLog(@"set send buffer:%d", res);}- (void)setReceiveBuffer { int optVal = 1024 * 1024 * 2; int optLen = sizeof(int); int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVBUF, (char*)&optVal,optLen ); NSLog(@"set send buffer:%d",res);}- (void)setSendingTimeout { struct timeval timeout = {10,0}; int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeof(int)); NSLog(@"set send timeout:%d", res);}- (void)setReceiveTimeout { struct timeval timeout = {10, 0}; int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout, sizeof(int)); NSLog(@"set send timeout:%d", res);}- (BOOL)connect { NSString *serverHost = [self ip]; struct hostent *server = gethostbyname([serverHost UTF8String]); if (server == NULL) { close(self.socket); NSLog(@"get host error"); return NO; } struct in_addr *remoteAddr = (struct in_addr *)server->h_addr_list[0]; struct sockaddr_in addr; addr.sin_family = AF_INET; addr.sin_addr = *remoteAddr; addr.sin_port = htons(CONNECTPORT); int res = connect(self.socket, (struct sockaddr *) &addr, sizeof(addr)); if (res == -1) { close(self.socket); NSLog(@"connect error"); return NO; } NSLog(@"socket connect to server success"); return YES;}- (BOOL)bind { struct sockaddr_in client; client.sin_family = AF_INET; NSString *ipStr = [self ip]; if (ipStr.length <= 0) { return NO; } const char *ip = [ipStr cStringUsingEncoding:NSASCIIStringEncoding]; client.sin_addr.s_addr = inet_addr(ip); client.sin_port = htons(CONNECTPORT); int bd = bind(self.socket, (struct sockaddr *) &client, sizeof(client)); if (bd == -1) { close(self.socket); NSLog(@"bind error: %d", bd); return NO; } return YES;}- (BOOL)listen { int ls = listen(self.socket, 128); if (ls == -1) { close(self.socket); NSLog(@"listen error: %d", ls); return NO; } return YES;}- (void)receive { dispatch_async(dispatch_get_global_queue(0, 0), ^{ [self receiveData]; });}- (NSString *)ip { NSString *ip = nil; struct ifaddrs *addrs = NULL; struct ifaddrs *tmpAddrs = NULL; BOOL res = getifaddrs(&addrs); if (res == 0) { tmpAddrs = addrs; while (tmpAddrs != NULL) { if (tmpAddrs->ifa_addr->sa_family == AF_INET) { // Check if interface is en0 which is the wifi connection on the iPhone NSLog(@"%@", [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]); if ([[NSString stringWithUTF8String:tmpAddrs->ifa_name] isEqualToString:@"en0"]) { // Get NSString from C String ip = [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]; } } tmpAddrs = tmpAddrs->ifa_next; } } // Free memory freeifaddrs(addrs); NSLog(@"%@",ip); return ip;}- (void)close { int res = close(self.socket); NSLog(@"shut down: %d", res);}- (void)receiveData {}- (void)dealloc { [self.receiveThread stop];}@end
首先创建了一个 Socket 的父类，然后用 Server Socket 和 Client Socket 分别继承类来实现链接、绑定等操作。可以看到有些数据可以设置，有些则不用，这里不是核心，核心是怎样收发数据。
1.3 发送屏幕共享数据
//// RongRTCClientSocket.m// SealRTC//// Created by Sun on 2020/5/7.// Copyright © 2020 RongCloud. All rights reserved.//#import "RongRTCClientSocket.h"#import <arpa/inet.h>#import <netdb.h>#import <sys/types.h>#import <sys/socket.h>#import <ifaddrs.h>#import "RongRTCThread.h"#import "RongRTCSocketHeader.h"#import "RongRTCVideoEncoder.h"@interface RongRTCClientSocket() <RongRTCCodecProtocol> { pthread_mutex_t lock;}/** video encoder */@property (nonatomic, strong) RongRTCVideoEncoder *encoder;/** encode queue */@property (nonatomic, strong) dispatch_queue_t encodeQueue;@end@implementation RongRTCClientSocket- (BOOL)createClientSocket { if ([self createSocket] == -1) { return NO; } BOOL isC = [self connect]; [self setSendBuffer]; [self setSendingTimeout]; if (isC) { _encodeQueue = dispatch_queue_create("cn.rongcloud.encodequeue", NULL); [self createVideoEncoder]; return YES; } else { return NO; }}- (void)createVideoEncoder { self.encoder = [[RongRTCVideoEncoder alloc] init]; self.encoder.delegate = self; RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init]; settings.width = 720; settings.height = 1280; settings.startBitrate = 300; settings.maxFramerate = 30; settings.minBitrate = 1000; [self.encoder configWithSettings:settings onQueue:_encodeQueue];}- (void)clientSend:(NSData *)data { //data length NSUInteger dataLength = data.length; // data header struct DataHeader dataH; memset((void *)&dataH, 0, sizeof(dataH)); // pre PreHeader preH; memset((void *)&preH, 0, sizeof(preH)); preH.pre[0] = '&'; preH.dataLength = dataLength; dataH.preH = preH; // buffer int headerlength = sizeof(dataH); int totalLength = dataLength + headerlength; // srcbuffer Byte *src = (Byte *)[data bytes]; // send buffer char *buffer = (char *)malloc(totalLength * sizeof(char)); memcpy(buffer, &dataH, headerlength); memcpy(buffer + headerlength, src, dataLength); // tosend [self sendBytes:buffer length:totalLength]; free(buffer);}- (void)encodeBuffer:(CMSampleBufferRef)sampleBuffer { [self.encoder encode:sampleBuffer];}- (void)sendBytes:(char *)bytes length:(int)length { LOCK(self->lock); int hasSendLength = 0; while (hasSendLength < length) { // connect socket success if (self.socket > 0) { // send int sendRes = send(self.socket, bytes, length - hasSendLength, 0); if (sendRes == -1 || sendRes == 0) { UNLOCK(self->lock); NSLog(@"send buffer error"); [self close]; break; } hasSendLength += sendRes; bytes += sendRes; } else { NSLog(@"client socket connect error"); UNLOCK(self->lock); } } UNLOCK(self->lock); }- (void)spsData:(NSData *)sps ppsData:(NSData *)pps { [self clientSend:sps]; [self clientSend:pps];}- (void)naluData:(NSData *)naluData { [self clientSend:naluData];}- (void)deallo c{ NSLog(@"dealoc cliect socket");}@end
这里的核心思想是拿到屏幕共享的数据之后，先进行压缩，当压缩完成后会通过回调上报给当前类。既而通过 clientSend 方法，发给主 App。发给主 App 的数据中自定义了一个头部，头部添加了一个前缀和一个每次发送字节的长度，当接收端收到数据包后解析即可。
- (void)clientSend:(NSData *)data { //data length NSUInteger dataLength = data.length; // data header struct DataHeader dataH; memset((void *)&dataH, 0, sizeof(dataH)); // pre PreHeader preH; memset((void *)&preH, 0, sizeof(preH)); preH.pre[0] = '&'; preH.dataLength = dataLength; dataH.preH = preH; // buffer int headerlength = sizeof(dataH); int totalLength = dataLength + headerlength; // srcbuffer Byte *src = (Byte *)[data bytes]; // send buffer char *buffer = (char *)malloc(totalLength * sizeof(char)); memcpy(buffer, &dataH, headerlength); memcpy(buffer + headerlength, src, dataLength); // to send [self sendBytes:buffer length:totalLength]; free(buffer);}
1.4 接收屏幕共享数据
//// RongRTCServerSocket.m// SealRTC//// Created by Sun on 2020/5/7.// Copyright © 2020 RongCloud. All rights reserved.//#import "RongRTCServerSocket.h"#import <arpa/inet.h>#import <netdb.h>#import <sys/types.h>#import <sys/socket.h>#import <ifaddrs.h>#import <UIKit/UIKit.h>#import "RongRTCThread.h"#import "RongRTCSocketHeader.h"#import "RongRTCVideoDecoder.h"@interface RongRTCServerSocket() <RongRTCCodecProtocol>{ pthread_mutex_t lock; int _frameTime; CMTime _lastPresentationTime; Float64 _currentMediaTime; Float64 _currentVideoTime; dispatch_queue_t _frameQueue;}@property (nonatomic, assign) int acceptSocket;/** data length */@property (nonatomic, assign) NSUInteger dataLength;/** timeData */@property (nonatomic, strong) NSData *timeData;/** decoder queue */@property (nonatomic, strong) dispatch_queue_t decoderQueue;/** decoder */@property (nonatomic, strong) RongRTCVideoDecoder *decoder;@end@implementation RongRTCServerSocket- (BOOL)createServerSocket { if ([self createSocket] == -1) { return NO; } [self setReceiveBuffer]; [self setReceiveTimeout]; BOOL isB = [self bind]; BOOL isL = [self listen]; if (isB && isL) { _decoderQueue = dispatch_queue_create("cn.rongcloud.decoderQueue", NULL); _frameTime = 0; [self createDecoder]; [self receive]; return YES; } else { return NO; }}- (void)createDecoder { self.decoder = [[RongRTCVideoDecoder alloc] init]; self.decoder.delegate = self; RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init]; settings.width = 720; settings.height = 1280; settings.startBitrate = 300; settings.maxFramerate = 30; settings.minBitrate = 1000; [self.decoder configWithSettings:settings onQueue:_decoderQueue];}- (void)receiveData { struct sockaddr_in rest; socklen_t rest_size = sizeof(struct sockaddr_in); self.acceptSocket = accept(self.socket, (struct sockaddr *) &rest, &rest_size); while (self.acceptSocket != -1) { DataHeader dataH; memset(&dataH, 0, sizeof(dataH)); if (![self receiveData:(char *)&dataH length:sizeof(dataH)]) { continue; } PreHeader preH = dataH.preH; char pre = preH.pre[0]; if (pre == '&') { // rongcloud socket NSUInteger dataLenght = preH.dataLength; char *buff = (char *)malloc(sizeof(char) * dataLenght); if ([self receiveData:(char *)buff length:dataLenght]) { NSData *data = [NSData dataWithBytes:buff length:dataLenght]; [self.decoder decode:data]; free(buff); } } else { NSLog(@"pre is not &"); return; } }}- (BOOL)receiveData:(char *)data length:(NSUInteger)length { LOCK(lock); int receiveLength = 0; while (receiveLength < length) { ssize_t res = recv(self.acceptSocket, data, length - receiveLength, 0); if (res == -1 || res == 0) { UNLOCK(lock); NSLog(@"receive data error"); break; } receiveLength += res; data += res; } UNLOCK(lock); return YES;}- (void)didGetDecodeBuffer:(CVPixelBufferRef)pixelBuffer { _frameTime += 1000; CMTime pts = CMTimeMake(_frameTime, 1000); CMSampleBufferRef sampleBuffer = [RongRTCBufferUtil sampleBufferFromPixbuffer:pixelBuffer time:pts]; // Check to see if there is a problem with the decoded data. If the image appears, you are right. UIImage *image = [RongRTCBufferUtil imageFromBuffer:sampleBuffer]; [self.delegate didProcessSampleBuffer:sampleBuffer]; CFRelease(sampleBuffer);}- (void)close { int res = close(self.acceptSocket); self.acceptSocket = -1; NSLog(@"shut down server: %d", res); [super close];}- (void)dealloc { NSLog(@"dealoc server socket");}@end
主 App 通过 Socket 会持续收到数据包，再将数据包进行解码，将解码后的数据通过代理 didGetDecodeBuffer 代理方法回调给 App 层。App 层就可以通过融云 RongRTCLib 的发送自定义流方法将视频数据发送到对端。
1.5 VideotoolBox 硬编码
//// RongRTCVideoEncoder.m// SealRTC//// Created by Sun on 2020/5/13.// Copyright © 2020 RongCloud. All rights reserved.//#import "RongRTCVideoEncoder.h"#import "helpers.h"@interface RongRTCVideoEncoder() { VTCompressionSessionRef _compressionSession; int _frameTime;}/** settings */@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;/** callback queue */@property (nonatomic , strong ) dispatch_queue_t callbackQueue;- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer;- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer;@endvoid compressionOutputCallback(void *encoder, void *params, OSStatus status, VTEncodeInfoFlags infoFlags, CMSampleBufferRef sampleBuffer) { RongRTCVideoEncoder *videoEncoder = (__bridge RongRTCVideoEncoder *)encoder; if (status != noErr) { return; } if (infoFlags & kVTEncodeInfo_FrameDropped) { return; } BOOL isKeyFrame = NO; CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0); if (attachments != nullptr && CFArrayGetCount(attachments)) { CFDictionaryRef attachment = static_cast<CFDictionaryRef>(CFArrayGetValueAtIndex(attachments, 0)) ; isKeyFrame = !CFDictionaryContainsKey(attachment, kCMSampleAttachmentKey_NotSync); } CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(sampleBuffer); CMBlockBufferRef contiguous_buffer = nullptr; if (!CMBlockBufferIsRangeContiguous(block_buffer, 0, 0)) { status = CMBlockBufferCreateContiguous(nullptr, block_buffer, nullptr, nullptr, 0, 0, 0, &contiguous_buffer); if (status != noErr) { return; } } else { contiguous_buffer = block_buffer; CFRetain(contiguous_buffer); block_buffer = nullptr; } size_t block_buffer_size = CMBlockBufferGetDataLength(contiguous_buffer); if (isKeyFrame) { [videoEncoder sendSpsAndPPSWithSampleBuffer:sampleBuffer]; } if (contiguous_buffer) { CFRelease(contiguous_buffer); } [videoEncoder sendNaluData:sampleBuffer];}@implementation RongRTCVideoEncoder@synthesize settings = _settings;@synthesize callbackQueue = _callbackQueue;- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(nonnull dispatch_queue_t)queue { self.settings = settings; if (queue) { _callbackQueue = queue; } else { _callbackQueue = dispatch_get_main_queue(); } if ([self resetCompressionSession:settings]) { _frameTime = 0; return YES; } else { return NO; }}- (BOOL)resetCompressionSession:(RongRTCVideoEncoderSettings *)settings { [self destroyCompressionSession]; OSStatus status = VTCompressionSessionCreate(nullptr, settings.width, settings.height, kCMVideoCodecType_H264, nullptr, nullptr, nullptr, compressionOutputCallback, (__bridge void * _Nullable)(self), &_compressionSession); if (status != noErr) { return NO; } [self configureCompressionSession:settings]; return YES;}- (void)configureCompressionSession:(RongRTCVideoEncoderSettings *)settings { if (_compressionSession) { SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_RealTime, true); SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel); SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AllowFrameReordering, false); SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, 10); uint32_t targetBps = settings.startBitrate * 1000; SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, targetBps); SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ExpectedFrameRate, settings.maxFramerate); int bitRate = settings.width * settings.height * 3 * 4 * 4; SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, bitRate); int bitRateLimit = settings.width * settings.height * 3 * 4; SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimit); }}- (void)encode:(CMSampleBufferRef)sampleBuffer { CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer); CMTime pts = CMTimeMake(self->_frameTime++, 1000); VTEncodeInfoFlags flags; OSStatus res = VTCompressionSessionEncodeFrame(self->_compressionSession, imageBuffer, pts, kCMTimeInvalid, NULL, NULL, &flags); if (res != noErr) { NSLog(@"encode frame error:%d", (int)res); VTCompressionSessionInvalidate(self->_compressionSession); CFRelease(self->_compressionSession); self->_compressionSession = NULL; return; }}- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer { CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer); const uint8_t *sps ; const uint8_t *pps; size_t spsSize ,ppsSize , spsCount,ppsCount; OSStatus spsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sps, &spsSize, &spsCount, NULL); OSStatus ppsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pps, &ppsSize, &ppsCount, NULL); if (spsStatus == noErr && ppsStatus == noErr) { const char bytes[] = "\x00\x00\x00\x01"; size_t length = (sizeof bytes) - 1; NSMutableData *spsData = [NSMutableData dataWithCapacity:4+ spsSize]; NSMutableData *ppsData = [NSMutableData dataWithCapacity:4 + ppsSize]; [spsData appendBytes:bytes length:length]; [spsData appendBytes:sps length:spsSize]; [ppsData appendBytes:bytes length:length]; [ppsData appendBytes:pps length:ppsSize]; if (self && self.callbackQueue) { dispatch_async(self.callbackQueue, ^{ if (self.delegate && [self.delegate respondsToSelector:@selector(spsData:ppsData:)]) { [self.delegate spsData:spsData ppsData:ppsData]; } }); } } else { NSLog(@"sps status:%@, pps status:%@", @(spsStatus), @(ppsStatus)); }}- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer { size_t totalLength = 0; size_t lengthAtOffset=0; char *dataPointer; CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer); OSStatus status1 = CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &dataPointer); if (status1 != noErr) { NSLog(@"video encoder error, status = %d", (int)status1); return; } static const int h264HeaderLength = 4; size_t bufferOffset = 0; while (bufferOffset < totalLength - h264HeaderLength) { uint32_t naluLength = 0; memcpy(&naluLength, dataPointer + bufferOffset, h264HeaderLength); naluLength = CFSwapInt32BigToHost(naluLength); const char bytes[] = "\x00\x00\x00\x01"; NSMutableData *naluData = [NSMutableData dataWithCapacity:4 + naluLength]; [naluData appendBytes:bytes length:4]; [naluData appendBytes:dataPointer + bufferOffset + h264HeaderLength length:naluLength]; dispatch_async(self.callbackQueue, ^{ if (self.delegate && [self.delegate respondsToSelector:@selector(naluData:)]) { [self.delegate naluData:naluData]; } }); bufferOffset += naluLength + h264HeaderLength; }}- (void)destroyCompressionSession { if (_compressionSession) { VTCompressionSessionInvalidate(_compressionSession); CFRelease(_compressionSession); _compressionSession = nullptr; }}- (void)dealloc { if (_compressionSession) { VTCompressionSessionCompleteFrames(_compressionSession, kCMTimeInvalid); VTCompressionSessionInvalidate(_compressionSession); CFRelease(_compressionSession); _compressionSession = NULL; }}@end
1.6 VideotoolBox 解码
//// RongRTCVideoDecoder.m// SealRTC//// Created by Sun on 2020/5/14.// Copyright © 2020 RongCloud. All rights reserved.//#import "RongRTCVideoDecoder.h"#import <UIKit/UIKit.h>#import "helpers.h"@interface RongRTCVideoDecoder() { uint8_t *_sps; NSUInteger _spsSize; uint8_t *_pps; NSUInteger _ppsSize; CMVideoFormatDescriptionRef _videoFormatDescription; VTDecompressionSessionRef _decompressionSession;}/** settings */@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;/** callback queue */@property (nonatomic, strong) dispatch_queue_t callbackQueue;@endvoid DecoderOutputCallback(void * CM_NULLABLE decompressionOutputRefCon, void * CM_NULLABLE sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CM_NULLABLE CVImageBufferRef imageBuffer, CMTime presentationTimeStamp, CMTime presentationDuration ) { if (status != noErr) { NSLog(@" decoder callback error :%@", @(status)); return; } CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon; *outputPixelBuffer = CVPixelBufferRetain(imageBuffer); RongRTCVideoDecoder *decoder = (__bridge RongRTCVideoDecoder *)(decompressionOutputRefCon); dispatch_async(decoder.callbackQueue, ^{ [decoder.delegate didGetDecodeBuffer:imageBuffer]; CVPixelBufferRelease(imageBuffer); });}@implementation RongRTCVideoDecoder@synthesize settings = _settings;@synthesize callbackQueue = _callbackQueue;- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(dispatch_queue_t)queue { self.settings = settings; if (queue) { _callbackQueue = queue; } else { _callbackQueue = dispatch_get_main_queue(); } return YES;}- (BOOL)createVT { if (_decompressionSession) { return YES; } const uint8_t * const parameterSetPointers[2] = {_sps, _pps}; const size_t parameterSetSizes[2] = {_spsSize, _ppsSize}; int naluHeaderLen = 4; OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_videoFormatDescription ); if (status != noErr) { NSLog(@"CMVideoFormatDescriptionCreateFromH264ParameterSets error:%@", @(status)); return false; } NSDictionary *destinationImageBufferAttributes = @{ (id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange], (id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:self.settings.width], (id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:self.settings.height], (id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true] }; VTDecompressionOutputCallbackRecord CallBack; CallBack.decompressionOutputCallback = DecoderOutputCallback; CallBack.decompressionOutputRefCon = (__bridge void * _Nullable)(self); status = VTDecompressionSessionCreate(kCFAllocatorDefault, _videoFormatDescription, NULL, (__bridge CFDictionaryRef _Nullable)(destinationImageBufferAttributes), &CallBack, &_decompressionSession); if (status != noErr) { NSLog(@"VTDecompressionSessionCreate error:%@", @(status)); return false; } status = VTSessionSetProperty(_decompressionSession, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue); return YES;}- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize { CVPixelBufferRef outputPixelBuffer = NULL; CMBlockBufferRef blockBuffer = NULL; CMBlockBufferFlags flag0 = 0; OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer); if (status != kCMBlockBufferNoErr) { NSLog(@"VCMBlockBufferCreateWithMemoryBlock code=%d", (int)status); CFRelease(blockBuffer); return outputPixelBuffer; } CMSampleBufferRef sampleBuffer = NULL; const size_t sampleSizeArray[] = {frameSize}; status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _videoFormatDescription, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer); if (status != noErr || !sampleBuffer) { NSLog(@"CMSampleBufferCreateReady failed status=%d", (int)status); CFRelease(blockBuffer); return outputPixelBuffer; } VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback; VTDecodeInfoFlags infoFlag = kVTDecodeInfo_Asynchronous; status = VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag); if (status == kVTInvalidSessionErr) { NSLog(@"decode frame error with session err status =%d", (int)status); [self resetVT]; } else { if (status != noErr) { NSLog(@"decode frame error with status =%d", (int)status); } } CFRelease(sampleBuffer); CFRelease(blockBuffer); return outputPixelBuffer;}- (void)resetVT { [self destorySession]; [self createVT];}- (void)decode:(NSData *)data { uint8_t *frame = (uint8_t*)[data bytes]; uint32_t length = data.length; uint32_t nalSize = (uint32_t)(length - 4); uint32_t *pNalSize = (uint32_t *)frame; *pNalSize = CFSwapInt32HostToBig(nalSize); int type = (frame[4] & 0x1F); CVPixelBufferRef pixelBuffer = NULL; switch (type) { case 0x05: if ([self createVT]) { pixelBuffer= [self decode:frame withSize:length]; } break; case 0x07: self->_spsSize = length - 4; self->_sps = (uint8_t *)malloc(self->_spsSize); memcpy(self->_sps, &frame[4], self->_spsSize); break; case 0x08: self->_ppsSize = length - 4; self->_pps = (uint8_t *)malloc(self->_ppsSize); memcpy(self->_pps, &frame[4], self->_ppsSize); break; default: if ([self createVT]) { pixelBuffer = [self decode:frame withSize:length]; } break; }}- (void)dealloc { [self destorySession];}- (void)destorySession { if (_decompressionSession) { VTDecompressionSessionInvalidate(_decompressionSession); CFRelease(_decompressionSession); _decompressionSession = NULL; }}@end
1.7 工具类
//// RongRTCBufferUtil.m// SealRTC//// Created by Sun on 2020/5/8.// Copyright © 2020 RongCloud. All rights reserved.//#import "RongRTCBufferUtil.h"// 下面的这些方法，一定要记得release，有的没有在方法里面release，但是在外面release了，要不然会内存泄漏@implementation RongRTCBufferUtil+ (UIImage *)imageFromBuffer:(CMSampleBufferRef)buffer { CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(buffer); CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixelBuffer]; CIContext *temporaryContext = [CIContext contextWithOptions:nil]; CGImageRef videoImage = [temporaryContext createCGImage:ciImage fromRect:CGRectMake(0, 0, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer))]; UIImage *image = [UIImage imageWithCGImage:videoImage]; CGImageRelease(videoImage); return image;}+ (UIImage *)compressImage:(UIImage *)image newWidth:(CGFloat)newImageWidth { if (!image) return nil; float imageWidth = image.size.width; float imageHeight = image.size.height; float width = newImageWidth; float height = image.size.height/(image.size.width/width); float widthScale = imageWidth /width; float heightScale = imageHeight /height; UIGraphicsBeginImageContext(CGSizeMake(width, height)); if (widthScale > heightScale) { [image drawInRect:CGRectMake(0, 0, imageWidth /heightScale , height)]; } else { [image drawInRect:CGRectMake(0, 0, width , imageHeight /widthScale)]; } UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext(); UIGraphicsEndImageContext(); return newImage;}+ (CVPixelBufferRef)CVPixelBufferRefFromUiImage:(UIImage *)img { CGSize size = img.size; CGImageRef image = [img CGImage]; NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey, [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil]; CVPixelBufferRef pxbuffer = NULL; CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width, size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options, &pxbuffer); NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL); CVPixelBufferLockBaseAddress(pxbuffer, 0); void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer); NSParameterAssert(pxdata != NULL); CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB(); CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height, 8, 4*size.width, rgbColorSpace, kCGImageAlphaPremultipliedFirst); NSParameterAssert(context); CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image), CGImageGetHeight(image)), image); CGColorSpaceRelease(rgbColorSpace); CGContextRelease(context); CVPixelBufferUnlockBaseAddress(pxbuffer, 0); return pxbuffer;}+ (CMSampleBufferRef)sampleBufferFromPixbuffer:(CVPixelBufferRef)pixbuffer time:(CMTime)time { CMSampleBufferRef sampleBuffer = NULL; //获取视频信息 CMVideoFormatDescriptionRef videoInfo = NULL; OSStatus result = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixbuffer, &videoInfo); CMTime currentTime = time; // CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid}; CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid}; result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,pixbuffer, true, NULL, NULL, videoInfo, &timing, &sampleBuffer); CFRelease(videoInfo); return sampleBuffer;}+ (size_t)getCMTimeSize { size_t size = sizeof(CMTime); return size;}@end
此工具类中实现是由 CPU 处理，当进行 CMSampleBufferRef 转 UIImage、UIImage 转 CVPixelBufferRef、 CVPixelBufferRef 转 CMSampleBufferRef 以及裁剪图片时，这里需要注意将使用后的对象及时释放，否则会出现内存大量泄漏。
2. 视频发送
2.1 准备阶段
使用融云的 RongRTCLib 的前提需要一个 AppKey，请在官网（www.rongcloud.cn/)获取，通过 AppKey 取得 token 之后进行 IM 连接，在连接成功后加入 RTC 房间，这是屏幕共享发送的准备阶段。
- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo { // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional. // 请填写您的 AppKey self.appKey = @""; // 请填写用户的 Token self.token = @""; // 请指定房间号 self.roomId = @"123456"; [[RCIMClient sharedRCIMClient] initWithAppKey:self.appKey]; [[RCIMClient sharedRCIMClient] setLogLevel:RC_Log_Level_Verbose]; // 连接 IM [[RCIMClient sharedRCIMClient] connectWithToken:self.token dbOpened:^(RCDBErrorCode code) { NSLog(@"dbOpened: %zd", code); } success:^(NSString *userId) { NSLog(@"connectWithToken success userId: %@", userId); // 加入房间 [[RCRTCEngine sharedInstance] joinRoom:self.roomId completion:^(RCRTCRoom * _Nullable room, RCRTCCode code) { self.room = room; self.room.delegate = self; [self publishScreenStream]; }]; } error:^(RCConnectErrorCode errorCode) { NSLog(@"ERROR status: %zd", errorCode); }];}
如上是连接 IM 和加入 RTC 房间的全过程，其中还包含调用发布自定义视频 [self publishScreenStream]; 此方法在加入房间成功后才可以进行。
- (void)publishScreenStream { RongRTCStreamParams *param = [[RongRTCStreamParams alloc] init]; param.videoSizePreset = RongRTCVideoSizePreset1280x720; self.videoOutputStream = [[RongRTCAVOutputStream alloc] initWithParameters:param tag:@"RongRTCScreenVideo"]; [self.room publishAVStream:self.videoOutputStream extra:@"" completion:^(BOOL isSuccess, RongRTCCode desc) { if (isSuccess) { NSLog(@"发布自定义流成功"); } }];}
自定义一个 RongRTCAVOutputStream 流即可，使用此流发送屏幕共享数据。
2.2 开始发送屏幕共享数据
上面我们已经连接了融云的 IM 和加入了 RTC 房间，并且自定义了一个发送屏幕共享的自定义流，接下来，如何将此流发布出去呢？
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType { switch (sampleBufferType) { case RPSampleBufferTypeVideo: // Handle video sample buffer [self.videoOutputStream write:sampleBuffer error:nil]; break; case RPSampleBufferTypeAudioApp: // Handle audio sample buffer for app audio break; case RPSampleBufferTypeAudioMic: // Handle audio sample buffer for mic audio break; default: break; }}
但我们接收到了苹果上报的数据之后，调用 RongRTCAVOutputStream 中的 write:error: 方法，将 sampleBuffer 发送给远端，至此，屏幕共享数据就发送出去啦。
[self.videoOutputStream write:sampleBuffer error:nil];
融云的核心代码就是通过上面的连接 IM，加入房间，发布自定义流，然后通过自定义流的 write:error: 方法将 sampleBuffer 发送出去。
不管是通过 ReplayKit 取得屏幕视频，还是使用 Socket 在进程间传输，都是为最终的 write:error: 服务。
总结
-
Extension内存是有限制的，最大 50M，所以在Extension里面处理数据需要格外注意内存释放； -
如果
VideotoolBox在后台解码一直失败，只需把VideotoolBox重启一下即可，此步骤在上面的代码中有体现； -
如果不需要将
Extension的数据传到主 App，只需在Extension里直接将流通过RongRTCLib发布出去即可，缺点是Extension中发布自定义流的用户与主 App 中的用户不是同一个，这也是上面通过Socket将数据传递给主App要解决的问题； -
如果主 App 需要拿到屏幕共享的数据处理，使用
Socket将流先发给主 App，然后在主 App 里面通过RongRTCLib将流发出去。
最后附上 Demo
