在上一篇文章加快apk的构建速度，如何把编译时间从130秒降到17秒中讲了优化的思路与初步的实现，经过一段时间的优化性能和稳定性都有很大的提高，这里要感谢大家提的建议以及github上的issue，这篇文章就把主要优化的点和新功能以及填的坑介绍下。
项目地址: github.com/typ0520/fas…
对应tag: github.com/typ0520/fas…
demo代码: github.com/typ0520/fas…
注: 建议把fastdex的代码和demo代码拉下来，本文中的绝大部分例子在demo工程中可以直接跑
注: 本文对gradle task做的说明都建立在关闭instant run的前提下
注: 本文所有的代码、gradle任务名、任务输出路径、全部使用debug这个buildType作说明
注: 本文使用./gradlew执行任务是在mac下，如果是windows换成gradlew.bat
###一、拦截transformClassesWithJarMergingForDebug任务
之前补丁打包的时候，是把没有变化的类从app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/combined.jar中移除，这样的做法有两个问题
- 1、combined.jar这个文件是 transformClassesWithJarMergingForDebug任务输出的，存在这个任务的前提是开启了multidex，如果没有开启那么执行到 transformClassesWithDexForDebug任务时输入就不在是combined.jar，而是项目的classes目录(app/build/intermediates/classes/debug)和依赖的library输出的jar以及第三方库的jar；
- 2、如果存在transformClassesWithJarMergingForDebug任务，先花费大量时间合成combined.jar，然后在把没有变化的类从combined.jar中移除，这样效率太低了，如果绕过combined.jar的合成直接拿变化class去生成dex对效率会有很大的提高
现在首先需要拿到transformClassesWithJarMergingForDebug任务执行前后的生命周期，实现的方式和拦截transformClassesWithDexForDebug时用的方案差不多,完整的测试代码地址
github.com/typ0520/fas…
public class MyJarMergingTransform extends Transform {
Transform base
MyJarMergingTransform(Transform base) {
this.base = base
}
@Override
void transform(TransformInvocation invocation) throws TransformException, IOException, InterruptedException {
List<JarInput> jarInputs = Lists.newArrayList();
List<DirectoryInput> dirInputs = Lists.newArrayList();
for (TransformInput input : invocation.getInputs()) {
jarInputs.addAll(input.getJarInputs());
}
for (TransformInput input : invocation.getInputs()) {
dirInputs.addAll(input.getDirectoryInputs());
}
for (JarInput jarInput : jarInputs) {
println("==jarmerge jar : ${jarInput.file}")
}
for (DirectoryInput directoryInput : dirInputs) {
println("==jarmerge directory: ${directoryInput.file}")
}
File combinedJar = invocation.outputProvider.getContentLocation("combined", base.getOutputTypes(), base.getScopes(), Format.JAR);
println("==combinedJar exists ${combinedJar.exists()} ${combinedJar}")
base.transform(invocation)
println("==combinedJar exists ${combinedJar.exists()} ${combinedJar}")
}
}
public class MyDexTransform extends Transform {
Transform base
MyDexTransform(Transform base) {
this.base = base
}
@Override
void transform(TransformInvocation transformInvocation) throws TransformException, IOException, InterruptedException {
List<JarInput> jarInputs = Lists.newArrayList();
List<DirectoryInput> dirInputs = Lists.newArrayList();
for (TransformInput input : transformInvocation.getInputs()) {
jarInputs.addAll(input.getJarInputs());
}
for (TransformInput input : transformInvocation.getInputs()) {
dirInputs.addAll(input.getDirectoryInputs());
}
for (JarInput jarInput : jarInputs) {
println("==dex jar : ${jarInput.file}")
}
for (DirectoryInput directoryInput : dirInputs) {
println("==dex directory: ${directoryInput.file}")
}
base.transform(transformInvocation)
}
}
project.afterEvaluate {
android.applicationVariants.all { variant ->
project.getGradle().getTaskGraph().addTaskExecutionGraphListener(new TaskExecutionGraphListener() {
@Override
public void graphPopulated(TaskExecutionGraph taskGraph) {
for (Task task : taskGraph.getAllTasks()) {
if (task.getProject().equals(project) && task instanceof TransformTask && task.name.toLowerCase().contains(variant.name.toLowerCase())) {
Transform transform = ((TransformTask) task).getTransform()
//如果开启了multidex有这个任务
if ((((transform instanceof JarMergingTransform)) && !(transform instanceof MyJarMergingTransform))) {
project.logger.error("==fastdex find jarmerging transform. transform class: " + task.transform.getClass() + " . task name: " + task.name)
MyJarMergingTransform jarMergingTransform = new MyJarMergingTransform(transform)
Field field = getFieldByName(task.getClass(),'transform')
field.setAccessible(true)
field.set(task,jarMergingTransform)
}
if ((((transform instanceof DexTransform)) && !(transform instanceof MyDexTransform))) {
project.logger.error("==fastdex find dex transform. transform class: " + task.transform.getClass() + " . task name: " + task.name)
//代理DexTransform,实现自定义的转换
MyDexTransform fastdexTransform = new MyDexTransform(transform)
Field field = getFieldByName(task.getClass(),'transform')
field.setAccessible(true)
field.set(task,fastdexTransform)
}
}
}
}
});
}
}把上面的代码放进app/build.gradle执行./gradlew assembleDebug
开启multidex(multiDexEnabled true)时的日志输出**
:app:mergeDebugAssets :app:transformClassesWithJarMergingForDebug ==jarmerge jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/libs/exist-in-app-libs-2.1.2.jar ==jarmerge jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/com.android.support/multidex/1.0.1/jars/classes.jar ==jarmerge jar : /Users/tong/Applications/android-sdk-macosx/extras/android/m2repository/com/android/support/support-annotations/23.3.0/support-annotations-23.3.0.jar ==jarmerge jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/com.jakewharton/butterknife/8.0.1/jars/classes.jar ==jarmerge jar : /Users/tong/.gradle/caches/modules-2/files-2.1/com.jakewharton/butterknife-annotations/8.0.1/345b89f45d02d8b09400b472fab7b7e38f4ede1f/butterknife-annotations-8.0.1.jar ==jarmerge jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar ==jarmerge jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/unspecified/jars/classes.jar ==jarmerge directory: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/classes/debug ==combinedJar exists false /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/combined.jar ==combinedJar exists true /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/combined.jar :app:transformClassesWithMultidexlistForDebug :app:transformClassesWithDexForDebug ===dex jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/combined.jar :app:mergeDebugJniLibFolders关闭multidex(multiDexEnabled false)时的日志输出**
:app:mergeDebugAssets
:app:transformClassesWithDexForDebug
===dex jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/libs/exist-in-app-libs-2.1.2.jar
===dex jar : /Users/tong/Applications/android-sdk-macosx/extras/android/m2repository/com/android/support/support-annotations/23.3.0/support-annotations-23.3.0.jar
===dex jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/com.jakewharton/butterknife/8.0.1/jars/classes.jar
===dex jar : /Users/tong/.gradle/caches/modules-2/files-2.1/com.jakewharton/butterknife-annotations/8.0.1/345b89f45d02d8b09400b472fab7b7e38f4ede1f/butterknife-annotations-8.0.1.jar
===dex jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar
===dex jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/unspecified/jars/classes.jar
===dex directory: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/classes/debug
:app:mergeDebugJniLibFolders从上面的日志输出可以看出，只需要在下图红色箭头指的地方做patch.jar的生成就可以了
另外之前全量打包做asm code注入的时候是遍历combined.jar如果entry对应的是项目代码就做注入，反之认为是第三方库跳过注入(第三方库不在修复之列，为了节省注入花费的时间所以忽略)；现在拦截了jarmerge任务，直接扫描所有的DirectoryInput对应目录下的所有class做注入就行了，效率会比之前的做法有很大提升
###二、对直接依赖的library工程做支持
以下面这个工程为例
github.com/typ0520/fas…
这个工程包含三个子工程
- app (android application project)
- aarlib (android library project)
- javalib (java project)
app工程依赖aarlib和javalib
dependencies {
compile fileTree(dir: 'libs', include: ['*.jar'])
compile 'com.jakewharton:butterknife:8.0.1'
apt 'com.jakewharton:butterknife-compiler:8.0.1'
compile project(':javalib')
compile project(':aarlib')
compile project(':libgroup:javalib2')
}对于使用compile project(':xxx')这种方式依赖的工程，在apk的构建过程中是当做jar处理的，从拦截transformClassesWithJarMergingForDebug任务时的日志输出可以证明
===dex jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar
===dex jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/unspecified/jars/classes.jar之前修改了library工程的代码补丁打包之所以没有生效，就是因为补丁打包时只从DirectoryInput中抽离变化的class而没有对library工程的输出jar做抽离，这个时候就需要知道JarInput中那些属于library工程那些属于第三方库。最直接的方式是通过文件系统路径区分，但是这样需要排除掉library工程中直接放在libs目录下依赖的jar比如
==jarmerge jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/libs/exist-in-app-libs-2.1.2.jar其次如果依赖的library目录和app工程不在同一个目录下还要做容错的判断
==jarmerge jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/libgroup/javalib2/build/libs/javalib2.jar最终放弃了判断路径的方式，转而去找android gradle的api拿到每个library工程的输出jar路径，翻阅了源码发现2.0.0、2.2.0、2.3.0对应的api都不一样，通过判断版本的方式可以解决，代码如下
public class LibDependency {
public final File jarFile;
public final Project dependencyProject;
public final boolean androidLibrary;
LibDependency(File jarFile, Project dependencyProject, boolean androidLibrary) {
this.jarFile = jarFile
this.dependencyProject = dependencyProject
this.androidLibrary = androidLibrary
}
boolean equals(o) {
if (this.is(o)) return true
if (getClass() != o.class) return false
LibDependency that = (LibDependency) o
if (jarFile != that.jarFile) return false
return true
}
int hashCode() {
return (jarFile != null ? jarFile.hashCode() : 0)
}
@Override
public String toString() {
return "LibDependency{" +
"jarFile=" + jarFile +
", dependencyProject=" + dependencyProject +
", androidLibrary=" + androidLibrary +
'}';
}
private static Project getProjectByPath(Collection<Project> allprojects, String path) {
return allprojects.find { it.path.equals(path) }
}
/**
* 扫描依赖(<= 2.3.0)
* @param library
* @param libraryDependencies
*/
private static final void scanDependency(com.android.builder.model.Library library,Set<com.android.builder.model.Library> libraryDependencies) {
if (library == null) {
return
}
if (library.getProject() == null) {
return
}
if (libraryDependencies.contains(library)) {
return
}
libraryDependencies.add(library)
if (library instanceof com.android.builder.model.AndroidLibrary) {
List<com.android.builder.model.Library> libraryList = library.getJavaDependencies()
if (libraryList != null) {
for (com.android.builder.model.Library item : libraryList) {
scanDependency(item,libraryDependencies)
}
}
libraryList = library.getLibraryDependencies()
if (libraryList != null) {
for (com.android.builder.model.Library item : libraryList) {
scanDependency(item,libraryDependencies)
}
}
}
else if (library instanceof com.android.builder.model.JavaLibrary) {
List<com.android.builder.model.Library> libraryList = library.getDependencies()
if (libraryList != null) {
for (com.android.builder.model.Library item : libraryList) {
scanDependency(item,libraryDependencies)
}
}
}
}
/**
* 扫描依赖(2.0.0 <= android-build-version <= 2.2.0)
* @param library
* @param libraryDependencies
*/
private static final void scanDependency_2_0_0(Object library,Set<com.android.builder.model.Library> libraryDependencies) {
if (library == null) {
return
}
if (library.getProject() == null){
return
}
if (libraryDependencies.contains(library)) {
return
}
libraryDependencies.add(library)
if (library instanceof com.android.builder.model.AndroidLibrary) {
List<com.android.builder.model.Library> libraryList = library.getLibraryDependencies()
if (libraryList != null) {
for (com.android.builder.model.Library item : libraryList) {
scanDependency_2_0_0(item,libraryDependencies)
}
}
}
}
/**
* 解析项目的工程依赖 compile project('xxx')
* @param project
* @return
*/
public static final Set<LibDependency> resolveProjectDependency(Project project, ApplicationVariant apkVariant) {
Set<LibDependency> libraryDependencySet = new HashSet<>()
VariantDependencies variantDeps = apkVariant.getVariantData().getVariantDependency();
if (Version.ANDROID_GRADLE_PLUGIN_VERSION.compareTo("2.3.0") >= 0) {
def allDependencies = new HashSet<>()
allDependencies.addAll(variantDeps.getCompileDependencies().getAllJavaDependencies())
allDependencies.addAll(variantDeps.getCompileDependencies().getAllAndroidDependencies())
for (Object dependency : allDependencies) {
if (dependency.projectPath != null) {
def dependencyProject = getProjectByPath(project.rootProject.allprojects,dependency.projectPath);
boolean androidLibrary = dependency.getClass().getName().equals("com.android.builder.dependency.level2.AndroidDependency");
File jarFile = null
if (androidLibrary) {
jarFile = dependency.getJarFile()
}
else {
jarFile = dependency.getArtifactFile()
}
LibDependency libraryDependency = new LibDependency(jarFile,dependencyProject,androidLibrary)
libraryDependencySet.add(libraryDependency)
}
}
}
else if (Version.ANDROID_GRADLE_PLUGIN_VERSION.compareTo("2.2.0") >= 0) {
Set<Library> librarySet = new HashSet<>()
for (Object jarLibrary : variantDeps.getCompileDependencies().getJarDependencies()) {
scanDependency(jarLibrary,librarySet)
}
for (Object androidLibrary : variantDeps.getCompileDependencies().getAndroidDependencies()) {
scanDependency(androidLibrary,librarySet)
}
for (com.android.builder.model.Library library : librarySet) {
boolean isAndroidLibrary = (library instanceof AndroidLibrary);
File jarFile = null
def dependencyProject = getProjectByPath(project.rootProject.allprojects,library.getProject());
if (isAndroidLibrary) {
com.android.builder.dependency.LibraryDependency androidLibrary = library;
jarFile = androidLibrary.getJarFile()
}
else {
jarFile = library.getJarFile();
}
LibDependency libraryDependency = new LibDependency(jarFile,dependencyProject,isAndroidLibrary)
libraryDependencySet.add(libraryDependency)
}
}
else {
Set librarySet = new HashSet<>()
for (Object jarLibrary : variantDeps.getJarDependencies()) {
if (jarLibrary.getProjectPath() != null) {
librarySet.add(jarLibrary)
}
//scanDependency_2_0_0(jarLibrary,librarySet)
}
for (Object androidLibrary : variantDeps.getAndroidDependencies()) {
scanDependency_2_0_0(androidLibrary,librarySet)
}
for (Object library : librarySet) {
boolean isAndroidLibrary = (library instanceof AndroidLibrary);
File jarFile = null
def projectPath = (library instanceof com.android.builder.dependency.JarDependency) ? library.getProjectPath() : library.getProject()
def dependencyProject = getProjectByPath(project.rootProject.allprojects,projectPath);
if (isAndroidLibrary) {
com.android.builder.dependency.LibraryDependency androidLibrary = library;
jarFile = androidLibrary.getJarFile()
}
else {
jarFile = library.getJarFile();
}
LibDependency libraryDependency = new LibDependency(jarFile,dependencyProject,isAndroidLibrary)
libraryDependencySet.add(libraryDependency)
}
}
return libraryDependencySet
}
}把上面的这段代码，和下面的代码都放进build.gradle中
project.afterEvaluate {
android.applicationVariants.all { variant ->
def variantName = variant.name.capitalize()
if ("Debug".equals(variantName)) {
LibDependency.resolveProjectDependency(project,variant).each {
println("==androidLibrary: " + it.androidLibrary + " ,jarFile: " + it.jarFile)
}
}
}
}
task resolveProjectDependency<< {
}执行./gradlew resolveProjectDependency 可以得到以下输出
==androidLibrary: true ,jarFile: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/unspecified/jars/classes.jar
==androidLibrary: false ,jarFile: /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar
==androidLibrary: false ,jarFile: /Users/tong/Projects/fastdex-test-project/jarmerging-test/libgroup/javalib2/build/libs/javalib2.jar有了这些路径我们就可以在遍历JarInput是进行匹配，只要在这个路径列表中的都属于library工程的输出jar，用到这块有两处地方
全量打包时注入library输出jar ClassInject.groovy
public static void injectJarInputFiles(FastdexVariant fastdexVariant, HashSet<File> jarInputFiles) { def project = fastdexVariant.project long start = System.currentTimeMillis() Set<LibDependency> libraryDependencies = fastdexVariant.libraryDependencies List<File> projectJarFiles = new ArrayList<>() //获取所有依赖工程的输出jar (compile project(':xxx')) for (LibDependency dependency : libraryDependencies) { projectJarFiles.add(dependency.jarFile) } if (fastdexVariant.configuration.debug) { project.logger.error("==fastdex projectJarFiles : ${projectJarFiles}") } for (File file : jarInputFiles) { if (!projectJarFiles.contains(file)) { continue } project.logger.error("==fastdex ==inject jar: ${file}") ClassInject.injectJar(fastdexVariant,file,file) } long end = System.currentTimeMillis() project.logger.error("==fastdex inject complete jar-size: ${projectJarFiles.size()} , use: ${end - start}ms") }- 补丁打包时从library工程输出jar中抽离变化的class JarOperation.groovy
public static void generatePatchJar(FastdexVariant fastdexVariant, TransformInvocation transformInvocation, File patchJar) throws IOException {
Set<LibDependency> libraryDependencies = fastdexVariant.libraryDependencies
Map<String,String> jarAndProjectPathMap = new HashMap<>()
List<File> projectJarFiles = new ArrayList<>()
//获取所有依赖工程的输出jar (compile project(':xxx'))
for (LibDependency dependency : libraryDependencies) {
projectJarFiles.add(dependency.jarFile)
jarAndProjectPathMap.put(dependency.jarFile.absolutePath,dependency.dependencyProject.projectDir.absolutePath)
}
//所有的class目录
Set<File> directoryInputFiles = new HashSet<>();
//所有输入的jar
Set<File> jarInputFiles = new HashSet<>();
for (TransformInput input : transformInvocation.getInputs()) {
Collection<DirectoryInput> directoryInputs = input.getDirectoryInputs()
if (directoryInputs != null) {
for (DirectoryInput directoryInput : directoryInputs) {
directoryInputFiles.add(directoryInput.getFile())
}
}
if (!projectJarFiles.isEmpty()) {
Collection<JarInput> jarInputs = input.getJarInputs()
if (jarInputs != null) {
for (JarInput jarInput : jarInputs) {
if (projectJarFiles.contains(jarInput.getFile())) {
jarInputFiles.add(jarInput.getFile())
}
}
}
}
}
def project = fastdexVariant.project
File tempDir = new File(fastdexVariant.buildDir,"temp")
FileUtils.deleteDir(tempDir)
FileUtils.ensumeDir(tempDir)
Set<File> moudleDirectoryInputFiles = new HashSet<>()
DiffResultSet diffResultSet = fastdexVariant.projectSnapshoot.diffResultSet
for (File file : jarInputFiles) {
String projectPath = jarAndProjectPathMap.get(file.absolutePath)
List<String> patterns = diffResultSet.addOrModifiedClassesMap.get(projectPath)
if (patterns != null && !patterns.isEmpty()) {
File classesDir = new File(tempDir,"${file.name}-${System.currentTimeMillis()}")
project.copy {
from project.zipTree(file)
for (String pattern : patterns) {
include pattern
}
into classesDir
}
moudleDirectoryInputFiles.add(classesDir)
directoryInputFiles.add(classesDir)
}
}
JarOperation.generatePatchJar(fastdexVariant,directoryInputFiles,moudleDirectoryInputFiles,patchJar);
}三、 全新的快照对比模块
fastdex目前需要对比的地方有三处
- 全量打包时对当前依赖的库做快照，补丁打包时对比是否发生变化
- 检测app工程和所有依赖的android library工程中所有AndroidManifest.xml是上次打包相比是否发生变化(免安装模块要用到manifest文件发生变化，必须要重新安装app)
- 全量打包时对所有的java文件和kotlin文件做快照，补丁打包时对比那些源文件发生变化
以第一种场景为例，说下对比的原理，全量打包时生成一个文本文件把当前的依赖写进去以换行符分割
/Users/tong/Projects/fastdex/sample/app/libs/fm-sdk-2.1.2.jar
/Users/tong/Projects/fastdex/sample/javalib/build/libs/javalib.jar补丁打包时先把这个文本文件读取到ArrayList中，然后把当前的依赖列表页放进ArrayList中
，通过以下操作可以获取新增项、删除项，只要发现有删除项和新增项就认为依赖发生了变化
ArrayList<String> old = new ArrayList<>();
old.add("/Users/tong/Projects/fastdex/sample/app/libs/fm-sdk-2.1.2.jar");
old.add("/Users/tong/Projects/fastdex/sample/javalib/build/libs/javalib.jar");
ArrayList<String> now = new ArrayList<>();
now.add("/Users/tong/Projects/fastdex/sample/app/libs/fm-sdk-2.1.2.jar");
now.add("/Users/tong/Projects/fastdex/sample/javalib/build/libs/new.jar");
//获取删除项
Set<String> deletedNodes = new HashSet<>();
deletedNodes.addAll(old);
deletedNodes.removeAll(now);
//新增项
Set<String> increasedNodes = new HashSet<>();
increasedNodes.addAll(now);
//如果不用ArrayList套一层有时候会发生移除不掉的情况 why?
increasedNodes.removeAll(old);
//需要检测是否变化的列表
Set<String> needDiffNodes = new HashSet<>();
needDiffNodes.addAll(now);
needDiffNodes.addAll(old);
needDiffNodes.removeAll(deletedNodes);
needDiffNodes.removeAll(increasedNodes);注: 文本的对比不存在更新，但是文件对比是存在这种情况的
所有的快照对比都是基于上面这段代码的抽象，具体可以参考这里
github.com/typ0520/fas…
四、 dex merge
全量打包以后，按照正常的开发节奏发生变化的源文件会越来越多，相应的参与dex生成的class也会越来越多，这样会导致补丁打包速度越来越慢。
解决这个问题比较简单的方式是把每次生成的patch.dex放进全量打包时的dex缓存中(必须排在之前的dex前面)，并且更新下源代码快照，这样做有两个坏处
- 1、每次补丁打包时都必须对class文件做注入，为了解决上篇文章中提到的pre-verify错误
- 2、每次补丁打包都需要缓存patch.dex，会导致下面这个目录的dex越来越多
app/build/intermediates/transforms/dex/debug/folders/1000/1f/main
解决第二个问题的方案是把patch.dex中的class合并到缓存的dex中，这样就不需要保留所有的patch.dex了,一个比较棘手的问题是如果缓存的dex的方法数已经有65535个了，在往里面加新增的class，肯定会爆掉了，最终fastdex选择的方案是第一次触发dex merge时直接把patch.dex扔进缓存(merged-patch.dex)，以后在触发dex merge时就拿patch.dex和merged-patch.dex做合并(这样做也存在潜在的问题，如果变化的class特别多也有可能导致合并dex时出现65535的错误)
解决第一个问题是加了一个可配置选项，默认是3个以上的源文件发生变化时触发merge，这样即不用每次都做代码注入和merge操作，也能在源文件变化多的时候恢复状态
这个dex merge工具是从freeline里找到的，感兴趣的话可以把下载下来试着调用下
github.com/typ0520/fas…
java -jar fastdex-dex-merge.jar output.dex patch.dex merged-patch.dex
五、支持注解生成器
在现阶段的Android开发中，注解越来越流行起来，比如ButterKnife，EventBus等等都选择使用注解来配置。按照处理时期，注解又分为两种类型，一种是运行时注解，另一种是编译时注解，运行时注解由于性能问题被一些人所诟病。编译时注解的核心依赖APT(Annotation Processing Tools)实现，原理是在某些代码元素上（如类型、函数、字段等）添加注解，在编译时编译器会检查AbstractProcessor的子类，并且调用该类型的process函数，然后将添加了注解的所有元素都传递到process函数中，使得开发人员可以在编译期进行相应的处理，例如，根据注解生成新的Java类，这也就是ButterKnife，EventBus等开源库的基本原理。Java API已经提供了扫描源码并解析注解的框架，你可以继承AbstractProcessor类来提供实现自己的解析注解逻辑
-- 引用自blog.csdn.net/industrious…
虽然能提高运行期的效率但也给开发带来一些麻烦
AbstractProcessor这些类只有在编译期才会用到，运行期是用不到的，但是如果通过compile方式依赖的包，会把这些类都打包进dex中
以这个项目为例(建议把代码拉下来，后面好几个地方会用到)
github.com/typ0520/fas…app中依赖了butterknife7.0.1
dependencies { compile 'com.jakewharton:butterknife:7.0.1' }butterknife7.0.1中的注解生成器叫ButterKnifeProcessor
执行./gradlew app:assembleDebug
从上图可以看出ButterKnifeProcessor.class被打包进dex中了
- 为了避免上述的这种情况，可以通过annotationProcessor的方式引入，butterknife8.8.1把ButterKnifeProcessor相关的独立成了butterknife-compiler模块，butterknife模块只保留运行期需要使用的代码
app2中依赖了butterknife8.8.1
apply plugin: 'com.jakewharton.butterknife'
dependencies {
compile 'com.jakewharton:butterknife:8.8.1'
annotationProcessor 'com.jakewharton:butterknife-compiler:8.8.1'
}执行./gradlew app2:assembleDebug
从上图可以看出butterknife.compiler包下所有的代码都没有被打包进dex。虽然通过annotationProcessor依赖AbstractProcessor相关代码有上述好处，但是会造成增量编译不可用，简单地说就是正常的项目执行compileDebugJavaWithJavac任务调用javac的时候只会编译内容发生变化的java源文件，如果使用了annotationProcessor每次执行compileDebugJavaWithJavac任务都会把项目中所有的java文件都参与编译，想象一下如果项目中有成百上千个java文件编译起来那酸爽。我们可以做个测试，还是使用这个项目
github.com/typ0520/fas…
annotation-generators包含三个子项目
app依赖7.0.1
compile 'com.jakewharton:butterknife:7.0.1'app2依赖8.8.1
dependencies { compile 'com.jakewharton:butterknife:8.8.1' annotationProcessor 'com.jakewharton:butterknife-compiler:8.8.1' }- app3不包含任何AbstractProcessor
这三个子工程都包含两个java文件
com/github/typ0520/annotation_generators/HAHA.java
com/github/typ0520/annotation_generators/MainActivity.java
测试的思路是先检查MainActivity.class文件的更新时间，然后修改HAHA.java执行编译，最后在检查MainActivity.class文件的更新时间是否和编译之前的一致，如果一致说明增量编译可用，反之不可用
通过increment_compile_test.sh这个shell脚本来做测试(使用windows的同学可以手动做测试V_V)
#!/bin/bash
sh gradlew assembleDebug
test_increment_compile() {
echo "========测试${1}是否支持增量, ${2}"
str=$(stat -x ${1}/build/intermediates/classes/debug/com/github/typ0520/annotation_generators/MainActivity.class | grep 'Modify')
echo $str
echo 'package com.github.typ0520.annotation_generators;' > ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
echo 'public class HAHA {' >> ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
echo " public long millis = $(date +%s);" >> ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
echo '}' >> ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
sh gradlew ${1}:assembleDebug > /dev/null
str2=$(stat -x ${1}/build/intermediates/classes/debug/com/github/typ0520/annotation_generators/MainActivity.class | grep 'Modify')
echo $str2
echo ' '
if [ "$str" == "$str2" ];then
echo "${1}只修改HAHA.java，MainActivity.class没有发生变化"
else
echo "${1}只修改HAHA.java，MainActivity.class发生变化"
fi
}
test_increment_compile app "compile 'com.jakewharton:butterknife:7.0.1'"
test_increment_compile app2 "annotationProcessor 'com.jakewharton:butterknife-compiler:8.8.1'"
test_increment_compile app3 "没有用任何AbstractProcessor"执行sh increment_compile_test.sh
日志的输出可以证明上面所描述的
既然原生不支持那么我们就在自定义的java compile任务中来做这个事情，通过之前的快照模块可以对比出那些java源文件发生了变化，那么就可以自己拼接javac命令参数然后调用仅编译变化的java文件
demo中写了一个编译任务方便大家理解这些参数都是怎么拼接的，代码太多了这里就不贴出来了
github.com/typ0520/fas…
github.com/typ0520/fas…
可以调用./gradlew mycompile1 或者 ./gradlew mycompile2看下最终拼接出来的命令
fastdex中对应模块的代码在
github.com/typ0520/fas…
六、填过的坑
解决的bug这块本来是不准备说的，因为这块最有价值的东西不是解决问题本身，而是怎么发现和重现问题的，这块确实不太好描述V_V，应简友的要求还是挑了一些相对比较有营养的问题说下，主要还是说解决的方法，至于问题是怎样定位和重现的只能尽力描述了。
1、issues#2
导致这个问题的原因是项目中原来的YtxApplication类被替换成了FastdexApplication，当在activity中执行类似于下面的操作时就会报ClassCastException
MyApplication app = (MyApplication) getApplication();解决的方法是在instant-run的源码里找到的，运行期把android api里所有引用Application的地方把实例替换掉
public static void monkeyPatchApplication( Context context,
Application bootstrap,
Application realApplication,
String externalResourceFile) {
try {
// Find the ActivityThread instance for the current thread
Class<?> activityThread = Class.forName("android.app.ActivityThread");
Object currentActivityThread = getActivityThread(context, activityThread);
// Find the mInitialApplication field of the ActivityThread to the real application
Field mInitialApplication = activityThread.getDeclaredField("mInitialApplication");
mInitialApplication.setAccessible(true);
Application initialApplication = (Application) mInitialApplication.get(currentActivityThread);
if (realApplication != null && initialApplication == bootstrap) {
mInitialApplication.set(currentActivityThread, realApplication);
}
// Replace all instance of the stub application in ActivityThread#mAllApplications with the
// real one
if (realApplication != null) {
Field mAllApplications = activityThread.getDeclaredField("mAllApplications");
mAllApplications.setAccessible(true);
List<Application> allApplications = (List<Application>) mAllApplications
.get(currentActivityThread);
for (int i = 0; i < allApplications.size(); i++) {
if (allApplications.get(i) == bootstrap) {
allApplications.set(i, realApplication);
}
}
}
// Figure out how loaded APKs are stored.
// API version 8 has PackageInfo, 10 has LoadedApk. 9, I don't know.
Class<?> loadedApkClass;
try {
loadedApkClass = Class.forName("android.app.LoadedApk");
} catch (ClassNotFoundException e) {
loadedApkClass = Class.forName("android.app.ActivityThread$PackageInfo");
}
Field mApplication = loadedApkClass.getDeclaredField("mApplication");
mApplication.setAccessible(true);
Field mResDir = loadedApkClass.getDeclaredField("mResDir");
mResDir.setAccessible(true);
Field mLoadedApk = null;
try {
mLoadedApk = Application.class.getDeclaredField("mLoadedApk");
} catch (NoSuchFieldException e) {
// According to testing, it's okay to ignore this.
}
for (String fieldName : new String[]{"mPackages", "mResourcePackages"}) {
Field field = activityThread.getDeclaredField(fieldName);
field.setAccessible(true);
Object value = field.get(currentActivityThread);
for (Map.Entry<String, WeakReference<?>> entry :
((Map<String, WeakReference<?>>) value).entrySet()) {
Object loadedApk = entry.getValue().get();
if (loadedApk == null) {
continue;
}
if (mApplication.get(loadedApk) == bootstrap) {
if (realApplication != null) {
mApplication.set(loadedApk, realApplication);
}
if (externalResourceFile != null) {
mResDir.set(loadedApk, externalResourceFile);
}
if (realApplication != null && mLoadedApk != null) {
mLoadedApk.set(realApplication, loadedApk);
}
}
}
}
} catch (Throwable e) {
throw new IllegalStateException(e);
}
}具体可以参考测试工程的代码
github.com/typ0520/fas…
2、issues#6
github.com/typ0520/fas…
@YuJunKui1995
这个错误的表现是如果项目里包含baidumapapi_v2_0_0.jar，正常打包是没问题的，只要使用fastdex就会报下面这个错误
Error:Error converting bytecode to dex:
Cause: PARSE ERROR:
class name (com/baidu/platform/comapi/map/a) does not match path (com/baidu/platform/comapi/map/A.class)
...while parsing com/baidu/platform/comapi/map/A.class经过分析使用fastdex打包时会有解压jar然后在压缩的操作，使用下面这段代码做测试
github.com/typ0520/fas…
task gen_dex2<< {
File tempDir = project.file('temp')
tempDir.deleteDir()
project.copy {
from project.zipTree(project.file('baidumapapi_v2_0_0.jar'))
into tempDir
}
File baidumapJar = project.file('temp/baidu.jar')
project.ant.zip(baseDir: tempDir, destFile: baidumapJar)
ProcessBuilder processBuilder = new ProcessBuilder('dx','--dex',"--output=" + project.file('baidu.dex').absolutePath, baidumapJar.absolutePath)
def process = processBuilder.start()
InputStream is = process.getInputStream()
BufferedReader reader = new BufferedReader(new InputStreamReader(is))
String line = null
while ((line = reader.readLine()) != null) {
println(line)
}
reader.close()
int status = process.waitFor()
reader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
reader.close();
try {
process.destroy()
} catch (Throwable e) {
}
}执行./gradlew gen_dex2
果不其然重现了这个问题，查了资料发现mac和windows一样文件系统大小写不敏感，如果jar包里有A.class，解压后有可能就变成a.class了，所以生成dex的时候会报不匹配的错误(类似的问题也会影响git，之前就发现改了一个文件名字的大小写git检测不到变化，当时没有细想这个问题，现在看来也是同样的问题)。知道问题是怎么发生的那么解决就简单了，既然在文件系统操作jar会有问题，那就放在内存做，对应java的api就是ZipOutputStream和ZipInputStream。
对于mac下文件系统大小写不敏感可以在终端执行下面这段命令，体会下输出
echo 'a' > a.txt;echo 'A' > A.txt;cat a.txt;cat A.txt
3、issues#8
github.com/typ0520/fas…
@dongzy
Error:Execution failed for task ':app:tinkerSupportProcess_360DebugManifest'.
java.io.FileNotFoundException: E:\newkp\kuaipiandroid\NewKp\app\src\main\java\com\dx168\fastdex\runtime\FastdexApplication.java (系统找不到指定的路径。)出现这个错误的原因是@dongzy的项目中使用了tinkerpatch的一键接入，tinkerpatch的gradle插件也有Application替换的功能，必须保证fastdexProcess{variantName}Manifest任务在最后执行才行
FastdexManifestTask manifestTask = project.tasks.create("fastdexProcess${variantName}Manifest", FastdexManifestTask)
manifestTask.fastdexVariant = fastdexVariant
manifestTask.mustRunAfter variantOutput.processManifest
variantOutput.processResources.dependsOn manifestTask
//fix issue#8
def tinkerPatchManifestTask = null
try {
tinkerPatchManifestTask = project.tasks.getByName("tinkerpatchSupportProcess${variantName}Manifest")
} catch (Throwable e) {}
if (tinkerPatchManifestTask != null) {
manifestTask.mustRunAfter tinkerPatchManifestTask
}4、issues#xxoo
这段不是解决问题的， 忍不住吐槽下这哥们，觉得浪费了他的时间，上来就是“亲测无软用，建议大家不要用什么什么的”，搞的我非常郁闷，果断用知乎上的一篇文章回应了过去
zhuanlan.zhihu.com/p/25768464
后来经过沟通发现这哥们在一个正常打包3秒的项目上做的测试，我也是无语了
说实在的真的希望大家对开源项目多一点尊重，觉得对自己有帮助就用。如果觉得不好，可以选择提建议，也可以选择默默离开，如果有时间有能力可以参与进来优化，解决自己工作问题的同时也服务了大家。在这个快节奏的社会大家的时间都宝贵，你觉得测试一下浪费了时间就开始吐槽，有没有想到开源项目的作者牺牲了大量的个人时间在解决一个一个问题、为了解决新功能的技术点一个一个方案的做测试做对比呢？
注: 如果项目的dex生成小于10秒，建议不要使用fastdex，几乎是感知不到效果的。
gradle编译速度优化建议
- 不要使用类似于com.android.tools.build:gradle:2.+的动态依赖，不然每次启动编译都需要请求maven server对比当前是否是新版本
少直接使用compile project(':xxx')依赖library工程，如果module比较多编译开始的时候需要遍历module根据build.gradle配置项目，另外每个library工程都包含大量的任务每个任务都需要对比输入和输出，这些小任务叠加到一块的时间消耗也是很可观的。 建议把library工程打成aar包丢到公司的maven服务器上，别和我说开发阶段library经常改直接依赖方便，每次修改打包到maven服务器上没有那么麻烦。我们团队的项目都是只有一个干净的application工程，library代码全丢进了maven服务器，dex方法数在12w左右，使用fastdex修改了几个java文件能稳定在8秒左右完成打包、发送补丁和app重启
任何情况都别在library工程里使用flavor
具体可以参考@依然范特稀西写的这篇文章
Android 优化APP 构建速度的17条建议
5、issues#17
github.com/typ0520/fas…
@junchenChow
[ant:javac] : warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[ant:javac] /Users/zhoujunchen/as/xx/app/build/fastdex/DevelopDebug/custom-combind/com/xx/xx/xx/xx/CourseDetailActivity.java:229: 错误: -source 1.7 中不支持 lambda 表达式
[ant:javac] wrapperControlsView.postDelayed(() -> wrapperControlsView.initiativeRefresh(), 500L);
[ant:javac] ^
[ant:javac] (请使用 -source 8 或更高版本以启用 lambda 表达式)
[ant:javac] /Users/zhoujunchen/as/android-donguo/app/build/fastdex/DevelopDebug/custom-combind/com/xx/xx/xx/xx/CourseDetailActivity.java:489: 错误: -source 1.7 中不支持方法引用
[ant:javac] .subscribe(conf -> ShareHelper.share(this, conf), Throwable::printStackTrace);
[ant:javac] ^
[ant:javac] (请使用 -source 8 或更高版本以启用方法引用)
[ant:javac] 2 个错误
:app:fastdexCustomCompileDevelopDebugJavaWithJavac FAILED
有什么选项没开启么 不支持lambda？这个错误的原因是之前自定义的编译任务写死了使用1.7去编译，查阅gradle-retrolambda的源码找到了这些代码
github.com/evant/gradl…
https://github.com/evant/gradle-retrolambda/blob/master/gradle-retrolambda/src/main/groovy/me/tatarka/RetrolambdaPluginAndroid.groovy
private static configureCompileJavaTask(Project project, BaseVariant variant, RetrolambdaTransform transform) {
variant.javaCompile.doFirst {
def retrolambda = project.extensions.getByType(RetrolambdaExtension)
def rt = "$retrolambda.jdk/jre/lib/rt.jar"
variant.javaCompile.classpath = variant.javaCompile.classpath + project.files(rt)
ensureCompileOnJava8(retrolambda, variant.javaCompile)
}
transform.putVariant(variant)
}
private static ensureCompileOnJava8(RetrolambdaExtension retrolambda, JavaCompile javaCompile) {
javaCompile.sourceCompatibility = "1.8"
javaCompile.targetCompatibility = "1.8"
if (!retrolambda.onJava8) {
// Set JDK 8 for the compiler task
def javac = "${retrolambda.tryGetJdk()}/bin/javac"
if (!checkIfExecutableExists(javac)) {
throw new ProjectConfigurationException("Cannot find executable: $javac", null)
}
javaCompile.options.fork = true
javaCompile.options.forkOptions.executable = javac
}
}从这些代码中我们可以得知以下信息
- 需要使用jdk1.8里的javac去编译
- sourceCompatibility和targetCompatibility必须设置成1.8
- classpath中需要添加1.8的rt.jar
有了这些信息就可以在自定义的编译任务做处理了
if (project.plugins.hasPlugin("me.tatarka.retrolambda")) {
def retrolambda = project.retrolambda
def rt = "${retrolambda.jdk}${File.separator}jre${File.separator}lib${File.separator}rt.jar"
classpath.add(rt)
executable = "${retrolambda.tryGetJdk()}${File.separator}bin${File.separator}javac"
if (Os.isFamily(Os.FAMILY_WINDOWS)) {
executable = "${executable}.exe"
}
}
List<String> cmdArgs = new ArrayList<>()
cmdArgs.add(executable)
cmdArgs.add("-encoding")
cmdArgs.add("UTF-8")
cmdArgs.add("-g")
cmdArgs.add("-target")
cmdArgs.add(javaCompile.targetCompatibility)
cmdArgs.add("-source")
cmdArgs.add(javaCompile.sourceCompatibility)
cmdArgs.add("-cp")
cmdArgs.add(joinClasspath(classpath))具体可以参考
github.com/typ0520/fas…
6、issues#24 #29 #35 #36
github.com/typ0520/fas…
@wsf5918 @ysnows @jianglei199212 @tianshaokai @Razhan
Caused by: java.lang.RuntimeException: ==fastdex jar input size is 117, expected is 1
at com.dx168.fastdex.build.transform.FastdexTransform.getCombinedJarFile(FastdexTransform.groovy:173)
at com.dx168.fastdex.build.transform.FastdexTransform$getCombinedJarFile.callCurrent(Unknown Source)
at com.dx168.fastdex.build.transform.FastdexTransform.transform(FastdexTransform.groovy:131)
at com.android.build.gradle.internal.pipeline.TransformTask$2.call(TransformTask.java:185)
at com.android.build.gradle.internal.pipeline.TransformTask$2.call(TransformTask.java:181)
at com.android.builder.profile.ThreadRecorder.record(ThreadRecorder.java:102)
at com.android.build.gradle.internal.pipeline.TransformTask.transform(TransformTask.java:176)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$IncrementalTaskAction.doExecute(DefaultTaskClassInfoStore.java:163)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$StandardTaskAction.execute(DefaultTaskClassInfoStore.java:134)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$StandardTaskAction.execute(DefaultTaskClassInfoStore.java:123)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:95)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:76)
... 78 more正常情况下开启multidex并且minSdkVersion < 21时会存在transformClassesWithJarMergingForDebug任务，用来合并所有的JarInput和DirectoryInput并且输出到build/intermediates/transforms/jarMerging/debug/jars/1/1f/combined.jar，而这个错误的表现是丢失了jarMerging任务，所以走到dexTransform时本来期望只有一个combined.jar，但是由于没有合并所以jar input的个数是117。当时由于一直无法重现这个问题，所以就采用加标示的手段解决的，具体是当走到FastdexJarMergingTransform并且执行完成以后就把executedJarMerge设置为true，走到dexTransform时判断如果开启了multidex并且executedJarMerge==false就说明是丢失了jarMerge任务，这个时候调用com.android.build.gradle.internal.transforms.JarMerger手动合并就可以解决了，具体可以参考GradleUtils的executeMerge方法
github.com/typ0520/fas…
后来在开发中发现了丢失jarMerging任务的规律如下
- com.android.tools.build:gradle的版本 >= 2.3.0
- build-type选择的是debug
- 只有点studio的run按钮打包时，命令行调用不行
- 点击run按钮打包时选择的设备是>=6.0的设备
看到这里第三点的表现是不是很奇怪，命令行和studio点击run最终都是走gradle的流程，既然表现不一样有可能是传的参数不一样，把下面这段代码放进build.gradle中
println "projectProperties: " + project.gradle.startParameter.projectProperties点击studio的run按钮选择一个6.0的设备
得到以下输出
projectProperties: [android.injected.build.density:560dpi, android.injected.build.api:23, android.injected.invoked.from.ide:true, android.injected.build.abi:x86]使用上面的这些参数一个一个做测试，发现是android.injected.build.api=23这个参数影响的，我们可以用这个测试项目做下测试
github.com/typ0520/fas…
执行./gradlew clean assembleDebug -Pandroid.injected.build.api=23
注: gradle传自定义的参数是以-P开头
从上面的日志输出中可以看出重现了丢失jarMerge任务，我们再来总结下重现这个问题的条件
- com.android.tools.build:gradle的版本 >= 2.3.0
- build-type选择的是debug
- 启动参数包含android.injected.build.api并且>=23
有了结论还没完，之所以2.3.0是这个行为是因为引入了build-cache机制，不合并是为了做jar级别的dex缓存，这样每次执行dex transform时只有第一次时第三方库才参与生成，为了提高效率也不会合并dex，如果项目比较大apk中可能是出现几十个甚至上百个dex
目前fastdex由于做了jar合并相当于把这个特性禁掉了，后面会考虑不再做合并使之能用dex缓存，这样全量打包时的速度应该可以提高很多，另外还可以引入到除了debug别的build-type打包中，还有设备必须大于6.0问题也可以处理下，理论上5.0以后系统就可以加载多个dex了，不知道为什么这个阈值设置的是6.0而不是5.0
==========================
本来想一鼓作气把这几个月做的功能和优化全在这篇一并说完的，写着写着简书提示字数快超限了，无奈只能分篇写了，下一篇主要讲免安装模块和idea插件的实现。快到中秋节了提前祝大家中秋快乐。未完待续，后会有期。。。。。。
如果你喜欢本文就来给我们star吧
github.com/typ0520/fas…
加快apk的构建速度，如何把编译时间从130秒降到17秒
加快apk的构建速度，如何把编译时间从130秒降到17秒(二)
参考的项目与文章
Instant Run
Tinker
Freeline
安卓App热补丁动态修复技术介绍
Android应用程序资源的编译和打包过程分析
关键字:
加快apk编译速度
加快app编译速度
加快android编译速度
加快android studio 编译速度
android 加快编译速度
android studio编译慢
android studio编译速度优化
android studio gradle 编译慢
本文出自typ0520的简书博客www.jianshu.com/p/53923d8f2…
