自定义parcel包在CM中部署自定义parcel包，将自己的项目制作成parcel包在CM中运行。复杂的制作流程。考虑

创建所需文件包

自定义parcel包想要在CM中部署，需要一个parcel包和一个CSD包。

如果自定义的parcel包是作为静态资源分发给各服务器使用的，是不需要CSD包的。CSD包作为parcel包的控制脚本存在。

cm_ext

为了帮助确保构建的parcel,csd是正确的，应该针对parcel和csd运行验证工具cm_ext。该工具提供了不同的操作粒度——它可以验证单个 json 文件、parcel 目录或 parcel 文件——因此可以在整个开发过程中使用它。

下载

$ mkdir -p ~/github/cloudera
# 手动下载地址 https://github.com/cloudera/cm_ext
$ git clone https://github.com/cloudera/cm_ext.git

安装

# 注意：如果是手动下载并直接解压到工作目录，目录名会是cm_ext-master。git clone目录名才会是cm_ext
$ cd cm_ext
$ mvn install
$ cd validator
$ java -jar target/validator.jar <arguments>

使用

# Validate a parcel.json file
$ java -jar /root/github/cloudera/cm_ext-master/validator/target/validator.jar -p parcel.json
# Validate an alternatives.json file
$ java -jar /root/github/cloudera/cm_ext-master/validator/target/validator.jar -a alternatives.json
# Validate an permissions.json file
$ java -jar /root/github/cloudera/cm_ext-master/validator/target/validator.jar -r permissions.json
# Validate a parcel directory
$ java -jar /root/github/cloudera/cm_ext-master/validator/target/validator.jar -d CDH-5.0.0-0.cdh5b2.p0.283/
# Validate a parcel file
$ java -jar /root/github/cloudera/cm_ext-master/validator/target/validator.jar -f CDH-5.0.0-0.cdh5b2.p0.283-el6.parcel

创建parcel包以及parcel.sha文件

创建parcel包

parcel包目录结构 parcel包有两部分组成：meta目录和项目目录

meta目录

meta目录是CM管理parcel的关键，如果没有meta，parcel包就只是一个普通的压缩包。 meta可以允许定义五个文件，分别为: parcel.json,The Parcel Defines Script，The alternatives.json file，The permissions.json file，release-notes.txt。正常情况下，只需要parcel.json和The Parcel Defines Script就可满足自定义parcel包的全部需求。
- parcel.json
  
  固定名称，为parcel包最重要的文件，其他文件都是可选的，该文件必须在meta目录中存在。用来描述parel包相关信息
- The Parcel Defines Script
  
  parcel定义脚本，通常通过export环境变量向csd暴露parcel文件位置。文件名可自定义，在parcel.json中通过scripts标签表明The Parcel Defines Script名称。
- alternatives.json
  
  github.com/cloudera/cm…
- permissions.json
  
  用于解压后，赋予指定文件指定权限。
  
  github.com/cloudera/cm…
- release-notes.txt
  
  github.com/cloudera/cm…
项目目录

项目目录，无格式要求，用户根据需要自定义。

#parcel包目录结构
--name：parcel包的名称
--version：parcel包的版本
[name]-[version]/
[name]-[version]/meta/
[name]-[version]/meta/parcel.json
[name]-[version]/meta/gplextras_env.sh
[name]-[version]/lib/
[name]-[version]/lib/ailab/
[name]-[version]/lib/ailab/lib/
[name]-[version]/lib/ailab/bin/
[name]-[version]/lib/ailab/conf/

创建parcel.json文件

# 注意：json文件是不允许有注释的，此处的注释只是为了解释各个标签的含义。
# 如果需要使用该json，请自行删除注释
{       # 必须为1
        "schema_version": 1,
        # parcel名
        "name": "AILAB",
        # parcel版本
        "version": "dev",
        # 当包裹被激活时，会创建parcel包的软连接
        "setActiveSymlink": true,
        # 列出与此parcel冲突的parcel。一次只能有一个冲突的parcel处于活动状态。
        "conflicts": "",
        # 向外暴露提供服务的Tag值，CSD 和parcel通过provides标签进行链接。
        "provides": ["ailab"],
        # 指定The Parcel Defines Script的名称（位置）。
        # 同provides标签以及csd的service.sdl文件的parcel标签配合才能发挥作用
        # 由于parcel和csd是相互独立的。但csd又是用来控制parcel的启动和关闭的。
        # 因此通常需要在The Parcel Defines Script中通过环境变量表明parcel包的位置。
        # 当csd包的service.sdl文件的parcel标签的value为该parcel.json文件的provides标签的value
        # 该csd包在执行start和stop命令时就可以读到该parcel.json文件的scripts标签指定的脚本所定义的环境变量
        "scripts": {
                # 即使parcel 不需要定义任何环境变量，也必须提供此脚本（脚本本身可以为空）
                "defines": "env.sh"
        },
        # 其余标签可有可无
        "packages": [{
                "name": "ailab",
                "version": "dev"
        }],
        "components": [{
                "name": "ailab",
                "version": "dev",
                "pkg_version": "dev"
        }],
        "users": {
                "datax": {
                        "longname": "AILAB",
                        "home": "/var/lib/ailab",
                        "shell": "/bin/bash",
                        "extra_groups": []
                }
        },
        "groups": ["ailab"]
}

创建The Parcel Defines Script文件

#!/bin/bash
# 其中$PARCEL_DIRNAME和$PARCELS_ROOT是CM提供的，以允许parcel 确定其所在位置
# $PARCELS_ROOT：这是文件系统上所有parcel所在的目录。（默认值：/opt/cloudera/parcels）
# $PARCEL_DIRNAME：位于$PARCELS_ROOT目录下的parcel的名称
# 即，parcel 目录的绝对路径是：$PARCELS_ROOT/$PARCEL_DIRNAME
AILAB_DIRNAME=${PARCEL_DIRNAME:-"AILAB"}

export AILAB_HOME=$PARCELS_ROOT/$AILAB_DIRNAME/lib/ailab
export DATAX_HOME=$AILAB_HOME/../../../DATAX/lib/datax

此脚本结合上面的parcel.json文件的provides和scripts标签，可向csd脚本提供“DATAX_HOME”和“AILAB_HOME”两个环境变量。

通过service.sdl文件的“parcel.requiredTags”标签的值指定

打包为`.parccel`文件并校验

# parcel名称格式固定
# parcel文件夹名为[name]-[version]，打包成parcel包后名称为[name]-[version]-[distro suffix].parcel 
# distro suffix:发行版后缀，表明将要部署parcel包的linux系统版本。 
# parcel有效的发行版本后缀：https://github.com/cloudera/cm_ext/wiki/Parcel-distro-suffixes
tar -zcvf AILAB-V1.2.1-el7.parcel AILAB-V1.2.1

# 官方下载的cm_ext程序，可以用来校验制作parcel包所需的所有文件。
# 网上其他制作parcel包的教程目录大多为cm_ext，而我下载后目录为cm_ext-master。大家根据自己下载的包决定路径。
java -jar /root/github/cloudera/cm_ext-master/validator/target/validator.jar -f AILAB-V1.2.1-el7.parcel

创建parcel.sha文件

创建好.parcel包后，为了使服务器校验parcel包以及确定包的完整性，需要生成该parcel包的.sha文件。命名为parcel包名.sha

# 新建.sha文件，将parcel包执行sha1sum命令后的值放入新建的.sha中
sha1sum AILAB-V1.2.1-el7.parcel | awk '{ print $1 }' > AILAB-V1.2.1-el7.parcel.sha

创建csd包

官方文档：github.com/cloudera/cm…

命名规则：[name]-[version].jar

目录结构

descriptor/service.sdl CSD配置文件
descriptor/service.mdl
scripts/ 脚本文件，一般用于运行parcel包中的程序
aux/ 辅助文件可以存在于该目录中，如果存在，则会与scripts目录一起发送到代理。如果有服务需要的静态配置文件（例如）topology.py且不是由 CSD 生成的，这会很有用。

service.sdl

标签示例：

更为详细信息请参照官方文档：github.com/cloudera/cm…

{
        // 版本号
        "version": "dev", 
	// CSD名称，同version标签组成CSD包名
        "name": "AILAB",
	// 服务名，用于在CM控制台显示
        "label": "Ailab",
	// CM控制台添加服务时对服务的的说明
        "description": "The ailab service",
        // 服务图标，改文件的相对路径，为服务规范，一般必须存在
        // 图片格式十分严格，必须为28*28的png
	"icon": "images/ailab.png",
	"runAs": {
		"user": "root",
		"group": "root"
	},
        // 指定该CSD需要获取哪些parcel提供的环境变量
	"parcel": {
                // CM服务器中必须存在该Tag的parcel
		"requiredTags": [
			"ailab"
		],
                // CM服务器中可以不存在该Tag的parcel
		"optionalTags": [
			"ailab"
		]
	},
        // 配置，可在CM控制台中填写自定义值 全服务可见
        // 用于生成配置文件，或向角色脚本提供环境变量
	"parameters": [
		{
			// 实际变量名
                        "name": "HADOOP_CONF_CORE_DIR",
                        // 别名，CM控制台中configName下方细体显示
			"label": "hadoop_conf_core_dir",
                        // 对配置变量的说明
			"description": "core-site.xml's location",
                        // CM控制台中的名称
			"configName": "HADOOP_CONF_CORE_DIR",
                        // 是否允许为null
			"required": true,
                        // 是不是必须配置的
                        configurableInWizard: true,
                        // 配置值的类型
			"type": "string",
                        // 配置的默认值
			"default": "/etc/hadoop/conf"
		}
	],
        // 角色。对于CM控制台，一个CSD是一个服务，一个服务可有多个角色，每个角色都拥有启动，停止，重启等操作
	"roles": [
		{
			// 角色名，name,label,pluralLabel一致即可
                        "name": "AILAB_CONSOLE",
			"label": "AILAB_CONSOLE",
			"pluralLabel": "AILAB_CONSOLE",
                        // 启动配置，指定当在CM控制台执行角色的启动命令时应如何操作
			"startRunner": {
                                // 运行脚本，相对路径
				"program": "scripts/control.sh",
                                // 运行脚本参数
				"args": [
					"start",
					"console"
				],
                                // 需要哪些变量作为环境变量
				"environmentVariables": {
                                        // 环境变量名:${配置变量名}
                                        //使用${}获取parameters中参数的值
					"HADOOP_CONF_HIVE_DIR": "${HADOOP_CONF_HIVE_DIR}",
					"HADOOP_CONF_HDFS_DIR": "${HADOOP_CONF_HDFS_DIR}",
					"HADOOP_CONF_CORE_DIR": "${HADOOP_CONF_CORE_DIR}",
					"HADOOP_CONF_YARN_DIR": "${HADOOP_CONF_YARN_DIR}"
				}
			},
                        // 停止配置，指定当在CM控制台执行角色的停止命令时应如何操作
			"stopRunner": {
                                // 三秒后脚本扔没有结束，强制停止
				"timeout": "30000",
				"runner": {
					"program": "scripts/control.sh",
					"args": [
						"stop",
						"console"
					]
				}
			},
                        // 角色的配置参数，在角色内可见
			"parameters": [
				{
					"name": "server.port",
					"label": "server.port",
					"description": "server port",
					"configName": "server.port",
					"required": "true",
					"type": "string",
					"default": "5555"
				},
				{
					"name": "log.home",
					"label": "log_home",
					"description": "log home",
					"configName": "log.home",
					"required": "true",
					"type": "string",
					"default": "logs/console"
				},
				{
					"name": "spring.datasource.url",
					"label": "spring.datasource.url",
					"description": "database dburl",
					"configName": "spring.datasource.url",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "jdbc:mysql://xxx:3306/dataworks?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&allowMultiQueries=true&useSSL=false&"
				},
				{
					"name": "spring.datasource.username",
					"label": "spring.datasource.username",
					"description": "spring.datasource.username",
					"configName": "spring.datasource.username",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx"
				},
				{
					"name": "spring.datasource.password",
					"label": "spring.datasource.password",
					"description": "spring.datasource.password",
					"configName": "spring.datasource.password",
					"required": "true",
					"type": "password",
					"configurableInWizard": true,
					"default": "xxx"
				},
				{
					"name": "spring.datasource.driverClassName",
					"label": "spring.datasource.driverClassName",
					"description": "spring.datasource.driverClassName",
					"configName": "spring.datasource.driverClassName",
					"required": "true",
					"type": "string",
					"default": "com.mysql.jdbc.Driver"
				}
			],
                        // 生成文件
			"configWriter": {
				"generators": [
					{
						// 生成的文件名为application-console.properties
                                                "filename": "application-console.properties",
						// 不需要哪些配置变量，如果不写会使用所有该角色可见变量
                                                "excludedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir",
							"HADOOP_CONF_CORE_DIR",
							"HADOOP_CONF_HDFS_DIR",
							"HADOOP_CONF_YARN_DIR",
							"HADOOP_CONF_YARN_DIR"
						],
                                                // 生成的文件类型
						"configFormat": "properties"
					},
					{
						"filename": "hadoop-defaults.conf",
                                                // 只需要哪些配置变量
						"includedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir"
						],
						"configFormat": "properties"
					}
				]
			},
                        // 指定角色日志
			"logging": {
                                // 角色日志位置
				"dir": "/var/log/ailab/console",
                                // 是否需要可以在配置UI中对dir进行修改
				"modifiable": true,
                                // 角色日志名
				"filename": "info.log",
                                // 日志类型
				"loggingType": "logback"
		    }
		},
                //可有多个角色。
		{
			"name": "AILAB_ONLINE",
			"label": "AILAB_ONLINE",
			"pluralLabel": "AILAB_ONLINE",
			"startRunner": {
				"program": "scripts/control.sh",
				"args": [
					"start",
					"online"
				],
				"environmentVariables": {
					"HADOOP_CONF_HIVE_DIR": "${HADOOP_CONF_HIVE_DIR}",
					"HADOOP_CONF_HDFS_DIR": "${HADOOP_CONF_HDFS_DIR}",
					"HADOOP_CONF_CORE_DIR": "${HADOOP_CONF_CORE_DIR}",
					"HADOOP_CONF_YARN_DIR": "${HADOOP_CONF_YARN_DIR}"
				}
			},
			"stopRunner": {
				"timeout": "30000",
				"runner": {
					"program": "scripts/control.sh",
					"args": [
						"stop",
						"online"
					]
				}
			},
			"parameters": [
				{
					"name": "server.port",
					"label": "server.port",
					"description": "server port",
					"configName": "server.port",
					"required": "true",
					"type": "string",
					"default": "5556"
				}
			],
			"configWriter": {
				"generators": [
					{
						"filename": "application-online.properties",
						"configFormat": "properties",
						"excludedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir",
							"HADOOP_CONF_CORE_DIR",
							"HADOOP_CONF_HDFS_DIR",
							"HADOOP_CONF_YARN_DIR",
							"HADOOP_CONF_YARN_DIR"
						]
					},
					{
						"filename": "hadoop-defaults.conf",
						"includedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir"
						],
						"configFormat": "properties"
					}
				]
			},
			"logging" : {
				"dir" : "/var/log/ailab/online",
				"modifiable" : true,
				"filename" : "info.log",
				"loggingType" : "logback"
		    }
		}
	]
}

项目实例：

{
"version": "dev",
	"name": "AILAB",
	"label": "Ailab",
	"description": "The ailab service",
	"icon": "images/ailab.png",
	"runAs": {
		"user": "root",
		"group": "root"
	},
	"parcel": {
		"requiredTags": [
			"ailab"
		],
		"optionalTags": [
			"ailab"
		]
	},
	"parameters": [
		{
			"name": "HADOOP_CONF_CORE_DIR",
			"label": "hadoop_conf_core_dir",
			"description": "core-site.xml's location",
			"configName": "HADOOP_CONF_CORE_DIR",
			"required": "true",
			"type": "string",
			"default": "/etc/hadoop/conf"
		},
		{
			"name": "HADOOP_CONF_HDFS_DIR",
			"label": "hadoop_conf_hdfs_dir",
			"description": "hdfs-site.xml's location",
			"configName": "HADOOP_CONF_HDFS_DIR",
			"required": "true",
			"type": "string",
			"default": "/etc/hadoop/conf"
		},
		{
			"name": "HADOOP_CONF_YARN_DIR",
			"label": "hadoop_conf_yarn_dir",
			"description": "yarn-site.xml's location",
			"configName": "HADOOP_CONF_YARN_DIR",
			"required": "true",
			"type": "string",
			"default": "/etc/hadoop/conf"
		},
		{
			"name": "HADOOP_CONF_HIVE_DIR",
			"label": "hadoop_conf_hive_dir",
			"description": "hive-site.xml's location",
			"configName": "HADOOP_CONF_HIVE_DIR",
			"required": "true",
			"type": "string",
			"default": "/etc/hive/conf"
		},
		{
			"name": "hadoop.kerberos.keytab",
			"label": "hadoop.kerberos.keytab",
			"description": "hadoop.kerberos.keytab",
			"configName": "hadoop.kerberos.keytab",
			"required": "true",
			"type": "string",
			"default": "/home/hiveall.keytab"
		},
		{
			"name": "hadoop.kerberos.principal",
			"label": "hadoop.kerberos.principal",
			"description": "hadoop.kerberos.principal",
			"configName": "hadoop.kerberos.principal",
			"required": "true",
			"type": "string",
			"default": "hive"
		},
		{
			"name": "hive.metastore.client.capability.check",
			"label": "hive.metastore.client.capability.check",
			"description": "hive.metastore.client.capability.check",
			"configName": "hive.metastore.client.capability.check",
			"required": "true",
			"type": "string",
			"default": "true"
		},
		{
			"name": "model.deploy.dir",
			"label": "model.deploy.dir",
			"description": "model.deploy.dir",
			"configName": "model.deploy.dir",
			"required": "true",
			"type": "string",
			"configurableInWizard": true,
			"default": "hdfs://xxx:8020/user/secsmart/deploy"
		}
	],
	"roles": [
		{
			"name": "AILAB_CONSOLE",
			"label": "AILAB_CONSOLE",
			"pluralLabel": "AILAB_CONSOLE",
			"startRunner": {
				"program": "scripts/control.sh",
				"args": [
					"start",
					"console"
				],
				"environmentVariables": {
					"HADOOP_CONF_HIVE_DIR": "${HADOOP_CONF_HIVE_DIR}",
					"HADOOP_CONF_HDFS_DIR": "${HADOOP_CONF_HDFS_DIR}",
					"HADOOP_CONF_CORE_DIR": "${HADOOP_CONF_CORE_DIR}",
					"HADOOP_CONF_YARN_DIR": "${HADOOP_CONF_YARN_DIR}"
				}
			},
			"stopRunner": {
				"timeout": "40000",
				"runner": {
					"program": "scripts/control.sh",
					"args": [
						"stop",
						"console"
					]
				}
			},
			"parameters": [
				{
					"name": "server.port",
					"label": "server.port",
					"description": "server port",
					"configName": "server.port",
					"required": "true",
					"type": "string",
					"default": "5555"
				},
				{
					"name": "log.home",
					"label": "log_home",
					"description": "log home",
					"configName": "log.home",
					"required": "true",
					"type": "string",
					"default": "logs/console"
				},
				{
					"name": "spring.http.multipart.max-file-size",
					"label": "spring.http.multipart.max-file-size",
					"description": "spring.http.multipart.max-file-size",
					"configName": "spring.http.multipart.max-file-size",
					"required": "true",
					"type": "string",
					"default": "100MB"
				},
				{
					"name": "spring.http.multipart.max-request-size",
					"label": "spring.http.multipart.max-request-size",
					"description": "spring.http.multipart.max-request-size",
					"configName": "spring.http.multipart.max-request-size",
					"required": "true",
					"type": "string",
					"default": "100MB"
				},
				{
					"name": "server.tomcat.max-http-post-size",
					"label": "server.tomcat.max-http-post-size",
					"description": "server.tomcat.max-http-post-size",
					"configName": "server.tomcat.max-http-post-size",
					"required": "true",
					"type": "string",
					"default": "104857600"
				},
				{
					"name": "server.tomcat.basedir",
					"label": "server.tomcat.basedir",
					"description": "server.tomcat.basedir",
					"configName": "server.tomcat.basedir",
					"required": "true",
					"type": "string",
					"default": "/var/tmp"
				},
				{
					"name": "spring.datasource.url",
					"label": "spring.datasource.url",
					"description": "database dburl",
					"configName": "spring.datasource.url",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "jdbc:mysql://xxx:3306/dataworks?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&allowMultiQueries=true&useSSL=false&"
				},
				{
					"name": "spring.datasource.username",
					"label": "spring.datasource.username",
					"description": "spring.datasource.username",
					"configName": "spring.datasource.username",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx"
				},
				{
					"name": "spring.datasource.password",
					"label": "spring.datasource.password",
					"description": "spring.datasource.password",
					"configName": "spring.datasource.password",
					"required": "true",
					"type": "password",
					"configurableInWizard": true,
					"default": "xxx"
				},
				{
					"name": "spring.datasource.driverClassName",
					"label": "spring.datasource.driverClassName",
					"description": "spring.datasource.driverClassName",
					"configName": "spring.datasource.driverClassName",
					"required": "true",
					"type": "string",
					"default": "com.mysql.jdbc.Driver"
				},
				{
					"name": "spring.datasource.hive.url",
					"label": "spring.datasource.hive.url",
					"description": "spring.datasource.hive.url",
					"configName": "spring.datasource.hive.url",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "jdbc:hive2://xxx:10000/dataworks;principal=hive/xxx"
				},
				{
					"name": "spring.datasource.hive.username",
					"label": "spring.datasource.hive.username",
					"description": "spring.datasource.hive.username",
					"configName": "spring.datasource.hive.username",
					"required": "true",
					"type": "string",
					"default": ""
				},
				{
					"name": "spring.datasource.hive.password",
					"label": "spring.datasource.hive.password",
					"description": "spring.datasource.hive.password",
					"configName": "spring.datasource.hive.password",
					"required": "true",
					"type": "password",
					"default": ""
				},
				{
					"name": "spring.datasource.hive.driver-class-name",
					"label": "spring.datasource.hive.driver-class-name",
					"description": "spring.datasource.hive.driver-class-name",
					"configName": "spring.datasource.hive.driver-class-name",
					"required": "true",
					"type": "string",
					"default": "org.apache.hive.jdbc.HiveDriver"
				},
				{
					"name": "spring.datasource.clickhouse.driver-class-name",
					"label": "spring.datasource.clickhouse.driver-class-name",
					"description": "spring.datasource.clickhouse.driver-class-name",
					"configName": "spring.datasource.clickhouse.driver-class-name",
					"required": "true",
					"type": "string",
					"default": "ru.yandex.clickhouse.ClickHouseDriver"
				},
				{
					"name": "service.cluster.zookeeper.address",
					"label": "service.cluster.zookeeper.address",
					"description": "service.cluster.zookeeper.address",
					"configName": "service.cluster.zookeeper.address",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx:2181,xxx:2181,xxx:2181"
				},
				{
					"name": "service.conf.zookeeper.address",
					"label": "service.conf.zookeeper.address",
					"description": "service.conf.zookeeper.address",
					"configName": "service.conf.zookeeper.address",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx:2181"
				},
				{
					"name": "kerberos.switch",
					"label": "kerberos.switch",
					"description": "kerberos.switch",
					"configName": "kerberos.switch",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "true"
				},
				{
					"name": "hdfs.root.dir",
					"label": "hdfs.root.dir",
					"description": "hdfs.root.dir",
					"configName": "hdfs.root.dir",
					"required": "true",
					"type": "string",
					"default": "/user/secsmart/dataworks"
				},
				{
					"name": "dubbo.application.name",
					"label": "dubbo.application.name",
					"description": "dubbo.application.name",
					"configName": "dubbo.application.name",
					"required": "true",
					"type": "string",
					"default": "login-dubbo-consumer"
				},
				{
					"name": "dubbo.registry.address",
					"label": "dubbo.registry.address",
					"description": "dubbo.registry.address",
					"configName": "dubbo.registry.address",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "zookeeper://xxx:2181?backup=xxx:2181"
				},
				{
					"name": "dubbo.service.version",
					"label": "dubbo.service.version",
					"description": "dubbo.service.version",
					"configName": "dubbo.service.version",
					"required": "true",
					"type": "string",
					"default": "1.0.0"
				},
				{
					"name": "dubbo.consumer.timeout",
					"label": "dubbo.consumer.timeout",
					"description": "dubbo.consumer.timeout",
					"configName": "dubbo.consumer.timeout",
					"required": "true",
					"type": "string",
					"default": "3000"
				},
				{
					"name": "flink.jobmanager.address",
					"label": "flink.jobmanager.address",
					"description": "flink.jobmanager.address",
					"configName": "flink.jobmanager.address",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx:8081,xxx:8081"
				}
			],
			"configWriter": {
				"generators": [
					{
						"filename": "application-console.properties",
						"excludedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir",
							"HADOOP_CONF_CORE_DIR",
							"HADOOP_CONF_HDFS_DIR",
							"HADOOP_CONF_YARN_DIR",
							"HADOOP_CONF_YARN_DIR"
						],
						"configFormat": "properties"
					},
					{
						"filename": "hadoop-defaults.conf",
						"includedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir"
						],
						"configFormat": "properties"
					}
				]
			},
			"logging": {
				"dir": "/var/log/ailab/console",
				"modifiable": true,
				"filename": "info.log",
				"loggingType": "logback"
		    }
		},
		{
			"name": "AILAB_ONLINE",
			"label": "AILAB_ONLINE",
			"pluralLabel": "AILAB_ONLINE",
			"startRunner": {
				"program": "scripts/control.sh",
				"args": [
					"start",
					"online"
				],
				"environmentVariables": {
					"HADOOP_CONF_HIVE_DIR": "${HADOOP_CONF_HIVE_DIR}",
					"HADOOP_CONF_HDFS_DIR": "${HADOOP_CONF_HDFS_DIR}",
					"HADOOP_CONF_CORE_DIR": "${HADOOP_CONF_CORE_DIR}",
					"HADOOP_CONF_YARN_DIR": "${HADOOP_CONF_YARN_DIR}"
				}
			},
			"stopRunner": {
				"timeout": "30000",
				"runner": {
					"program": "scripts/control.sh",
					"args": [
						"stop",
						"online"
					]
				}
			},
			"parameters": [
				{
					"name": "server.port",
					"label": "server.port",
					"description": "server port",
					"configName": "server.port",
					"required": "true",
					"type": "string",
					"default": "5556"
				},
				{
					"name": "log.home",
					"label": "log_home",
					"description": "log home",
					"configName": "log.home",
					"required": "true",
					"type": "string",
					"default": "logs/online"
				},
				{
					"name": "spring.datasource.url",
					"label": "spring.datasource.url",
					"description": "database dburl",
					"configName": "spring.datasource.url",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "jdbc:mysql://xxx:3306/dataworks?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&allowMultiQueries=true&useSSL=false&"
				},
				{
					"name": "spring.datasource.username",
					"label": "spring.datasource.username",
					"description": "spring.datasource.username",
					"configName": "spring.datasource.username",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx"
				},
				{
					"name": "spring.datasource.password",
					"label": "spring.datasource.password",
					"description": "spring.datasource.password",
					"configName": "spring.datasource.password",
					"required": "true",
					"type": "password",
					"configurableInWizard": true,
					"default": "xxxx"
				},
				{
					"name": "spring.datasource.driverClassName",
					"label": "spring.datasource.driverClassName",
					"description": "spring.datasource.driverClassName",
					"configName": "spring.datasource.driverClassName",
					"required": "true",
					"type": "string",
					"default": "com.mysql.jdbc.Driver"
				},
				{
					"name": "service.cluster.zookeeper.address",
					"label": "service.cluster.zookeeper.address",
					"description": "service.cluster.zookeeper.address",
					"configName": "service.cluster.zookeeper.address",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx:2181,xxx:2181,xxx:2181"
				},
				{
					"name": "service.registryMode",
					"label": "service.registryMode",
					"description": "service.registryMode",
					"configName": "service.registryMode",
					"required": "true",
					"type": "string",
					"default": "ip"
				},
				{
					"name": "service.network-interface-name",
					"label": "service.network-interface-name",
					"description": "service.network-interface-name",
					"configName": "service.network-interface-name",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "ens32"
				},
				{
					"name": "kerberos.switch",
					"label": "kerberos.switch",
					"description": "kerberos.switch",
					"configName": "kerberos.switch",
					"required": "true",
					"type": "string",
					"default": "true"
				},
				{
					"name": "service.console.address",
					"label": "service.console.address",
					"description": "service.console.address",
					"configName": "service.console.address",
					"required": "true",
					"type": "string",
					"configurableInWizard": true,
					"default": "xxx:5555"
				},
				{
					"name": "hdfs.root.dir",
					"label": "hdfs.root.dir",
					"description": "hdfs.root.dir",
					"configName": "hdfs.root.dir",
					"required": "true",
					"type": "string",
					"default": "/user/secsmart/dataworks"
				}
			],
			"configWriter": {
				"generators": [
					{
						"filename": "application-online.properties",
						"configFormat": "properties",
						"excludedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir",
							"HADOOP_CONF_CORE_DIR",
							"HADOOP_CONF_HDFS_DIR",
							"HADOOP_CONF_YARN_DIR",
							"HADOOP_CONF_YARN_DIR"
						]
					},
					{
						"filename": "hadoop-defaults.conf",
						"includedParams": [
							"hadoop.kerberos.keytab",
							"hadoop.kerberos.principal",
							"hive.metastore.client.capability.check",
							"model.deploy.dir"
						],
						"configFormat": "properties"
					}
				]
			},
			"logging" : {
				"dir" : "/var/log/ailab/online",
				"modifiable" : true,
				"filename" : "info.log",
				"loggingType" : "logback"
		    }
		}
	]
}

scripts/*.sh

最为重要的文件，CM控制台通过该文件对parcel进行控制

#!/bin/sh

echo "======Show variables======"
echo "AILAB_HOME : $AILAB_HOME"
echo "DATAX_HOME : $DATAX_HOME"
echo "PARCELS_ROOT : $PARCELS_ROOT"
echo "PARCEL_DIRNAME : $PARCEL_DIRNAME"
echo "====Show variables over===="

start_console() {
    echo "Running AILAB"
    nohup $CONF_DIR/bin/startup-console.sh > /var/log/ailab/console/info.log &
    tailf /var/log/ailab/console/info.log
}

stop_console() {
    echo "stop console"
    cp -r "$AILAB_HOME/bin" "$CONF_DIR"
    source $CONF_DIR/bin/stop-console.sh
    echo "stop console End...exit 0"
}

start_online() {
    echo "Running ONLINE"
    nohup $CONF_DIR/bin/startup-online.sh > /var/log/ailab/online/info.log &
    tailf /var/log/ailab/online/info.log
    echo "Running ONLINE End..."
}

stop_online() {
    echo "stop ONLINE"
    cp -r "$AILAB_HOME/bin" "$CONF_DIR"
    source $CONF_DIR/bin/stop-online.sh
    echo "stop ONLINE End...dir"
}

init() {
  echo "hadoop conf init start"
  
  cp -r "$AILAB_HOME/conf" \
        "$AILAB_HOME/bin" \
        "$AILAB_HOME/data" \
        "$AILAB_HOME/python" \
        "$AILAB_HOME/workspace" \
        "$AILAB_HOME/krb5" \
        "$CONF_DIR"

    echo "hadoop conf path environment variables:"
    echo "HADOOP_CONF_HDFS_DIR:$HADOOP_CONF_HDFS_DIR"
    echo "HADOOP_CONF_YARN_DIR:$HADOOP_CONF_YARN_DIR"
    echo "HADOOP_CONF_HIVE_DIR:$HADOOP_CONF_HIVE_DIR"
    echo "HADOOP_CONF_CORE_DIR:$HADOOP_CONF_CORE_DIR"
    
    cp "$HADOOP_CONF_CORE_DIR/core-site.xml" \
    "$HADOOP_CONF_HDFS_DIR/hdfs-site.xml" \
    "$HADOOP_CONF_YARN_DIR/yarn-site.xml" \
    "$HADOOP_CONF_HIVE_DIR/hive-site.xml" \
    "$CONF_DIR/conf/hadoop/"
    
    sed -i "s/=/  /" "$CONF_DIR/hadoop-defaults.conf"
    mv "$CONF_DIR/hadoop-defaults.conf" "$CONF_DIR/conf/hadoop/"


  echo "hadoop conf init end"
}

init_role() {
  init
  echo "$ROLE init"
  mv "$CONF_DIR/application-$ROLE.properties" "$CONF_DIR/conf/$ROLE/application.properties"
  echo "$ROLE init end...."
}

ROLE=$2
case "$1" in
    start)
      init_role
      start_"$ROLE"
      ;;
    stop)
      stop_"$ROLE"
      ;;
    *)
      echo "Usage AILAB {start|stop} {console|online}"
      ;;
esac

校验并打包成csdjar包

# 校验.sdl文件
java -jar /root/github/cloudera/cm_ext-master/validator/target/validator.jar -s descriptor/service.sdl
# 打包为csdjar包
jar -cvf AILABCSD-1.0.jar *

上传数据包

将parcel包和parcel.sh文件使用scp命令发送至cm-server端的 /opt/cloudera/parcel-repo 文件夹下

scp AILAB-V1.2.1-el7.parcel.sha nn1:/opt/cloudera/parcel-repo/
scp AILAB-V1.2.1-el7.parcel nn1:/opt/cloudera/parcel-repo/

将csd包使用scp命令发送至cm-server端的 /opt/cloudera/csd 文件夹下

scp AILABCSD-1.0/AILABCSD-1.0.jar nn1:/opt/cloudera/csd/

分发，激活包

进入CM Client点击parcel按钮
点击“检查新Parcel”按钮，cm会扫描/opt/cloudera/cloudera-repo路径下的parcle包，并将结果刷新会页面。

成功页面

此步骤页面发生错误就代表parcel包格式或者sha文件校验失败。需要重新制作parcel包以及.sha文件。

点击分配按钮，将parcel包分发至所有cm-agent服务器下的/opt/cloudera/parcels路径下并进行解压。动作完成后，分配按钮变为激活按钮。

分配前

分配后

点击激活按钮，在/opt/cloudera/parcels路径下会生成一个对应parcel的软连接。官方解释：a parcel causes the Cloudera Manager to link to the new components

生成软连接

重启cm-server服务

重启cm-server服务是为了更新CSD包，parcel是动态感知的，文件上传后CM控制台就可以感知到。但CSD则需要重启。

# 重启后CM Client不能访问 时间可能较长，1分钟左右，耐心等待
systemctl restart cloudera-scm-server

为了方便开发官方还提供了不需要重启的方法，但因为可能会出现意料之外的问题，正式环境不建议使用

登录cm控制台，在浏览器中访问如下url
http://xxx:7180/cmf/csd/refresh
如果是新安装则调用该接口
http://xxx:7180/cmf/csd/install?csdName=csd名
如果是重新安装则调用该接口 
http://xxx:7180/cmf/csd/reinstall?csdName=AILAB-dev
注：csdName为jar包的名称

进入CM Client，首页点击添加服务，出现服务类型列表，其中会有自定义的parcel服务
选择服务，点击继续，选择想要部署的主机。确定即可。

运行包

删除包，服务

在CM客户端中单击图标parcel按钮
点击停用按钮，当此动作完成，此时按钮变为已分配, 已激活

软连接删除

点击选择从服务器中移除按钮，当此动作完成，parcel状态变为已下载
点击删除按钮，将从cm-server的/opt/cloudera/cloudera-repo中删除该parcel。
在cm-sever服务器的/opt/cloudera/csd路径下，删除csd文件jar包。
所有的cm-agent服务器中，在/opt/cloudera/parcel-cache路径下，删除服务的.torrent文件（未经核实，不清楚不删可不可以，保证万一，最好删除）。
最为重要的，所有的cm-agent服务器中，在/opt/cloudera/cloudera-repo路径下，有一个隐藏文件夹.flood,其中会有parcel包和.torrent文件，一定要删除。否则，如果之后有同名parcel包，分发时不会从server服务器的/opt/cloudera/cloudera-repo路径中下载，而是从本地.flood中获取。从而导致使用的是之前的parcel。

# 删除缓存及隐藏文件。
rm -rf /opt/cloudera/parcels/.flood/NGINX-V1.19.6-el7.parcel
rm -rf /opt/cloudera/parcels/.flood/NGINX-V1.19.6-el7.parcel.torrent
rm -rf /opt/cloudera/parcel-cache/NGINX-V1.19.6-el7.parcel.torrent

如有问题请留言

issue：待补充

自定义parcel包在CM中部署

创建所需文件包

cm_ext

创建parcel包以及parcel.sha文件

创建parcel包

创建parcel.json文件

创建The Parcel Defines Script文件

打包为.parccel文件并校验

创建parcel.sha文件

创建csd包

目录结构

service.sdl

标签示例：

项目实例：

scripts/*.sh

校验并打包成csdjar包

上传数据包

分发，激活包

运行包

删除包，服务

打包为`.parccel`文件并校验