Azkaban Executor 设置任务并行度

112 阅读1分钟

近期需求,想对Azkaban当前运行的最大flow数有所控制。flow中可以包含多个job。(默认读者熟悉azkaban)

查到官网配置文件: azkaban.github.io/azkaban/doc…

在Azkaban executor的conf/azkaban.properties中配置:

executor.flow.threads 当前可以运行的flow的数量。默认为30 flow.num.job.threads 每个flow中并发运行的job数。默认为10

Executor Server Properties

ParameterDescriptionDefault
executor.portThe port for azkaban executor server12321
executor.global.propertiesA path to the properties that will be the parent for all jobs.none
azkaban.execution.dirThe folder for executing working directoriesexecutions
azkaban.project.dirThe folder for storing temporary copies of project files used for executionsprojects
executor.flow.threadsThe number of simulateous flows that can be run. These threads are mostly idle.30
job.log.chunk.sizeFor rolling job logs. The chuck size for each roll over5MB
job.log.backup.indexThe number of log chunks. The max size of each logs is then the index * chunksize4
flow.num.job.threadsThe number of concurrent running jobs in each flow. These threads are mostly idle.10
job.max.XmsThe maximum initial amount of memory each job can request. If a job requests more than this, then Azkaban server will not launch this job1GB
job.max.XmxThe maximum amount of memory each job can request. If a job requests more than this, then Azkaban server will not launch this job2GB
azkaban.server.flow.max.running.minutesThe maximum time in minutes a flow will be living inside azkaban after being executed. If a flow runs longer than this, it will be killed. If smaller or equal to 0, there's no restriction on running time.-1

MySQL Connection Parameter

ParameterDescriptionDefault
database.typeThe database type. Currently, the only database supported is mysql.mysql
mysql.portThe port to the mysql db3306
mysql.hostThe mysql hostlocalhost
mysql.databaseThe mysql database
mysql.userThe mysql user
mysql.passwordThe mysql password
mysql.numconnectionsThe number of connections that Azkaban web client can open to the database100