近期需求,想对Azkaban当前运行的最大flow数有所控制。flow中可以包含多个job。(默认读者熟悉azkaban)
查到官网配置文件: azkaban.github.io/azkaban/doc…
在Azkaban executor的conf/azkaban.properties中配置:
executor.flow.threads 当前可以运行的flow的数量。默认为30 flow.num.job.threads 每个flow中并发运行的job数。默认为10
Executor Server Properties
Parameter | Description | Default |
---|---|---|
executor.port | The port for azkaban executor server | 12321 |
executor.global.properties | A path to the properties that will be the parent for all jobs. | none |
azkaban.execution.dir | The folder for executing working directories | executions |
azkaban.project.dir | The folder for storing temporary copies of project files used for executions | projects |
executor.flow.threads | The number of simulateous flows that can be run. These threads are mostly idle. | 30 |
job.log.chunk.size | For rolling job logs. The chuck size for each roll over | 5MB |
job.log.backup.index | The number of log chunks. The max size of each logs is then the index * chunksize | 4 |
flow.num.job.threads | The number of concurrent running jobs in each flow. These threads are mostly idle. | 10 |
job.max.Xms | The maximum initial amount of memory each job can request. If a job requests more than this, then Azkaban server will not launch this job | 1GB |
job.max.Xmx | The maximum amount of memory each job can request. If a job requests more than this, then Azkaban server will not launch this job | 2GB |
azkaban.server.flow.max.running.minutes | The maximum time in minutes a flow will be living inside azkaban after being executed. If a flow runs longer than this, it will be killed. If smaller or equal to 0, there's no restriction on running time. | -1 |
MySQL Connection Parameter
Parameter | Description | Default |
---|---|---|
database.type | The database type. Currently, the only database supported is mysql. | mysql |
mysql.port | The port to the mysql db | 3306 |
mysql.host | The mysql host | localhost |
mysql.database | The mysql database | |
mysql.user | The mysql user | |
mysql.password | The mysql password | |
mysql.numconnections | The number of connections that Azkaban web client can open to the database | 100 |