阅读 721

使用cli为aws elbv2批量添加监控告警

需求:

为aws上各个region的elbv2进行配置监控告警,监控使用的是aws的cloudwatch,告警使用的是aws的sns

  1. HealthyHostCount,一分钟检查一次,当健康主机数量等于0,就告警。健康主机的最大值小于等于0就告警
  2. UnHealthyHostCount 五分钟检查一次,当不健康主机数量大于或等于1个,就告警. 不健康主机数量的最小值大于等于1就告警
  3. HTTP_5XX 一分钟采集一次,周期为1分钟,1个数据点中有1次超过阈值就告警,当5xx超过10个为超过阈值
  4. HTTP_4XX 一分钟采集一次,周期为5分钟,5个数据点中有三次超过阈值就告警,当4xx超过10%为超过阈值

  5. 难点:

    每一个region下有多个elb2,每一个elb2下又分为80和443的listen,目前是需要对每一个listen都配置以上四个监控需求,所以整体来说工作量比较大。

    解决方式:

    采用aws的cli工具进行批量创建告警。

    涉及到的awscli命令有以下:

    aws elbv2 describe-load-balancers; //获取到该可用区内所有lb的信息
    复制代码
    aws elbv2 describe-listeners --load-balancer-arn XXXXXXX; //获取某一个lb的信息复制代码

    需要配置的地方:

    • 由于每一个region需要不同的sns配置,以及执行脚本的主机是不同region的ops机器,所以每一次执行的时候要到指定ops机器以及修改成该region下的sns地址。如下代码:
    if __name__ == '__main__':
        # 1.获取lb2的arn
        
        print("1. 获取当前可用区所有lb2所有的arn。")
        arndict = getAllArn()
        sns = "arn:aws:sns:ap-south-1:2617960:AWS_Alert_Ops"
        getSimpleArnAndTaggroup(arndict,sns)复制代码
    • 然后需要对执行的ops授权iam,目前是所有region下的ops机器有权限。
    • 登录ops机器后需要执行:
    aws configure //key等信息不需要输入直接回车即可,直接输入该ops机器所在的region,目前美图所涉及的region有:
        ap-south-1,    //孟买
        us-west-2,   //俄勒冈
        ap-southeast-1, //新加坡
        ap-northeast-1, //东京复制代码
    • 最后直接执行脚本python aws_elb_add_alert.py

    脚本中每一个aws命令具体参数的意思如下:(更加细节的可以看一下awscli文档)

    def getHealthyHostCountComm(lbname,port,taggroup,lbarn,sns):
        comm = '''
    %s cloudwatch put-metric-alarm \
    --alarm-name 'AWS_ELB_%s_PORT_%s_HealthyHostCount' \   //警报的名称
    --alarm-description 'aws elb HealthyHostCount' \       //警报描述
    --metric-name HealthyHostCount \					   //警报类型,可选类型有HTTPCode_Target_4XX_Count,UnHealthyHostCount,HealthyHostCount等
    --namespace AWS/ApplicationELB \					   //报警的对象,可选类型有AWS/ApplicationELB,AWS/RDS,AWS/EC2等
    --statistic Maximum \								   //对采集数据的一个判断
    --period 60 \										   //每次采集数据的周期
    --threshold 0 \										   //告警的阈值
    --evaluation-periods 1 \							   //采集的次数,总时间=次数*周期时间
    --datapoints-to-alarm 1 \							   //满足告警条件超过阈值的次数 
    --comparison-operator LessThanOrEqualToThreshold \	   //当前值与阈值对于操作,GreaterThanOrEqualToThreshold,GreaterThanThreshold,LessThanThreshold,LessThanOrEqualToThreshold
    --treat-missing-data notBreaching \					   //对于不满足告警条件的数据处理方式,missing,notBreaching,breaching,ignore
    --alarm-actions '%s' \								   //告警方式,这里填写的是sns的arn值
    --dimensions 'Name=TargetGroup,Value=targetgroup/%s' 'Name=LoadBalancer,Value=app/%s' ''' %(Contants['AWSCLI'],lbname,port,sns,taggroup,lbarn)
        return comm复制代码

    最终代码:

    #!/usr/bin/python
    # -*- coding: utf-8 -*-
    
    # @Version : 1.0
    # @Time    : 2018/5/28 18:12
    # @Author  : *************
    # @File    : aws_elb_add_alert.py
    # @Function: aws elb批量配置告警
    # @Note    : 由于每一个可用区都有一台ops机器,所以每一个可用区需要单独执行此脚本,或者用ansible
    
    import os,sys,commands,re,json
    
    Contants = {
        "AWSCLI":'/usr/bin/aws',
        "AWSREGION":['ap-south-1','us-west-2','ap-southeast-1','ap-northeast-1'] #孟买,俄勒冈,新加坡,东京
    }
    
    # 构造字典
    class CreateDict(dict):
        def __getitem__(self, item):
            try:
                return dict.__getitem__(self, item)
            except KeyError:
                value = self[item] = type(self)()
                return value
    
    #########################################################################################################
    # 配置告警
    
    # HealthyHostCount,一分钟检查一次,当健康主机数量等于0,就告警。健康主机的最大值小于等于0就告警
    def getHealthyHostCountComm(lbname,port,taggroup,lbarn,sns):
        comm = '''
    %s cloudwatch put-metric-alarm \
    --alarm-name 'AWS_ELB_%s_PORT_%s_HealthyHostCount' \
    --alarm-description 'aws elb HealthyHostCount' \
    --metric-name HealthyHostCount \
    --namespace AWS/ApplicationELB \
    --statistic Maximum \
    --period 60 \
    --threshold 0 \
    --evaluation-periods 1 \
    --datapoints-to-alarm 1 \
    --comparison-operator LessThanOrEqualToThreshold \
    --treat-missing-data notBreaching \
    --alarm-actions '%s' \
    --dimensions 'Name=TargetGroup,Value=targetgroup/%s' 'Name=LoadBalancer,Value=app/%s' ''' %(Contants['AWSCLI'],lbname,port,sns,taggroup,lbarn)
        return comm
    # UnHealthyHostCount 五分钟检查一次,当不健康主机数量大于或等于1个,就告警.  不健康主机数量的最小值大于等于1就告警
    def getUnHealthyHostCountComm(lbname,port,taggroup,lbarn,sns):
        comm = '''
    %s cloudwatch put-metric-alarm \
    --alarm-name 'AWS_ELB_%s_PORT_%s_UnHealthyHostCount' \
    --alarm-description 'aws elb UnHealthyHostCount' \
    --metric-name UnHealthyHostCount \
    --namespace AWS/ApplicationELB \
    --statistic Minimum \
    --period 300 \
    --threshold 1 \
    --evaluation-periods 1 \
    --datapoints-to-alarm 1 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --treat-missing-data notBreaching \
    --alarm-actions '%s' \
    --dimensions 'Name=TargetGroup,Value=targetgroup/%s' 'Name=LoadBalancer,Value=app/%s' ''' %(Contants['AWSCLI'],lbname,port,sns,taggroup,lbarn)
        return comm
    
    # HTTP_5XX 一分钟采集一次,周期为1分钟,1个数据点中有1次超过阈值就告警,当5xx超过10个为超过阈值
    def getHTTP_5XXComm(lbname,port,taggroup,lbarn,sns):
        comm = '''
    %s cloudwatch put-metric-alarm \
    --alarm-name 'AWS_ELB_%s_PORT_%s_HTTP_5XX' \
    --alarm-description 'aws elb http 5xx alert' \
    --metric-name HTTPCode_Target_5XX_Count \
    --namespace AWS/ApplicationELB \
    --statistic Sum \
    --period 60 \
    --threshold 10 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --treat-missing-data notBreaching \
    --evaluation-periods 1 \
    --datapoints-to-alarm 1 \
    --alarm-actions '%s' \
    --dimensions 'Name=LoadBalancer,Value=app/%s' '''%(Contants['AWSCLI'],lbname,port,sns,lbarn)
        return comm
    
    # HTTP_4XX 一分钟采集一次,周期为5分钟,5个数据点中有三次超过阈值就告警,当4xx超过10%为超过阈值
    def getHTTP_4XXComm(lbname,port,taggroup,lbarn,sns):
        comm = '''
    %s cloudwatch put-metric-alarm \
    --alarm-name 'AWS_ELB_%s_PORT_%s_HTTP_4XX' \
    --alarm-description 'aws elb http 4xx alert' \
    --metric-name HTTPCode_Target_4XX_Count \
    --namespace AWS/ApplicationELB \
    --statistic Sum \
    --period 60 \
    --threshold 10 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --treat-missing-data notBreaching \
    --evaluation-periods 5 \
    --datapoints-to-alarm 3 \
    --unit Percent \
    --alarm-actions '%s' \
    --dimensions 'Name=TargetGroup,Value=targetgroup/%s' 'Name=LoadBalancer,Value=app/%s' ''' %(Contants['AWSCLI'],lbname,port,sns,taggroup,lbarn)
        return comm
    
    # 执行命令函数
    def execCommand(comm):
        try:
            (exitstatus, outtext) = commands.getstatusoutput(comm)
            return outtext
        except Exception as e:
            print(e)
    # 获取当前可用区内所有lb2的基础信息
    def getAllArn():
        comm1 = "%s elbv2 describe-load-balancers" % Contants['AWSCLI']
        AllLb2Details = eval(execCommand(comm1))['LoadBalancers']
        arndict = CreateDict()
        for i in range(0,len(AllLb2Details)):
            lbarn = AllLb2Details[i]["LoadBalancerArn"]
            lbname = AllLb2Details[i]["LoadBalancerName"]
            arndict[lbname]["lbarn"] = lbarn
            comm2 = "%s elbv2 describe-listeners --load-balancer-arn %s" %(Contants['AWSCLI'],lbarn)
            alllisten = eval(execCommand(comm2))['Listeners']
            for j in range(0,len(alllisten)):
                taggroup = alllisten[j]['DefaultActions'][0]['TargetGroupArn']
                port = alllisten[j]['Port']
                arndict[lbname]["lbgroup"][port] = taggroup
        return json.dumps(arndict)
    # 获取简写
    def getSimpleArnAndTaggroup(arndict,sns):
        for lbname,lbvalue in eval(arndict).iteritems():
            lbarn = re.split(r'loadbalancer/app/',lbvalue['lbarn'])[-1]
            for port,taggroup in lbvalue['lbgroup'].iteritems():
                print("######################################################")
                taggroup = re.split(r':targetgroup/',taggroup)[-1]
                print("#####开始配置HealthyHostCountAlert#####")
                comm1 = getHealthyHostCountComm(lbname,port,taggroup,lbarn,sns)
                print(comm1)
                execCommand(comm1)
                print("#####开始配置UnHealthyHostCountAlert#####")
                comm2 = getUnHealthyHostCountComm(lbname,port,taggroup,lbarn,sns)
                print(comm2)
                execCommand(comm2)
                print("#####开始配置HTTP_5XX#####")
                comm3 = getHTTP_5XXComm(lbname,port,taggroup,lbarn,sns)
                print(comm3)
                execCommand(comm3)
                print("#####开始配置HTTP_4XX#####")
                comm4 = getHTTP_4XXComm(lbname,port,taggroup,lbarn,sns)
                print(comm4)
                execCommand(comm4)
    
    if __name__ == '__main__':
        # 1.获取lb2的arn
        print("1. 获取当前可用区所有lb2所有的arn。")
        arndict = getAllArn()
        sns = "arn:aws:sns:us-west-2:217608:AWS_Alert_Ops"
        getSimpleArnAndTaggroup(arndict,sns)
    复制代码


    文章分类
    开发工具