征服国产生态!Ambari 完美适配银河麒麟 V20 实战指南:打通大数据管理“最后一公里”

458 阅读4分钟

征服国产生态!Ambari 完美适配银河麒麟 V20 实战指南:打通大数据管理“最后一公里”

背景挑战
随着国产化替代进程加速,银河麒麟(Kylin)V20 操作系统已成为关键领域的主流选择。然而,将 Apache Ambari 这一成熟的大数据集群管理平台迁移至 Kylin V20,面临 依赖库兼容性、系统权限约束、服务启动冲突 等多重壁垒,传统安装方式几乎寸步难行。

本文价值
本指南首次系统化破解 Ambari 在 Kylin V20 上的适配困局,提供一套经生产验证的完整解决方案

方法一统一主机的 OS 类型标识(伪装成以支持版本kylin10)

若无法更换仓库,可修改主机 OS 信息,使其与当前仓库支持的 OS 类型(如 Kylin10)一致。

  1. 修改主机的 OS 识别文件
    在每台注册的主机上操作(如 hadoop-01):

    # 编辑 /etc/os-release
    sudo vi /etc/os-release
    

    修改 VERSION_ID 和 PRETTY_NAME 为 Kylin10:

    NAME="Kylin"
    VERSION="10 (SP2)"
    ID=kylin
    VERSION_ID="10"  # 关键修改点
    PRETTY_NAME="Kylin 10 SP2"
    
  2. 同步其他 OS 标识文件

    # 修改 redhat-release 文件(如有)
    sudo echo "Kylin release 10 (SP2)" > /etc/redhat-release
    
  3. 重启 Ambari Agent 并刷新注册信息

    ambari-agent restart
    # 在 Ambari Server 上清除旧注册缓存(谨慎操作)
    ambari-server refresh-host-registration --hostname hadoop-01
    

方法二适配kylin20

定位Ambari服务端的OS识别文件

/usr/lib/ambari-server/lib/ambari_commons/resources/os_family.json

#修改
"kylin": {
        "extends" : "redhat",
        "distro": [
          "kylin"
        ],
        "versions": [
          10,
          20
        ]
      },

/usr/lib/ambari-server/lib/ambari_commons/os_check.py

#修改我是uos识别成kylin,正常不需要修改
#_IS_KYLIN_LINUX = os.path.exists('/etc/kylin-release')
_IS_KYLIN_LINUX = os.path.exists('/etc/os-release')

不该会报错

/usr/lib/ambari-server/lib/ambari_server/os_check_type.py

#修改
if current_os == cluster_os or current_os == 'kylin10' or cluster_os == 'kylin10' or current_os == 'redhat8' or cluster_os == 'redhat8' or current_os == 'anolis20' or cluster_os == 'anolis20' or current_os == 'bclinux21' or cluster_os == 'bclinux21' or current_os == 'kylin20' or cluster_os == 'kylin20':
    sys.exit(0)

验证仓库与操作系统的映射

确保 Ambari 仓库配置中定义了 kylin20 的软件源。

  1. 检查仓库配置文件
    在 Ambari 服务端查看仓库配置(如 HDP-3.1.repo):

    vi /var/lib/ambari-server/resources/stacks/HDP/3.1/repos/repoinfo.xml
    

/var/lib/ambari-server/resources/stacks/<HDP>/<version>/repos/repoinfo.xml

<repo>
  <os family="kylin20"/>  <!-- 匹配 kylin20 -->
  <baseurl>http://your-repo.example.com/kylin20/HDP/</baseurl>
</repo>

修改yum_manager

/usr/lib/ambari-server/lib/ambari_commons/repo_manager/yum_manager.py

#大约在112行左右
current_os = OSCheck.get_os_family() + OSCheck.get_os_major_version()
    if current_os == 'kylin10' or current_os == 'kylin20' or current_os == 'centos8' or current_os == 'anolis20' or current_os == 'bclinux21':
      return [package for package in available_packages + installed_packages]
    else:
      return [package[0] for package in available_packages + installed_packages]
      ......
#大约在128行左右
if current_os == 'kylin10' or current_os == 'kylin20' or current_os == 'centos8' or current_os == 'anolis20' or current_os == 'bclinux21':
      cmd = "/usr/bin/yum list available --showduplicates"
      if repo_filter:
        cmd="/usr/bin/yum list available --showduplicates --disablerepo=* --enablerepo={0}".format(repo_filter)
      p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
      re = p.stdout.readlines()
      for i in range(2,len(re)):
        res = re[i].decode('utf-8').strip('\r\n').split()
        if pkg_names and not pkg_names in res[0]:
          continue
        packages.append(res[0].strip().partition(".")[0])
        ......
#大约在162行左右  
 packages = []
    if current_os == 'kylin10' or current_os == 'kylin20' or current_os == 'centos8' or current_os == 'anolis20' or current_os == 'bclinux21':
      cmd = "/usr/bin/yum list installed --showduplicates"
      p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
      re = p.stdout.readlines()
      for i in range(1,len(re)):
        res = re[i].decode('utf-8').strip('\r\n').split()
        if pkg_names and not pkg_names in res[0]:
          continue
        if repo_filter and not repo_filter.lower() in res[2].lower():
          continue
        packages.append(res[0].strip().partition(".")[0])
    else:
    ......
# 309
current_os = OSCheck.get_os_family() + OSCheck.get_os_major_version()
    if os.geteuid() == 0:
      if current_os == 'kylin10' or current_os == 'kylin20' or current_os == 'centos8' or current_os == 'anolis20' or current_os == 'bclinux21':
        return self.yum_installed_check_package(name)
      elif os.path.exists('/etc/openEuler-release') or os.path.exists('/etc/UnionTech-release'):
        return self.rpm_check_package_available(name)
      else:
        return self.yum_check_package_available(name)
    else:
      if current_os == 'kylin10' or current_os == 'kylin20' or current_os == 'centos8' or current_os == 'anolis20' or current_os == 'bclinux21':
        return self.yum_installed_check_package(name)
      else:
        return self.rpm_check_package_available(name)
......

修改ambari-server-2.7.3.0-139.jar

/usr/lib/ambari-server/ambari-server-2.7.3.0-139.jar

version_definition.xsd

<xs:simpleType name="family-type">
  <xs:restriction base="xs:string">
    <xs:enumeration value="redhat6" />
    <xs:enumeration value="redhat7" />
    <xs:enumeration value="redhat8" />
    <xs:enumeration value="kylin10" />
    <!-- 添加kylin20 -->
    <xs:enumeration value="kylin20" />
    <xs:enumeration value="anolis20" />
    <xs:enumeration value="bclinux21" />
    <xs:enumeration value="redhat-ppc6" />
    <xs:enumeration value="redhat-ppc7" />
    <xs:enumeration value="debian6" />
    <xs:enumeration value="debian7" />
    <xs:enumeration value="debian9" />
    <xs:enumeration value="ubuntu12" />
    <xs:enumeration value="ubuntu14" />
    <xs:enumeration value="ubuntu16" />
    <xs:enumeration value="ubuntu18" />
    <xs:enumeration value="suse11" />
    <xs:enumeration value="suse12" />
    <xs:enumeration value="amazonlinux2" />
  </xs:restriction>
</xs:simpleType>

修改metainfo

在 Stack 的根目录中,编辑 metainfo.xml 文件以定义 Kylin20 的支持。

  1. 编辑文件

    sudo vi /var/lib/ambari-server/resources/stacks/HDP/3.1/metainfo.xml
    
  2. 添加或修改 <os> 标签
    在 <os> 部分添加 Kylin20 的类型和版本匹配规则:

    <os>
      <type>kylin</type>
      <!-- 支持 Kylin10 和 Kylin20 -->
      <family>kylin</family>
      <versions>
        <version>10</version>
        <version>20</version>  <!-- 新增 Kylin20 -->
      </versions>
      <!-- 定义 OS 识别正则 -->
      <packageVersion>^kylin(10|20).*$</packageVersion>
    </os>
    

修改服务定义的 Metadata(可选)

如果某些服务(如 HDFS、YARN)需要在 Kylin20 上特殊配置,需在其服务的 metainfo.xml 中添加 OS 支持。

  1. 示例:修改 HDFS 服务的 Metadata

    sudo vi /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HDFS/metainfo.xml
    
  2. 添加 OS 类型限制(如有必要):

    <osSpecifics>
      <osSpecific>
        <osFamily>kylin</osFamily>
        <osVersions>
          <osVersion>10</osVersion>
          <osVersion>20</osVersion>  <!-- 允许在 Kylin20 上安装 -->
        </osVersions>
      </osSpecific>
    </osSpecifics>
    

zookeeper

/var/lib/ambari-server/resources/stacks/<HDP>/<low-version>/services/ZOOKEEPER/

<osSpecifics>
        <osSpecific>
          <osFamily>amazonlinux2,redhat6,redhat7,redhat8,kylin10,kylin20,anolis20,bclinux21,suse11,suse12</osFamily>
          <packages>
            <package>
              <name>zookeeper_${stack_version}</name>
            </package>
            <package>
              <name>zookeeper_${stack_version}-server</name>
            </package>
          </packages>
        </osSpecific>
        <osSpecific>
          <osFamily>debian7,debian9,ubuntu12,ubuntu14,ubuntu16,ubuntu18</osFamily>
          <packages>
            <package>
              <name>zookeeper-${stack_version}</name>
            </package>
            <package>
              <name>zookeeper-${stack_version}-server</name>
            </package>
          </packages>
        </osSpecific>
      </osSpecifics>