资讯详情

离线自动化部署CDH6.3.2

离线CDH集群自动化部署工具

离线CDH简单支持集群安装部署的自动脚本工具「」。

一、Features

已实现的自动化功能(仅支持)Redhat/CentOS系列）： Usage: init_ssh|install_softs|init_sys|init_dev|init_mysql|test_sys|init_cm|install_all

  1. init_ssh : Configure password-free login from the control machine to other machines
  2. install_softs : Make a custom yum source. Includes: Install http service, extract Centos7 rpm package, ansible rpm package, cdh related software.
  3. init_sys : Initialize the system configuration. Includes selinux, firewall, tuned, hugepage, swappiness, tmout, kernel, maxfiles
  4. init_dev : Configure jdk with version jdk-8u231-linux-x64
  5. init_mysql : Configure the scm configuration database, initialize the hive, scm, amon, … databases and authorize it
  6. test_sys : Test cluster network connectivity
  7. init_cm : Control node installation cloudera-scm-* and init the cloudera-scm-server with mysql
  8. install_all : Execute all scripts except init_ssh sequentially. sequen : install_softs->init_system->init_devenv->init_mysql->init_cm->test_system

二、Getting Start

在主机中选择一个执行过程，以下简称「」，并确保主机外网可用。

2.1 安装包准备

目前只支持离线安装，需要提前下载所需的软件安装包，如下：

  • JDK
    • 下载地址：Cloudera Archive CM
    • 注:版本无限，只支持 Oracle JDK RPM包，名称格式为
    • 示例安装包：jdk-8u231-linux-x64.rpm
  • Mariadb
    • 下载地址：Mariadb
    • 说明：5.仅支持7个以上版本RPM Bundle包，名称格式为
    • 示例安装包：mariadb-server.x86_64.rpm
  • MySQL驱动包（5.1.46）
    • 下载地址：MySQL Connector
    • 说明：与MySQL版本对应，名称格式为
    • 示例安装包：mysql-connector-java-5.1.46.tar.gz
  • Cloudera Manager
    • 下载地址：Cloudera Archive CM
    • 说明:版本无限
    • 示例安装包：
      • cloudera-manager-agent.x86_64 : The Cloudera Manager Agent
      • cloudera-manager-server.x86_64 : The Cloudera Manager Server
      • cloudera-manager-server-db-2.x86_64 : Embedded database for the Cloudera Manager Server
      • cloudera-manager-daemons.x86_64 : Provides daemons for monitoring Hadoop and related tools.
  • CDH
    • 下载地址：Cloudera Archive CDH
    • 说明:版本无限
    • 示例安装包：
      • manifest.json
      • CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
      • allkeys.asc
      • CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
      • CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
  • Centos rpm
    • Centos 官网 ISO
    • 说明:版本系列与机器版本保持一致:
    • 示例安装包：
      • centos7
  • ansible 安装包
    • rpm仓库
    • 说明：Centos7原生没有对应rpm包，自行下载
    • 示例安装包：
      • ansible-2.9.21-1.el7.noarch.rpm

百度云打包下载所有脚本：

链接：https://pan.baidu.com/s/1Qz_BFu0-4aOZ84nvC1FnOw  提取码：lsom

2.2 修改hosts文件

vi hosts

将集群所有节点的ip与hostname写入hosts文件中格式相同//etc/hosts，ip与hosts空间隔开。 示例：

192.168.1.156 autocdh1 192.168.1.157 autocdh2 192.168.1.158 autocdh3

2.3 修改配置文件

vi deploy_robot.config
  • control_host: 控制主机节点hostname
  • host_user: 集群主机用户名需要有root权限
  • host_passwd: 所有主机都需要一致的集群主机用户密码
  • contrast_host: 节点用于网络测试hostname
  • mysql_host: MySQL安装的节点hostname
  • http_server_host: Http服务节点
  • mysql_root_passwd: MySQL 初始化后使用此密码作为密码 root 用户密码
  • cm_host: cm server节点hostname
  • cm_db_passwd: cm数据库初始化密码

示例：

  • control_host=autocdh3
  • host_user=root
  • host_passwd=123123
  • contrasthost=autocdh3
  • http_server_host=autocdh2
  • mysql_host=autocdh1
  • mysql_root_passwd=123123
  • cm_host=autocdh3
  • cm_db_passwd=scm

2.4 执行脚本

# 配置ssh
sh deploy_robot.sh init_ssh
# 执行安装
sh deploy_robot.sh install_all

That’s All!

# 查看server启动日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
# 等待几分钟后看到 Started Jetty server.

访问cm主机的7180端口即可，账号密码为默认的admin/admin，登录即可。

三、使用说明

3.1 脚本参数

sh deploy_robot.sh 
Usage: <type> init_ssh|install_softs|init_sys|init_dev|init_mysql|test_sys|init_cm|install_all

1. init_ssh : Configure password-free login from the control machine to other machines 
2. install_softs : Make a custom yum source. Includes: Install http service, extract Centos7 rpm package, ansible rpm package, cdh related software. 
3. init_sys : Initialize the system configuration. Includes selinux, firewall, tuned, hugepage, swappiness, tmout, kernel, maxfiles
4. init_dev : Configure jdk with version jdk-8u231-linux-x64
5. init_mysql : Configure the scm configuration database, initialize the hive, scm, amon, ........... databases and authorize it 
6. test_sys : Test cluster network connectivity
7. init_cm : Control node installation cloudera-scm-* and init the cloudera-scm-server with mysql
8. install_all : Execute all scripts except init_ssh sequentially. sequen :  install_softs->init_system->init_devenv->init_mysql->init_cm->test_system

执行脚本需要指定「」参数。

除了Features中各个功能的执行参数之外， 将会依次执行2-7步骤。

3.2 yum软件安装列表

yum_requirements.txt 文件中描述了需要在集群各个节点上安装的软件名。

。

四、其他

4.1 注意事项

  1. 使用init_ssh设置免密登录时，如果已生成过ssh密钥，那么ssh的key与认证信息将被。如果之前配置过ssh信息请注意更新ssh key。或者选择不要执行init_ssh，手动进行ssh配置。
  2. 建议在即将投入生产或者干净的主机上运行，以免破坏已有生产系统环境。
  3. CM安装过程中。

五、附录

5.1 主控脚本

#!/bin/bash
#
if [ $# -lt 1 ]
then
    echo "Usage: <type> init_ssh|install_softs|init_sys|init_dev|init_mysql|test_sys|init_cm|install_all"
    echo "init_ssh : Configure password-free login from the control machine to other machines "
    echo "install_softs : Make a custom yum source. Includes: Install http service, extract Centos7 rpm package, ansible rpm package, cdh related software. "
    echo "init_sys : Initialize the system configuration. Includes selinux, firewall, tuned, hugepage, swappiness, tmout, kernel, maxfiles"
    echo "init_dev : Configure jdk with version jdk-8u231-linux-x64"
    echo "init_mysql : Configure the scm configuration database, initialize the hive, scm, amon, ........... databases and authorize it "
    echo "test_sys : Test cluster network connectivity"
    echo "init_cm : Control node installation cloudera-scm-* and init the cloudera-scm-server with mysql"
    echo "install_all : Execute all scripts except init_ssh sequentially. sequen : install_softs->init_system->init_devenv->init_mysql->init_cm->test_system"
    exit 1
fi

# declare config
declare -A CONFIG_NANME=(
    ["CTRL_HOST"]="control_host"
    ["USER"]="host_user"
    ["PASSWD"]="host_passwd"
    ["CONTRAST"]="contrast_host"
    ["DB_HOST"]="mysql_host"
    ["DB_ROOT_PASSWD"]="mysql_root_passwd"
    ["CM_HOST"]="cm_host"
    ["CM_DB_PASSWD"]="cm_db_passwd"
    ["HTTP_SERVER_HOST"]="http_server_host"
    )

declare -A config_map=()
declare -A hostip_map=()

export TOP_PID=$$
trap 'exit 1' TERM

function quit { 
        
    kill -s TERM $TOP_PID
}

function say { 
        
    printf '\033[1;4;%sm %s: %s \033[0m\n' "$1" "$2" "$3"
}

function err { 
        
    say "31" "!!!![error]!!!! deploy failed" "$1" >&2
    exit 1
}

function info { 
        
    say "32" "####[info]#### process info" "$1" >&1
}

function check_cmd { 
        
    command -v "$1" > /dev/null 2>&1
}

function need_cmd { 
        
    if ! check_cmd "$1"; then
        err "need '$1' (command not found)"
    fi
}

function need_ok { 
        
    if [[ $? -ne 0 ]]; then err "$1"; fi
}

function have_fun { 
        
    fun=`cat $SELF/have_fun`
    printf '\033[1;32m %s \033[0m\n\n' "$fun"
}

function need_config { 
        
    if [ -z $1 ]
    then
        err "need key to get config"
    else
        c=${config_map[$1]}
        if [ -z $c ]
        then
            err "config not found $1"
        fi
    fi
}

function get_config { 
        
    need_config $1
    eval $2=${config_map[$1]}
    info "get config $1:${config_map[$1]}."
}


function have { 
        
    if [ ! -f $1 ]
    then
        err "$1 file doesn't exists"
    fi
}


function init_config { 
        
    info "config loading..."
    config_file=$1
    info "get config_file $config_file."
    have $config_file
    config_arr=`cat $config_file | grep '='`
    for c in ${config_arr}
    do
        arr=(${c//=/ })
        key=${arr[0]}
        value=${arr[1]}
        config_map[$key]=$value
    done
    info "config load finished."
    echo 
}

function get_home { 
        
    if [ $1 == "root" ]
    then
        user_home="/root"
    else
        user_home="/home/$user"
    fi
    info "operate user home: $user_home"
    echo
}


function init_hosts { 
        
    info "hosts loading..."
    info "get host_file $SELF/hosts"
    host_file=$SELF/hosts
    have $host_file
    host_arr=`cat $host_file | sed s'/ /,/'`
    for a in ${host_arr[*]}
    do
        if [ -n $a ]
        then
            arr=(${a//,/ })
            ip=${arr[0]}
            host=${arr[1]}
            need_ok "host dosen't formated: $a"
            hostip_map[$host]=$ip
        fi
    done
    #key
    hosts=${ 
        !hostip_map[@]} 
    info "hosts: $hosts"
    #value
    ips=${hostip_map[@]} 
    info "ips: $ips"
    info "hosts load finished."
    echo
}

function set_hosts { 
        
    # 主机名设置
    info "start setting /etc/hosts and config ssh keys."
    rpm -ivh $install_path/expect-5.45-14.el7_1.x86_64.rpm $install_path/tcl-8.5.13-8.el7.x86_64.rpm
    need_cmd expect
    have $expect_file
    if [ -f /tmp/autocdh/hosts.bak ]
    then
        info "restore hosts from /tmp/hosts.bak"
        cat /tmp/autocdh/hosts.bak > /etc/hosts
    else
        info "backup hosts file to /tmp/hosts.bak"
        cat /etc/hosts > /tmp/autocdh/hosts.bak
    fi
    cat $host_file >> /etc/hosts
    info "hosts added to /etc/hosts"
    curr_hosts=`cat /etc/hosts`
    info "current hosts: ${curr_hosts[*]}"
    info "start config ssh key(all hosts)."
    for host in ${hosts[*]}
    do
        expect $expect_file ssh $host $user $passwd "rm -rf $user_home/.ssh"
        expect $expect_file ssh $host $user $passwd "hostnamectl set-hostname $host"
        expect $expect_file scp $host $user $passwd /etc/hosts
        expect $expect_file ssh $host $user $passwd "ssh-keygen -t rsa"
        key=`expect $expect_file ssh $host $user $passwd "cat $user_home/.ssh/id_rsa.pub"`
        echo $key | awk -F 'ssh-rsa' '{printf "ssh-rsa%s\n",$2}' >> $user_home/.ssh/authorized_keys
    done
    info "all hosts ssh key done."
    info "scp authorized_keys to all hosts."
    for host in ${hosts[*]}
    do
        expect $expect_file scp $host $user $passwd $user_home/.ssh/authorized_keys
    done
    info "scp authorized_keys done."
    ssh $db_host date
    need_ok "ssh failed."
    echo
}

function install_ansible { 
        
    info "ansible control host clean and makecache"
    yum clean all && yum makecache
    info "start install ansible."
    yum install -y ansible
    need_cmd ansible
    echo "[all]" > /etc/ansible/hosts
    for host in ${hosts[*]}
    do
        echo $host >> /etc/ansible/hosts
    done
    info "ansible finish config."
    echo
}

function ansible_command { 
        
    ansible all -a "$1"
    need_ok "ansible command failed: $1"
}

function ansible_shell { 
        
    ansible all -m shell -a "$1"
    need_ok "ansible shell failed: $1"
}

function ansible_copy { 
        
    ansible all -m copy -a "$1"
    need_ok "ansible copy failed: $1"
}


function install_httpd { 
        
    info "httpd server host: $http_server_host"
    ssh autocdh2 "httpd -version"
    if [[ $?  -eq 0 ]]; then
        info "httpd service already installed"
    else
        apr_file=`ls $install_path/apr-1.4.8-3.el7_4.1.x86_64.rpm`
        apr_util_file=`ls $install_path/apr-util-1.5.2-6.el7.x86_64.rpm`
        httpd_file=`ls $install_path/httpd-2.4.6-88.el7.centos.x86_64.rpm`
        httpd_tools_file=`ls $install_path/httpd-tools-2.4.6-88.el7.centos.x86_64.rpm`
        mailcap_file=`ls $install_path/mailcap-2.1.41-2.el7.noarch.rpm`
        have $apr_file
        have $apr_util_file
        have $httpd_file
        have $httpd_tools_file
        have $mailcap_file
        info "mkdir and scp to http host."
        ssh $http_server_host <<EOF mkdir /tmp/autocdh/ EOF
        scp $apr_file $http_server_host:$install_path/
        scp $apr_util_file $http_server_host:$install_path/
        scp $httpd_file $http_server_host:$install_path/
        scp $httpd_tools_file $http_server_host:$install_path/
        scp $mailcap_file $http_server_host:$install_path/
        info "scp httpd rpm done."
        
        info "install httpd, wait a moment..."
        ssh $http_server_host <<EOF cd /tmp/autocdh/ rpm -ivh apr-1.4.8-3.el7_4.1.x86_64.rpm apr-util-1.5.2-6.el7.x86_64.rpm httpd-2.4.6-88.el7.centos.x86_64.rpm httpd-tools-2.4.6-88.el7.centos.x86_64.rpm mailcap-2.1.41-2.el7.noarch.rpm if [ -f /etc/httpd/conf/httpd.conf ] then echo "add the httpd conf to support .parcel" sed -i 's?AddType application/x-gzip .gz .tgz?AddType application/x-gzip .gz .tgz .parcel?' /etc/httpd/conf/httpd.conf systemctl start httpd.service systemctl enable httpd.service systemctl stop firewalld else echo "http conf file not found" exit fi EOF
        info "httpd service install done"
    fi
    
}

function unpack_repo_gz { 
        

    info "httpd server host: $http_server_host"
    cdh_gz_file=`ls $install_path/cdh.tar.gz`
    rhel7_gz_file=`ls $install_path/centos7.tar.gz`
    ansible_gz_file=`ls $install_path/ansible.tar.gz`
    
    have $cdh_gz_file
    have $rhel7_gz_file
    have $ansible_gz_file

    info "mkdir and scp to http host."
    scp $cdh_gz_file $http_server_host:$install_path
    scp $rhel7_gz_file $http_server_host:$install_path
    scp $ansible_gz_file $http_server_host:$install_path
    info "scp cdh.tar.gz and centos7.tar.gz and ansible.tar.gz done."

    ssh $http_server_host <<EOF echo "unpack the cdh.tar.gz and centos7.tar.gz and ansible.tar.gz into /var/www/html/" tar -zxvf /tmp/autocdh/cdh.tar.gz -C /var/www/html/ tar -zxvf /tmp/autocdh/centos7.tar.gz -C /var/www/html/ tar -zxvf /tmp/autocdh/ansible.tar.gz -C /var/www/html/ echo "unpack the gz file success" EOF

}

function delete_void_repo { 
        

    info "delete void repo"
    ansible_shell "cd /etc/yum.repos.d/ && ls | grep -v TZ | xargs rm -rf"
    info "bakup all repo success"
}


function intranet_yum_repo { 
        
    info "start set intranet yum repo"
    if [[ -f /etc/yum.repos.d/TZcdh.repo ]]; then
        info "cdh repo already exists"
    else 
        echo -e "[TZcdh] name=TZcdh baseurl=http://$http_server_host/cdh/ gpgcheck=0 enabled=1"  >> /etc/yum.repos.d/TZcdh.repo
    fi
    if [[ -f /etc/yum.repos.d/TZcentos7.repo ]]; then
        info "TZcentos7 repo already exists"
    else 
        echo -e "[base] name=TZCentOS-7 baseurl=http://$http_server_host/CentOS-7/ gpgcheck=1 enabled=1 gpgkey=http://$http_server_host/CentOS-7/RPM-GPG-KEY-CentOS-7"  >> /etc/yum.repos.d/TZcentos7.repo
    fi
    if [[ -f /etc/yum.repos.d/TZansible.repo ]]; then
        info "TZansible repo already exists"
    else 
        echo -e "[TZansible] name=TZansible baseurl=http://$http_server_host/ansible/ gpgcheck=0 enabled=1"  >> /etc/yum.repos.d/TZansible.repo
    fi
    echo

}
function sync_repo { 
        
    info "sync repo for all cluster "
    ansible all -m copy -a "src=/etc/yum.repos.d/TZcdh.repo dest=/etc/yum.repos.d/TZcdh.repo"
    ansible all -m copy -a "src=/etc/yum.repos.d/TZansible.repo dest=/etc/yum.repos.d/TZansible.repo"
    ansible all -m copy -a "src=/etc/yum.repos.d/TZcentos7.repo dest=/etc/yum.repos.d/TZcentos7.repo"
    info "yum repo sync done."
}

function set_yum { 
        
    info "start set yum."
    info "clean and makecache,it may take a little time, please wait a moment..."
    ansible_shell "yum clean all && yum makecache"
    info "install softs."
    yum_requrements=`cat $SELF/yum_requirements.txt`
    echo "yum install -y $yum_requrements"
    ansible_command "yum install -y $yum_requrements"
    info "install softs done."
    echo
}

function set_selinux { 
        
    selinux_stat=`getenforce | tr 'A-Z' 'a-z'`
    info "setting selinux, current: $selinux_stat."
    if [ $selinux_stat == "enforcing" ]
    then
        info "change to disable..."
        sed -i 's/SELINUX=enforcing/SELINUX=disable/' /etc/selinux/config
        sed -i 's/SELINUX=Enforcing/SELINUX=disable/' /etc/selinux/config
        curr=`cat /etc/selinux/config | grep -v '#' | grep SELINUX=`
        info "current: $curr."
    fi
    info "sync to hosts."
    ansible all -m copy -a "src=/etc/selinux/config dest=/etc/selinux/config"
    info "done."
    echo
}


function set_firewall { 
        
    info "disable firewalld..."
    ansible all -a "systemctl status firewalld"
    ansible all -a "systemctl stop firewalld"
    ansible all -a "systemctl disable firewalld"
    info "firewalld disabled."
    echo
}


function set_ntp { 
        
    # ntp配置
    info "start ntp server..."
    info "change timezone info to Shanghai."
    ansible all -m shell -a "rm -rf /etc/localtime && ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime"
    info "install ntpd."
    ansible all -a "yum install ntp -y"
    info "sync time to 0.cn.pool.ntp.org"
    ansible all -a "ntpdate -u 0.cn.pool.ntp.org"
    info "add ntp1.aliyun.com to ntp.conf"
    is_exists=`cat /etc/ntp.conf | grep aliyun | wc -l`
    if [ $is_exists -eq 0 ]
    then
        echo "server ntp1.aliyun.com" >> /etc/ntp.conf
    fi
    ansible all -a "systemctl start ntpd"
    ansible all -a "systemctl enable ntpd"
    info "ntp done."
    echo
}

function set_java { 
        
    info "start set java..."
    info "remove the old java | jdk"
    
    ansible all -a "rpm -qa | grep java | xargs rpm -e --nodeps"
    ansible all -a "rpm -qa | grep jdk | xargs rpm -e --nodeps"
    ansible all -a "rpm -ivh http://$http_server_host/cdh/jdk-8u231-linux-x64.rpm"    

    if ! check_cmd java; then
        info "java install failed"
    else
        info "java already installed."
    fi
    info "java install done."
    echo
}


function set_tuned { 
        
    info "disable tuned..."
    ansible all -a "systemctl start tuned"
    ansible all -a "systemctl status tuned"
    ansible all -a "tuned-adm off"
    ansible all -a "tuned-adm list"
    ansible all -a "systemctl stop tuned"
    ansible all -a "systemctl disable tuned"    
    info "tuned config done."
    echo
}

function set_hugepage { 
        
    info "disable hugepage..."
    curr0=`cat /sys/kernel/mm/transparent_hugepage/enabled`
    curr1=`cat /sys/kernel/mm/transparent_hugepage/defrag`
    info "current: $curr0 , $curr1"
    is_enable=`cat /sys/kernel/mm/transparent_hugepage/enabled | grep \\[always\\] | wc -l`
    is_defrag=`cat /sys/kernel/mm/transparent_hugepage/defrag | grep \\[always\\] | wc -l`
    if [ $is_enable -eq 1 ]
    then
        ansible all -m shell -a "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
        ansible all -m shell -a "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
        echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local
        echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
        chmod +x /etc/rc.d/rc.local
        ansible all -m copy -a "src=/etc/rc.d/rc.local dest=/etc/rc.d/rc.local"
        original=`cat /etc/default/grub | grep GRUB_CMDLINE_LINUX | awk -F '="' '{print $2}'`
        result="GRUB_CMDLINE_LINUX=\"transparent_hugepage=never "$original
        cat /etc/default/grub | grep -v GRUB_CMDLINE_LINUX > /etc/default/grub
        echo $result >> /etc/default/grub
        curr2=`cat /etc/default/grub | grep GRUB_CMDLINE_LINUX`
        info "current: $curr2"
        ansible all -m copy -a "src=/etc/default/grub dest=/etc/default/grub"
        ansible all -a "grub2-mkconfig -o /boot/grub2/grub.cfg"
    fi
    info "setting hugepage done."
    echo
}

function set_swappiness {

标签： aoz1014aic集成电路

锐单商城拥有海量元器件数据手册、 IC替代型号，打造 电子元器件IC百科大全！

锐单商城 - 一站式电子元器件采购平台