• 欢迎访问圣洁的白莲!
  • 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧
  • 厚德载物,自强不息!

使用RKE部署rancher高可用

docker 菜根譚 2分钟前 1次浏览 已收录 0个评论 扫描二维码
文章目录[隐藏]

前提说明:

本次环境:

组件 版本
docker 19.03.12
rke v1.1.6
kubectl Client:v1.19.0 Server: v1.18.6
helm v3
rancher v2.4.8
kubernetes v1.18.6

一、环境准备

1.机器准备

2.80 四层代理服务器 节点上需要安装rke kubectl helm nginx

80节点需要和其它三个节点建立ssh互信实现免密登录 rke是通过ssh登录来远程操作其它节点的

80主机名设置为cf.rancher.com

IP地址 主机名
2.81 k8s-node1
2.82 k8s-node2
2.83 k8s-node3

81,82,83三个节点通过rke部署一个k8s集群 这个集群只用来运行Rancher Server实现Rancher Server的高可用

在所有机器上都要安装docker

yum remove docker docker-common docker-selinux docker-engine
yum install -y yum-utils device-mapper-persistent-data lvm2


wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.huaweicloud.com/docker-ce/linux/centos/docker-ce.repo


sed -i 's+download.docker.com+mirrors.huaweicloud.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo


yum makecache fast
yum install docker-ce -y
systemctl enable docker
systemctl start docker

cat > /etc/docker/daemon.json<<EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
  "insecure-registries": ["192.168.2.70"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
systemctl restart docker

helm

https://mirrors.huaweicloud.com/helm/v3.3.0/

添加rancher仓库:
helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable

kubectl create namespace cattle-system

2.安装rke

下载地址

https://github.com/rancher/rke/releases

​ RKE的工作方式是通过SSH连接到每个服务器,并在此服务器上建立到Docker socket的隧道,这意味着SSH用户必须能够访问此服务器上的Docker引擎。要启用对SSH用户的访问,您可以将此用户添加到Docker组:

usermod -aG docker

关闭selinux

1)CentOS7下可修改配置文件

vim /etc/sysconfig/selinux

2)设置 SELINUX=disabled,重启后永久关闭。

配置hosts

cat >>/etc/hosts <<EOF
192.168.2.80 cf.rancher.com
192.168.2.81 rancher-node1
192.168.2.82 rancher-node2
192.168.2.83 rancher-node3
EOF

设置IPV4转发

必须开启!

1)CentOS7 下可编辑配置文件:

# vi /etc/sysctl.conf

2)设置:

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

3)执行如下命令生效:

#sudo sysctl -p

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

禁用Swap

一定要禁用swap,否则kubelet组件无法运行。

1)永久禁用swap

可以直接修改

# vi /etc/fstab

文件,注释掉swap项。

2)临时禁用

# swapoff -a

设置SSH

RKE通过SSH tunnel进行安装部署,需要事先建立RKE到各节点的SSH免密登录。如果集群中有3个节点,需要在RKE机器上执行1遍秘钥生成命令ssh-keygen,并将生成侧公钥通过分发命令:ssh-copy-id {user}@{ip}。

1)在各个节点(192.168.2.81,192.168.2.82,192.168.2.83)上创建ssh用户,并将其添加至docker组中:

#useradd admin
#usermod -aG docker admin

注意:重启系统以后才能生效,只重启Docker服务是不行的!重启后,docker_user用户也可以直接使用docker run命令。

2)在rke所在主机上创建密钥:

ssh-keygen

3)将所生成的密钥的公钥分发到各个节点:

ssh-copy-id admin@192.168.2.81
ssh-copy-id admin@192.168.2.82
ssh-copy-id admin@192.168.2.83

重启机器。

3.rke安装k8s

安装     rke up --config=./rancher-cluster.yml

卸载    rke remove --config=./rancher-cluster.yml
nodes:
  - address: 192.168.2.81
    user: admin
    role: [controlplane,worker,etcd]
  - address: 192.168.2.82
    user: admin
    role: [controlplane,worker,etcd]
  - address: 192.168.2.83
    user: admin
    role: [controlplane,worker,etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

ingress:
  provider: nginx
  options:
    use-forwarded-headers: "true"

集群配置文件

默认情况下,RKE将查找名为cluster.yml的文件,该文件中包含有关将在服务器上运行的远程服务器和服务的信息。
最小文件应该是这样的:

参考

集群配置文件包含一个节点列表。每个节点至少应包含以下值:

  • 地址 – 服务器的SSH IP / FQDN
  • 用户 – 连接到服务器的SSH用户
  • 角色 – 主机角色列表:worker,controlplane或etcd

另一节是“服务”,其中包含有关将在远程服务器上部署的Kubernetes组件的信息。

有三种类型的角色可以使用主机:

  • etcd – 这些主机可以用来保存集群的数据。
  • controlplane – 这些主机可以用来存放运行K8s所需的Kubernetes API服务器和其他组件。
  • worker – 这些是您的应用程序可以部署的主机。
[root@node-x ~]# mkdir ~/.kube
[root@node-x ~]# cp kube_config_cluster.yml ~/.kube/config
[root@node-x ~]# kubectl get nodes
NAME           STATUS   ROLES                      AGE     VERSION
192.168.2.81   Ready    controlplane,etcd,worker   3m51s   v1.18.6
192.168.2.82   Ready    controlplane,etcd,worker   3m50s   v1.18.6
192.168.2.83   Ready    controlplane,etcd,worker   3m51s   v1.18.6

[root@node-x ~]# kubectl get pods -A
NAMESPACE       NAME                                      READY   STATUS              RESTARTS   AGE
ingress-nginx   default-http-backend-598b7d7dbd-hd9jj     1/1     Running             0          3m11s
ingress-nginx   nginx-ingress-controller-pd47v            1/1     Running             0          3m11s
ingress-nginx   nginx-ingress-controller-s7vqv            1/1     Running             1          3m11s
ingress-nginx   nginx-ingress-controller-z4s27            1/1     Running             0          3m11s
kube-system     canal-8plrk                               1/2     Running             0          3m30s
kube-system     canal-p2l5t                               2/2     Running             0          3m30s
kube-system     canal-z2j7w                               2/2     Running             0          3m30s
kube-system     coredns-849545576b-g58zm                  1/1     Running             0          37s
kube-system     coredns-849545576b-tw5kk                  1/1     Running            0          3m21s
kube-system     coredns-autoscaler-5dcd676cbd-4tfcf       1/1     Running             0          3m20s
kube-system     metrics-server-697746ff48-bf5sp           1/1     Running             0          3m16s
kube-system     rke-coredns-addon-deploy-job-72tss        0/1     Completed           0          3m22s
kube-system     rke-ingress-controller-deploy-job-d65fd   0/1     Completed           0          3m12s
kube-system     rke-metrics-addon-deploy-job-fznv8        0/1     Completed           0          3m17s
kube-system     rke-network-plugin-deploy-job-hkvmd       0/1     Completed           0          3m47s

二、使用自签名证书安装Rancher server

​ Rancher server设计默认需要开启SSL/TLS配置来保证安全,将ssl证书以Kubernetes Secret卷的形式传递给rancher server或Ingress Controller。首先创建证书密文,以便Rancher和Ingress Controller可以使用。

mkdir -p /home/admin/certs
cd /home/admin/certs/

1、 生成自签名证书

​ 在负载均衡器上执行一键生成ssl自签名证书脚本将自动生成tls.crt、tls.key、cacerts.pem三个文件。文件名称不能修改。ca文件名称必须是cacerts.pem

self_ssl.sh

#!/bin/bash -e

help ()
{
    echo  ' ================================================================ '
    echo  ' --ssl-domain: 生成ssl证书需要的主域名,如不指定则默认为www.rancher.local,如果是ip访问服务,则可忽略;'
    echo  ' --ssl-trusted-domain: 如果想多个域名访问,则添加扩展域名(SSL_TRUSTED_DOMAIN),多个扩展域名用逗号隔开;'
    echo  ' --ssl-size: ssl加密位数,默认2048;'
    echo  ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
    echo  ' 使用示例:'
    echo  ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
    echo  ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
    echo  ' ================================================================'
}

case "$1" in
    -h|--help) help; exit;;
esac

if [[ $1 == '' ]];then
    help;
    exit;
fi

CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
    key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
    value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
    case "$key" in
        --ssl-domain) SSL_DOMAIN=$value ;;
        --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
        --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
        --ssl-size) SSL_SIZE=$value ;;
        --ssl-date) SSL_DATE=$value ;;
        --ca-date) CA_DATE=$value ;;
        --ssl-cn) CN=$value ;;
    esac
done

# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca

# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}

## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}

SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt

echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m       | 生成 SSL Cert |       \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"

if [[ -e ./${CA_KEY} ]]; then
    echo -e "\033[32m ====> 1. 发现已存在CA私钥,备份"${CA_KEY}"为"${CA_KEY}"-bak,然后重新创建 \033[0m"
    mv ${CA_KEY} "${CA_KEY}"-bak
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
    echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi

if [[ -e ./${CA_CERT} ]]; then
    echo -e "\033[32m ====> 2. 发现已存在CA证书,先备份"${CA_CERT}"为"${CA_CERT}"-bak,然后重新创建 \033[0m"
    mv ${CA_CERT} "${CA_CERT}"-bak
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
    echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi

echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOM

if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then
    cat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOM
    IFS=","
    dns=(${SSL_TRUSTED_DOMAIN})
    dns+=(${SSL_DOMAIN})
    for i in "${!dns[@]}"; do
      echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
    done

    if [[ -n ${SSL_TRUSTED_IP} ]]; then
        ip=(${SSL_TRUSTED_IP})
        for i in "${!ip[@]}"; do
          echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
        done
    fi
fi

echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}

echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}

echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
    -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
    -days ${SSL_DATE} -extensions v3_req \
    -extfile ${SSL_CONFIG}

echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt
#执行脚本生成证书
./create_self-signed-cert.sh --ssl-domain=cf.rancher.com --ssl-trusted-ip=192.168.2.80,192.168.2.81,192.168.2.82,192.168.2.83 --ssl-size=2048 --ssl-date=3650

2、使用kubectl创建tls类型的secrets

#创建命名空间

kubectl create namespace cattle-system

#服务证书和私钥密文

kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=./tls.crt --key=./tls.key 

#ca证书密文

kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem 

3、安装rancher server

#使用helm安装rancher HA

helm install rancher rancher-stable/rancher \
--namespace cattle-system \
--set hostname=cf.rancher.com \
--set ingress.tls.source=secret \
--set privateCA=true

修改pod中的hosts解析

kubectl  -n cattle-system \
patch deployments rancher --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                        "hostnames":
                        [
                            "cf.rancher.com"
                        ],
                            "ip": "192.168.2.128"
                    }
                ]
            }
        }
    }
}'

查看创建

[root@cf certs]# kubectl get ingress -n cattle-system
NAME      CLASS    HOSTS            ADDRESS                                  PORTS     AGE
rancher   <none>   cf.rancher.com   192.168.2.81,192.168.2.82,192.168.2.83   80, 443   2m38s

[root@cf certs]# kubectl get pods -A
NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
cattle-system   rancher-767cc66f9d-d5947                  1/1     Running     3          11m
cattle-system   rancher-767cc66f9d-mb6dk                  1/1     Running     3          11m
cattle-system   rancher-767cc66f9d-ztzpc                  1/1     Running     3          11m
ingress-nginx   default-http-backend-598b7d7dbd-r47pq     1/1     Running     0          42h
ingress-nginx   nginx-ingress-controller-6jnnh            1/1     Running     1          42h
ingress-nginx   nginx-ingress-controller-9lnlr            1/1     Running     1          42h
ingress-nginx   nginx-ingress-controller-j2zc7            1/1     Running     2          42h
kube-system     canal-gb48d                               2/2     Running     0          42h
kube-system     canal-jv9zf                               2/2     Running     2          42h
kube-system     canal-pjszr                               2/2     Running     0          42h
kube-system     coredns-849545576b-spp8p                  1/1     Running     0          15m
kube-system     coredns-849545576b-t5pxz                  1/1     Running     0          42h
kube-system     coredns-autoscaler-5dcd676cbd-bnbs2       1/1     Running     0          42h
kube-system     metrics-server-697746ff48-x6p44           1/1     Running     0          42h
kube-system     rke-coredns-addon-deploy-job-l6nmm        0/1     Completed   0          42h
kube-system     rke-ingress-controller-deploy-job-hxt8n   0/1     Completed   0          42h
kube-system     rke-metrics-addon-deploy-job-k8msp        0/1     Completed   0          42h
kube-system     rke-network-plugin-deploy-job-lvcpr       0/1     Completed   0          42h

三、Nginx配置反向代理访问RancherUI

nginx中证书对应的域名需要与hostname选项匹配,否则ingress将无法代理访问Rancher

在192.168.2.80上操作

1.配置hosts文件

192.168.2.80 cf.rancher.com

2.配置nginx文件/etc/nginx/conf.d

rancher.conf

upstream rancher {
        server 192.168.2.81:80 max_fails=3 fail_timeout=5s;
        server 192.168.2.82:80 max_fails=3 fail_timeout=5s;
        server 192.168.2.83:80 max_fails=3 fail_timeout=5s;
    }

    map $http_upgrade $connection_upgrade {
        default Upgrade;
        ''      close;
    }

    server {
        listen 443 ssl http2;
        server_name cf.rancher.com;
        ssl_certificate /home/admin/certs/tls.crt;
        ssl_certificate_key /home/admin/certs/tls.key;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Port $server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://rancher;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            # 这里将允许您在 Rancher UI 中打开命令行窗口时,窗口可以保留最多15分钟。没有这个参数时,默认值为1分钟,一分钟后在Rancher>中的shell会自动关闭。
            proxy_read_timeout 900s;
            proxy_buffering off;
        }
    }

    server {
        listen 80;
        server_name cf.rancher.com;
        return 301 https://$server_name$request_uri;
    }

3.重新启动nginx

nginx -t
nginx -s reload

windows客户端配置

修改 c:\windows\system32\drivers\etc\hosts文件

192.168.2.80 cf.rancher.com

打开浏览器访问(只能用域名访问,不能用nginx服务器的IP访问)

参考文档

参考文档1

参考文档2

参考文档3


圣洁的白莲 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:使用RKE部署rancher高可用
喜欢 (0)
[]
分享 (0)
菜根譚
关于作者:
浩瀚学海,虔诚分享!

您必须 登录 才能发表评论!