[root@cu2 make]# python kubernetes/prepare
Traceback (most recent call last):
File "kubernetes/prepare", line 145, in <module>
pkey = subprocess.check_output(['openssl','genrsa','4096'], stderr=devnull)
AttributeError: 'module' object has no attribute 'check_output'
> Python should be version 2.7 or higher. Note that you may have to install Python on Linux distributions (Gentoo, Arch) that do not come with a Python interpreter installed by default
https://github.com/h2oai/h2o-2/wiki/installing-python-2.7-on-centos-6.3.-follow-this-sequence-exactly-for-centos-machine-only
https://gist.github.com/dalegaspi/dec44117fa5e7597a559 我按这个小写的安装的
[root@cu2 make]# yum install centos-release-scl
[root@cu2 make]# yum install -y python27
[root@cu2 make]# scl enable python27 bash
[root@cu2 make]# /opt/rh/python27/root/usr/bin/python -V
Python 2.7.8
[root@cu2 make]# less harbor.cfg
[root@cu2 make]# /opt/rh/python27/root/usr/bin/python kubernetes/prepare
Warning: Key(ldap_searchdn) is not existing. Use empty string as default
Warning: Key(ldap_search_pwd) is not existing. Use empty string as default
Warning: Key(ldap_filter) is not existing. Use empty string as default
[root@cu2 kubernetes]# git clone https://github.com/OpenVPN/easy-rsa.git
https://github.com/OpenVPN/easy-rsa/blob/master/README.quickstart.md
[root@cu2 easyrsa3]# ll
total 56
-rwxr-xr-x 1 root root 35253 Mar 13 01:04 easyrsa
-rw-r--r-- 1 root root 4560 Mar 13 01:04 openssl-1.0.cnf
-rw-r--r-- 1 root root 8126 Mar 13 01:04 vars.example
drwxr-xr-x 2 root root 4096 Mar 13 01:04 x509-types
[root@cu2 easyrsa3]# ./easyrsa init-pki
init-pki complete; you may now create a CA or requests.
Your newly created PKI dir is: /data/kubernetes/easy-rsa/easyrsa3/pki
[root@cu2 easyrsa3]# ./easyrsa build-ca
Generating a 2048 bit RSA private key
.............................+++
..............................................+++
writing new private key to '/data/kubernetes/easy-rsa/easyrsa3/pki/private/ca.key.Nj5oHgfZC5'
Enter PEM pass phrase: 123456
Verifying - Enter PEM pass phrase: 123456
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:esw.cn
CA creation complete and you may now import and sign cert requests.
Your new CA certificate file for publishing is at:
/data/kubernetes/easy-rsa/easyrsa3/pki/ca.crt
[root@cu2 easyrsa3]# ./easyrsa gen-req cu nopass
Generating a 2048 bit RSA private key
..........+++
.................................+++
writing new private key to '/data/kubernetes/easy-rsa/easyrsa3/pki/private/cu.key.LQX3Dr2jG3'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [cu]:cu.esw.cn
Keypair and certificate request completed. Your files are:
req: /data/kubernetes/easy-rsa/easyrsa3/pki/reqs/cu.req
key: /data/kubernetes/easy-rsa/easyrsa3/pki/private/cu.key
[root@cu2 easyrsa3]# ./easyrsa sign-req server cu
You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.
Request subject, to be signed as a server certificate for 3650 days:
subject=
commonName = cu.esw.cn
Type the word 'yes' to continue, or any other input to abort.
Confirm request details: yes
Using configuration from /data/kubernetes/easy-rsa/easyrsa3/openssl-1.0.cnf
Enter pass phrase for /data/kubernetes/easy-rsa/easyrsa3/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName :PRINTABLE:'cu.esw.cn'
Certificate is to be certified until Mar 10 23:36:42 2027 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
Certificate created at: /data/kubernetes/easy-rsa/easyrsa3/pki/issued/cu.crt
[root@cu2 easyrsa3]#
这里得用签发server端证书，如果是client使用时会报错： v2 ping attempt failed with error: Get https://cu.esw.cn/v2/: x509: certificate specifies an incompatible key usage
[root@cu2 easyrsa3]# tree .
.
├── easyrsa
├── openssl-1.0.cnf
├── pki
│ ├── ca.crt
│ ├── certs_by_serial
│ │ └── 01.pem
│ ├── index.txt
│ ├── index.txt.attr
│ ├── index.txt.old
│ ├── issued
│ │ └── cu.crt
│ ├── private
│ │ ├── ca.key
│ │ └── cu.key
│ ├── reqs
│ │ └── cu.req
│ ├── serial
│ └── serial.old
├── vars.example
└── x509-types
├── ca
├── client
├── COMMON
└── server
6 directories, 18 files
[root@bigdata1 data]# ip link set docker0 down
[root@bigdata1 data]# ip link del docker0
RTNETLINK answers: Operation not supported
[root@bigdata1 data]# ip addr
删不掉，但是可以修改ip地址来实现相似的效果
ifconfig docker0 ${FLANNEL_SUBNET}
或者
[root@bigdata1 data]# ip link set dev docker0 mtu 1460
[root@bigdata1 data]# ip addr del 172.17.42.1/16 dev docker0
[root@bigdata1 data]# ip addr add ${FLANNEL_SUBNET} dev docker0
[root@bigdata1 data]# ip link set dev docker0 up
[root@bigdata1 data]# ifconfig # 查看重新分配的IP
先添加参数在前端运行
[root@bigdata1 data]# docker -d --mtu=1472 --bip=10.1.42.1/24
启动
[root@bigdata1 data]# sed -i 's/other_args=/other_args="--mtu=1472 --bip=10.1.42.1/24"/' /etc/sysconfig/docker
[root@bigdata1 data]# service docker start
Starting docker: [确定]
[root@bigdata1 data]# service docker status
docker (pid 4542) 正在运行...
[root@k8s ~]# yum install -y yum-utils
[root@k8s ~]# yum-config-manager --add-repo https://docs.docker.com/engine/installation/linux/repo_files/centos/docker.repo
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
adding repo from: https://docs.docker.com/engine/installation/linux/repo_files/centos/docker.repo
grabbing file https://docs.docker.com/engine/installation/linux/repo_files/centos/docker.repo to /etc/yum.repos.d/docker.repo
repo saved to /etc/yum.repos.d/docker.repo
[root@k8s ~]# yum makecache fast
[root@k8s ~]# yum -y install docker-engine
# 把保存数据的目录转移到大磁盘下面去
先启动服务来产生docker目录
[root@k8s ~]# service docker start
[root@k8s ~]# service docker stop
[root@k8s ~]# rm -rf /var/lib/docker/
[root@k8s ~]# ln -s /data/var/lib/docker /var/lib/
[root@k8s ~]# cd kube-deploy/docker-multinode/
[root@k8s docker-multinode]# ./master.sh
+++ [0206 19:07:23] K8S_VERSION is set to: v1.5.2
+++ [0206 19:07:23] ETCD_VERSION is set to: 3.0.4
+++ [0206 19:07:23] FLANNEL_VERSION is set to: v0.6.1
+++ [0206 19:07:23] FLANNEL_IPMASQ is set to: true
+++ [0206 19:07:23] FLANNEL_NETWORK is set to: 10.1.0.0/16
+++ [0206 19:07:23] FLANNEL_BACKEND is set to: udp
+++ [0206 19:07:23] RESTART_POLICY is set to: unless-stopped
+++ [0206 19:07:23] MASTER_IP is set to: localhost
+++ [0206 19:07:23] ARCH is set to: amd64
+++ [0206 19:07:23] IP_ADDRESS is set to: 192.168.1.112
+++ [0206 19:07:23] USE_CNI is set to: false
+++ [0206 19:07:23] USE_CONTAINERIZED is set to: false
+++ [0206 19:07:23] --------------------------------------------
+++ [0206 19:07:23] Killing docker bootstrap...
+++ [0206 19:07:24] Killing all kubernetes containers...
Do you want to clean /var/lib/kubelet? [Y/n] y
+++ [0206 19:07:27] Launching docker bootstrap...
+++ [0206 19:07:28] Launching etcd...
3ff0f0fd7a08282930449b2f496f786b9857f6290698d612cebc2086d1a1765c
+++ [0206 19:07:31] Launching flannel...
{"action":"set","node":{"key":"/coreos.com/network/config","value":"{ \"Network\": \"10.1.0.0/16\", \"Backend\": {\"Type\": \"udp\"}}","modifiedIndex":4,"createdIndex":4}}
3651d077f453900a898ce6ad9fe67a7422f0c8084ec86b6e6a1a2ab6b9b1c629
+++ [0206 19:07:33] FLANNEL_SUBNET is set to: 10.1.42.1/24
+++ [0206 19:07:33] FLANNEL_MTU is set to: 1472
+++ [0206 19:07:33] Restarting main docker daemon...
+++ [0206 19:07:38] Restarted docker with the new flannel settings
+++ [0206 19:07:38] Launching Kubernetes master components...
d10130677853022fe37742437e39b21b3fcfbb90b3f24075457f469e238b0712
+++ [0206 19:07:42] Done. It may take about a minute before apiserver is up.
[root@k8s docker-multinode]# docker ps -a
...一堆容器列表
# 设置代理。如果有docker镜像下载失败的话再配置docker环境变量
export NO_PROXY="localhost,127.0.0.1,10.0.0.0/8"
export https_proxy=http://k8s:8118/
export http_proxy=http://k8s:8118/
[root@bigdata-dev docker-multinode]# export MASTER_IP=192.168.1.112
[root@bigdata-dev docker-multinode]# ./worker.sh
+++ [0208 08:59:37] K8S_VERSION is set to: v1.5.2
+++ [0208 08:59:37] ETCD_VERSION is set to: 3.0.4
+++ [0208 08:59:37] FLANNEL_VERSION is set to: v0.6.1
+++ [0208 08:59:37] FLANNEL_IPMASQ is set to: true
+++ [0208 08:59:37] FLANNEL_NETWORK is set to: 10.1.0.0/16
+++ [0208 08:59:37] FLANNEL_BACKEND is set to: udp
+++ [0208 08:59:37] RESTART_POLICY is set to: unless-stopped
+++ [0208 08:59:37] MASTER_IP is set to: 192.168.1.112
+++ [0208 08:59:37] ARCH is set to: amd64
+++ [0208 08:59:37] IP_ADDRESS is set to: 192.168.1.248
+++ [0208 08:59:37] USE_CNI is set to: false
+++ [0208 08:59:37] USE_CONTAINERIZED is set to: false
+++ [0208 08:59:37] --------------------------------------------
+++ [0208 08:59:37] Killing all kubernetes containers...
+++ [0208 08:59:37] Launching docker bootstrap...
+++ [0208 08:59:38] Launching flannel...
+++ [0208 08:59:39] FLANNEL_SUBNET is set to: 10.1.42.1/24
+++ [0208 08:59:39] FLANNEL_MTU is set to: 1472
+++ [0208 08:59:39] Restarting main docker daemon...
+++ [0208 08:59:43] Restarted docker with the new flannel settings
+++ [0208 08:59:43] Launching Kubernetes worker components...
1ce6ee6af709485668c9f170b1bc234b34d55d18e53116295c887c88046ca231
+++ [0208 08:59:44] Done. After about a minute the node should be ready.
[root@k8s ~]# kubectl cluster-info
Kubernetes master is running at http://localhost:8080
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@k8s ~]# kubectl get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1 <none> 443/TCP 16d
[root@k8s ~]# kubectl get nodes
NAME STATUS AGE
192.168.1.112 Ready 16d
192.168.1.248 Ready 16d
[root@k8s ~]# kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
k8s-master-192.168.1.112 4/4 Running 9 1d
k8s-proxy-v1-4hp8c 1/1 Running 0 1d
k8s-proxy-v1-htrrf 1/1 Running 0 1d
kube-addon-manager-192.168.1.112 2/2 Running 0 1d
kube-dns-4101612645-q0kcw 4/4 Running 0 1d
kubernetes-dashboard-3543765157-hsls9 1/1 Running 0 1d
dashboard运行正常的话，就可以通过浏览器查看以及管理集群
== https://kubernetes.io/docs/user-guide/ui/
== 走socks5代理
http://k8s:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=default
问题及处理
镜像或者启动失败的问题可以 set -x 输出脚本调试信息，获取到出错位置的命令单独重新执行来定位。
