Winse Blog

走走停停都是风景, 熙熙攘攘都向最好, 忙忙碌碌都为明朝, 何畏之.

K8s Harbor Config

为了对比，还是想写写在centos7上面安装Harbor：太简单了，想想当初在6上面安装那酸爽($.$)。。。

环境说明

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[root@cu2 kube-deploy]# yum install -y redhat-lsb
[root@cu2 kube-deploy]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.3.1611 (Core) 
Release:        7.3.1611
Codename:       Core

[root@cu2 kube-deploy]# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64

使用docker-multinode搭建的环境

1
2
3

[root@cu2 kube-deploy]# kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.6", GitCommit:"114f8911f9597be669a747ab72787e0bd74c9359", GitTreeState:"clean", BuildDate:"2017-03-28T13:36:31Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.6", GitCommit:"114f8911f9597be669a747ab72787e0bd74c9359", GitTreeState:"clean", BuildDate:"2017-03-28T13:36:31Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

安装配置

  • 证书准备
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

[root@cu2 ~]# cd /data/kubernetes/
[root@cu2 kubernetes]# cd kube-deploy/
[root@cu2 kube-deploy]# cat easy-rsa.sh 
#!/bin/sh

# cd /data/kubernetes
cd ..

git clone https://github.com/OpenVPN/easy-rsa.git
cd easy-rsa/easyrsa3

echo "# ======  CA  ======= #"
./easyrsa init-pki
./easyrsa build-ca #记住输入的密码，下面颁发证书还会用到

echo "# ======  CERT  ======= #"
./easyrsa gen-req cu nopass
./easyrsa sign-req server cu #commonName填将要用到的域名咯
  • 下载离线镜像

harbor-offline-installer-0.5.0.tgz，加载harbor.0.5.0.tgz里面的镜像

1
2
3
4
5
6
7

[root@cu2 harbor-make]# docker images 
REPOSITORY                                            TAG                 IMAGE ID            CREATED             SIZE
vmware/harbor-jobservice                              0.5.0               1700fbe602a0        3 months ago        148.4 MB
vmware/harbor-ui                                      0.5.0               6db5718f2012        3 months ago        209.6 MB
vmware/harbor-db                                      0.5.0               c401344852c6        3 months ago        326.8 MB
nginx                                                 1.11.5              cc16e49f1304        4 months ago        181.4 MB
registry                                              2.5.0               44a8766d1758        8 months ago        33.28 MB
  • 修改配置和yaml配置的镜像名称
1
2
3
4
5
6
7
8
9
10
11
12
13

[root@cu2 harbor-make]# vi harbor.cfg

hostname = cu.esw.cn
ui_url_protocol = https
ssl_cert = /data/kubernetes/easy-rsa/easyrsa3/pki/issued/cu.crt
ssl_cert_key = /data/kubernetes/easy-rsa/easyrsa3/pki/private/cu.key 

[root@cu2 harbor-make]# find kubernetes/ -name "*.rc.yaml" 
kubernetes/nginx/nginx.rc.yaml
kubernetes/mysql/mysql.rc.yaml
kubernetes/registry/registry.rc.yaml
kubernetes/ui/ui.rc.yaml
kubernetes/jobservice/jobservice.rc.yaml
  • 启动

k8s启动的配置用github上最新的，不要用release下面的！！

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

[root@cu2 harbor-make]# cd kubernetes/
[root@cu2 kubernetes]# python prepare 

[root@cu2 kubernetes]# cat kube.sh 
#!/bin/sh

OP=${1:-"apply"}

kubectl $OP -f pv/

kubectl $OP -f jobservice/jobservice.cm.yaml
kubectl $OP -f mysql/mysql.cm.yaml
kubectl $OP -f nginx/nginx.cm.yaml
kubectl $OP -f registry/registry.cm.yaml
kubectl $OP -f ui/ui.cm.yaml

kubectl $OP -f jobservice/jobservice.svc.yaml
kubectl $OP -f mysql/mysql.svc.yaml
kubectl $OP -f nginx/nginx.svc.yaml
kubectl $OP -f registry/registry.svc.yaml
kubectl $OP -f ui/ui.svc.yaml

kubectl $OP -f registry/registry.rc.yaml 
kubectl $OP -f mysql/mysql.rc.yaml 
kubectl $OP -f jobservice/jobservice.rc.yaml 
kubectl $OP -f ui/ui.rc.yaml 
kubectl $OP -f nginx/nginx.rc.yaml

客户端CA

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

[root@cu2 kube-deploy]# cat rsync-deploy.sh 
#!/bin/sh

SERVICES="$@"
HOSTS=$(echo cu{1..5} )

#########
# Harbor
if echo "$SERVICES" | grep harbor >/dev/null ; then 

  sed -i '/cu.esw.cn/d' /etc/hosts

  cat >>/etc/hosts <<EOF
$( kubectl get service nginx -n default -o jsonpath="{..clusterIP}" ) cu.esw.cn
EOF
  echo "Updated Local Hosts"

  for h in $HOSTS ; do
    if [[ $h != "$(hostname)" ]] ; then
      rsync -az /etc/hosts $h:/etc/
    fi

    ssh $h "mkdir -p /etc/docker/certs.d/cu.esw.cn/"
    rsync -az /data/kubernetes/easy-rsa/easyrsa3/pki/ca.crt $h:/etc/docker/certs.d/cu.esw.cn/

    ssh $h "docker login -u admin -p Harbor12345 cu.esw.cn"
  done
  echo "Harbor Rsync Succeeded"

fi

搞定，上传下载一个镜像试试：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

[root@cu1 dev]# docker build -t cu.esw.cn/library/codis:3.2 codis/
[root@cu1 dev]# docker push cu.esw.cn/library/codis:3.2

[root@cu2 ~]# docker pull cu.esw.cn/library/codis:3.2
3.2: Pulling from library/codis
386a066cd84a: Already exists 
a3ed95caeb02: Pull complete 
b1d31257c103: Pull complete 
0e627f083b66: Pull complete 
83912002f3f9: Pull complete 
fc5e0ef7d361: Pull complete 
47fe51a74a06: Pull complete 
08dacccac43c: Pull complete 
ec5a5e8fd71b: Pull complete 
83f9da97d228: Pull complete 
d4735c06cafa: Pull complete 
3a4dc262a84d: Pull complete 
bcf78ab0a1a9: Pull complete 
7ac5a6fd0bf8: Pull complete 
Digest: sha256:1c9280840222d736b7419b7e896b6286709d08e53890ae9e3d18062d61a9ad58
Status: Downloaded newer image for cu.esw.cn/library/codis:3.2

[root@cu3 ~]# docker pull cu.esw.cn/library/codis:3.2
...
layers from manifest don't match image configuration 暂时不清楚啥问题，临时解决。。。囧
[root@cu2 data]# docker save cu.esw.cn/library/codis:3.2 | ssh cu3 docker load

小结

高版本的docker和k8s对环境变量和config volumes都支持，配置相对就很简单。

–END

Comments