Winse Blog

走走停停都是风景, 熙熙攘攘都向最好, 忙忙碌碌都为明朝, 何畏之.

K8s Harbor Config on Centos6

前传

前面有写在 centos6 安装k8s的文章,后来重启一台worker节点后该节点的网络就不通了 connect: Invalid argument 。更新到最新的0.7.0后worker节点重启网络都能正常连通。

言归正传,来说说harbor的安装。想的是安装一个类似maven私服的功能(原来都是一台机一台机的save/load,麻烦):

  • 本来安装registry就好了,每次都要加端口很烦有没有!!!
  • 弄了个service整到80端口,还得加 –insecure-registry 参数。还行吧,但是没有图形界面
  • 好了,看到有人用nexus3做docker私服。主要吧真没弄通,第二nexus3不会用!反正就是没搭成功了。
  • 本来前面有看到过vmware harbor,但是官网说是要docker1.10+的,差点就打消念头了,但是nexus3实在是搞不懂,只能硬着头皮尝试下harbor。

这hardor是一坑货啊,功能是狠牛逼但是文档版本都对不上的!!!

这里还是在 centos6 上面安装。并且老版本k8s-1.2各种配置不能用,一个个坑填的好苦!行,先爽一把,看看修改后的简单的安装操作流程:

简单配置

版本信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@cu2 ~]# lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.8 (Final)
Release:        6.8
Codename:       Final
[root@cu2 ~]# docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d/1.7.1
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d/1.7.1
OS/Arch (server): linux/amd64
  • 创建CA和证书
1
2
3
4
5
6
7
8
9
[root@cu2 kubernetes]# git clone https://github.com/OpenVPN/easy-rsa.git

[root@cu2 easyrsa3]# ./easyrsa init-pki
[root@cu2 easyrsa3]# ./easyrsa build-ca #记住输入的密码,下面颁发证书还会用到

[root@cu2 easyrsa3]# ./easyrsa gen-req cu nopass
[root@cu2 easyrsa3]# ./easyrsa sign-req server cu #commonName填将要用到的域名咯

生成的key和证书在pki/private和pki/issued下
  • 下载配置
1
2
git clone https://github.com/winse/docker-hadoop.git
cd docker-hadoop/k8s-centos6/containers/harbor-make/
  • 修改harbor.cfg配置

域名证书路径 修改成自己的。

  • 生成ConfigMaps配置
1
2
scl enable python27 bash
python2.7 kubernetes/prepare 
  • 创建服务和容器

这里需要先下载官网的离线包harbor-offline-installer-0.5.0.tgz,加载harbor.0.5.0.tgz里面的镜像

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@cu2 harbor]# docker images 
REPOSITORY                                            TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
gcr.io/google_containers/heapster-grafana-amd64       v4.0.2              74d2c72849cc        7 weeks ago         131.5 MB
gcr.io/google_containers/heapster-influxdb-amd64      v1.1.1              55d63942e2eb        7 weeks ago         11.59 MB
gcr.io/google_containers/heapster-amd64               v1.3.0-beta.1       026fb02eca65        7 weeks ago         101.3 MB
quay.io/coreos/flannel                                v0.7.0-amd64        072e88d50780        8 weeks ago         73.75 MB
gcr.io/google_containers/kubernetes-dashboard-amd64   v1.5.1              9af7d5c61ccf        8 weeks ago         103.6 MB
vmware/harbor-log                                     0.5.0               5cccdd11efe0        3 months ago        190.5 MB
vmware/harbor-jobservice                              0.5.0               573d0bbd91ee        3 months ago        169.4 MB
vmware/harbor-ui                                      0.5.0               990d3476bf93        3 months ago        233 MB
vmware/harbor-db                                      0.5.0               9a595c26d6bc        3 months ago        326.8 MB
nginx                                                 1.11.5              98f8314de615        4 months ago        181.4 MB
gcr.io/google_containers/hyperkube-amd64              v1.2.7              1dd7250ed1b3        4 months ago        231.4 MB
quay.io/coreos/flannel                                v0.6.1-amd64        ef86f3a53de0        6 months ago        27.89 MB
gcr.io/google_containers/etcd-amd64                   3.0.4               ef5e89d609f1        7 months ago        39.62 MB
registry                                              2.5.0               8cc599785872        7 months ago        33.28 MB
gcr.io/google_containers/kube2sky-amd64               1.15                f93305484d65        10 months ago       29.16 MB
gcr.io/google_containers/etcd-amd64                   2.2.5               a6752fb962b5        11 months ago       30.45 MB
gcr.io/google_containers/skydns-amd64                 1.0                 a925f95d080a        11 months ago       15.57 MB
gcr.io/google_containers/exechealthz-amd64            1.0                 5b9ac190b20c        11 months ago       7.116 MB
gcr.io/google_containers/pause                        2.0                 9981ca1bbdb5        17 months ago       350.2 kB

---

cd kubernetes/
sh apply.sh
  • 手动修复容器的配置文件
1
sh config.sh

CentOS6-K8S上面麻烦点,在CentOS7-K8S_V1.5+上面ConfigMap Volumn是可以用的,就不需要自己手动拷贝配置了。

  • 使用
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@cu2 easyrsa3]# kubectl get services 
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
jobservice   10.0.0.154   <none>        80/TCP              1d
kubernetes   10.0.0.1     <none>        443/TCP             2d
mysql        10.0.0.176   <none>        3306/TCP            1d
nginx        10.0.0.78    <none>        80/TCP,443/TCP      1d
registry     10.0.0.46    <none>        5000/TCP,5001/TCP   1d
ui           10.0.0.11    <none>        80/TCP              1d

# 域名
[root@cu3 ~]# vi /etc/hosts
10.0.0.78 cu.esw.cn

# 证书
[root@cu3 ~]# mkdir -p /etc/docker/certs.d/cu.esw.cn/

[root@cu2 pki]# scp ca.crt cu3:/etc/docker/certs.d/cu.esw.cn/

# 登录
[root@cu3 certs.d]# docker login cu.esw.cn
Username: admin
Password: Harbor12345
Email: 1
WARNING: login credentials saved in /root/.docker/config.json
Login Succeeded

# https://cu.esw.cn 通过WEB页面创建项目 google_containers

# PUSH
[root@cu3 certs.d]# docker tag gcr.io/google_containers/pause:2.0 cu.esw.cn/google_containers/pause:2.0

[root@cu3 certs.d]# docker push cu.esw.cn/google_containers/pause:2.0
The push refers to a repository [cu.esw.cn/google_containers/pause] (len: 1)
9981ca1bbdb5: Image already exists 
6995a49b90f2: Image successfully pushed 
Digest: sha256:139471770ffc22a2f15ae2ad8e3a0b3b9cbd620ad32400c7e8024a3d09ebec7d

—— 下面是记流水账内容 ——

简单搭建配置

参考阅读

简略步骤:

  • 下载0.5.0的离线压缩包 harbor-offline-installer-0.5.0.tgz
  • 把镜像加载到本地(解压offline后在目录下有tgz的镜像压缩包) docker load -i harbor.0.5.0.tgz
  • 下载github主干的源码 harbor-master.zip ,对是主干,不是release页面的源码!!!(香菇,release源码包里面的k8s配置文件尽然是不配套的,那打什么版本咯!!文档也不说明下。非常非常感谢 www.zoues.com 博主,这才是明灯啊)
  • 安装python2.7(prepare脚本需要) yum install centos-release-scl; yum install -y python27
  • 解压进入到 harbor-master/make 目录
  • 修改harbor.cfg文件配置。(这里我就改了域名而已,会有https的问题。先不管跑起来先,后面在讲https的处理)
  • 执行prepare脚本,用于生成配置键值对cm文件(ConfigMaps)。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@cu2 make]# python kubernetes/prepare 
Traceback (most recent call last):
  File "kubernetes/prepare", line 145, in <module>
    pkey = subprocess.check_output(['openssl','genrsa','4096'], stderr=devnull)
AttributeError: 'module' object has no attribute 'check_output'

> Python should be version 2.7 or higher. Note that you may have to install Python on Linux distributions (Gentoo, Arch) that do not come with a Python interpreter installed by default

https://github.com/h2oai/h2o-2/wiki/installing-python-2.7-on-centos-6.3.-follow-this-sequence-exactly-for-centos-machine-only
https://gist.github.com/dalegaspi/dec44117fa5e7597a559  我按这个小写的安装的
[root@cu2 make]# yum install centos-release-scl
[root@cu2 make]# yum install -y python27

[root@cu2 make]# scl enable python27 bash
[root@cu2 make]# /opt/rh/python27/root/usr/bin/python -V
Python 2.7.8

[root@cu2 make]# less harbor.cfg 

[root@cu2 make]# /opt/rh/python27/root/usr/bin/python kubernetes/prepare 
Warning: Key(ldap_searchdn) is not existing. Use empty string as default
Warning: Key(ldap_search_pwd) is not existing. Use empty string as default
Warning: Key(ldap_filter) is not existing. Use empty string as default
  • 然后就是愉快的执行apply就好:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
kubectl apply -f pv/

kubectl apply -f jobservice/jobservice.cm.yaml
kubectl apply -f mysql/mysql.cm.yaml
kubectl apply -f nginx/nginx.cm.yaml
kubectl apply -f registry/registry.cm.yaml
kubectl apply -f ui/ui.cm.yaml

kubectl apply -f jobservice/jobservice.svc.yaml
kubectl apply -f mysql/mysql.svc.yaml
kubectl apply -f nginx/nginx.svc.yaml
kubectl apply -f registry/registry.svc.yaml
kubectl apply -f ui/ui.svc.yaml

kubectl apply -f registry/registry.rc.yaml
kubectl apply -f mysql/mysql.rc.yaml
kubectl apply -f jobservice/jobservice.rc.yaml
kubectl apply -f ui/ui.rc.yaml
kubectl apply -f nginx/nginx.rc.yaml

由于ConfigMaps方式不能正确的创建文件需要把配置文件拷贝到对应容器的config目录下:

1
sh config.sh

除了nginx报https的证书问题外,其他都正常跑起来了。把nginx.conf的https server部分先删掉,先查看效果。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@cu2 kubernetes]# kubectl get rc
NAME            DESIRED   CURRENT   AGE
jobservice-rc   1         1         4h
mysql-rc        1         1         4h
nginx-rc        1         1         4h
registry-rc     1         1         4h
ui-rc           1         1         4h
[root@cu2 kubernetes]# kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
jobservice-rc-3hhea        1/1       Running   0          4h
k8s-master-192.168.0.214   4/4       Running   28         2d
k8s-proxy-192.168.0.214    1/1       Running   4          2d
mysql-rc-nyk6z             1/1       Running   0          4h
nexus-3126345715-mfteg     1/1       Running   0          2d # 这个是maven私服
nginx-rc-93cdr             1/1       Running   15         4h
registry-rc-qbdfk          1/1       Running   12         4h
ui-rc-7e76i                1/1       Running   10         4h

[root@cu2 kubernetes]# kubectl get services nginx
NAME      CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
nginx     10.0.0.78    <none>        80/TCP,443/TCP   1d

访问nginx:

安装完了后,使用harbor.cfg配置文件里面的admin和密码进行登录。然后看看官网的操作文档 https://github.com/vmware/harbor/blob/master/docs/user_guide.md

现在PUSH要加 --insecure-registry 参数,还得重启docker太麻烦了。等下先弄https,搞好后添加证书直接push比较爽。

修改配置过程中遇到的一些问题

pvc在v1.2的时刻不支持selector。使用volumeName属性来代替。

巨坑,键名对不能用下划线、不能大写字母,到1.4才修复。

configmap~volumn用于创建volumns好像有问题,没有创建对应文件。

在1.5.3上面是可以生成的。。。囧,相比puppet的文档,k8s的文档真的差了十万八千里啊!!!

1
2
3
4
[root@k8s kube-deploy]# kubectl logs nginx-rc-fr52v
https.crt
https.key
nginx.conf

后面看到nginx的v1.2用了secrets修改后也不行。

其实就是docker版本老的不支持shared,其实在kubelet的容器里面是创建了对应的文件的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# docker logs
I0316 08:22:19.729825   13206 volumes.go:279] Used volume plugin "kubernetes.io/configmap" to mount config
I0316 08:22:19.729860   13206 configmap.go:118] Setting up volume config for pod cfe8b3f6-09fb-11e7-bdde-020047eb000e at /var/lib/kubelet/pods/cfe8b3f6-09fb-11e7-bdde-020047eb000e/volumes/kubernetes.io~configmap/config
I0316 08:22:19.729915   13206 volumes.go:279] Used volume plugin "kubernetes.io/empty-dir" to mount wrapped_config
...
I0316 08:22:19.733309   13206 configmap.go:145] Received configMap default/harbor-ui-config containing (30) pieces of data, 3739 total bytes
I0316 08:22:19.733470   13206 atomic_writer.go:316] /var/lib/kubelet/pods/cfe8b3f6-09fb-11e7-bdde-020047eb000e/volumes/kubernetes.io~configmap/config: current paths:   [app.conf private_key.pem]
I0316 08:22:19.733493   13206 atomic_writer.go:328] /var/lib/kubelet/pods/cfe8b3f6-09fb-11e7-bdde-020047eb000e/volumes/kubernetes.io~configmap/config: new paths:       [app.conf private_key.pem]
I0316 08:22:19.733502   13206 atomic_writer.go:331] /var/lib/kubelet/pods/cfe8b3f6-09fb-11e7-bdde-020047eb000e/volumes/kubernetes.io~configmap/config: paths to remove: map[]
I0316 08:22:19.733552   13206 atomic_writer.go:136] pod default/ui-rc-psjzs volume config: no update required for target directory /var/lib/kubelet/pods/cfe8b3f6-09fb-11e7-bdde-020047eb000e/volumes/kubernetes.io~configmap/config

[root@cu3 config]# docker exec -ti b34c51260dda bash
root@cu3:/# ls -al /var/lib/kubelet/pods/cfe8b3f6-09fb-11e7-bdde-020047eb000e/volumes/kubernetes.io~configmap/config
total 4
drwxrwxrwt 3 root root  120 Mar 16 04:08 .
drwxr-xr-x 3 root root 4096 Mar 16 04:08 ..
drwxr-xr-x 2 root root   80 Mar 16 04:08 ..3983_16_03_04_08_50.565987072
lrwxrwxrwx 1 root root   31 Mar 16 04:08 ..data -> ..3983_16_03_04_08_50.565987072
lrwxrwxrwx 1 root root   15 Mar 16 04:08 app.conf -> ..data/app.conf
lrwxrwxrwx 1 root root   22 Mar 16 04:08 private_key.pem -> ..data/private_key.pem

最后放弃了,直接用脚本来创建文件,然后把文件拷贝到对应的机器。

1
2
[root@cu2 kubernetes]# cd harbor-make/kubernetes/
[root@cu2 kubernetes]# sh config.sh 

HTTPS

生成CA和证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
[root@cu2 kubernetes]# git clone https://github.com/OpenVPN/easy-rsa.git

https://github.com/OpenVPN/easy-rsa/blob/master/README.quickstart.md

[root@cu2 easyrsa3]# ll
total 56
-rwxr-xr-x 1 root root 35253 Mar 13 01:04 easyrsa
-rw-r--r-- 1 root root  4560 Mar 13 01:04 openssl-1.0.cnf
-rw-r--r-- 1 root root  8126 Mar 13 01:04 vars.example
drwxr-xr-x 2 root root  4096 Mar 13 01:04 x509-types
[root@cu2 easyrsa3]# ./easyrsa init-pki

init-pki complete; you may now create a CA or requests.
Your newly created PKI dir is: /data/kubernetes/easy-rsa/easyrsa3/pki

[root@cu2 easyrsa3]# ./easyrsa build-ca
Generating a 2048 bit RSA private key
.............................+++
..............................................+++
writing new private key to '/data/kubernetes/easy-rsa/easyrsa3/pki/private/ca.key.Nj5oHgfZC5'
Enter PEM pass phrase: 123456
Verifying - Enter PEM pass phrase: 123456
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:esw.cn

CA creation complete and you may now import and sign cert requests.
Your new CA certificate file for publishing is at:
/data/kubernetes/easy-rsa/easyrsa3/pki/ca.crt


[root@cu2 easyrsa3]# ./easyrsa gen-req cu nopass
Generating a 2048 bit RSA private key
..........+++
.................................+++
writing new private key to '/data/kubernetes/easy-rsa/easyrsa3/pki/private/cu.key.LQX3Dr2jG3'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [cu]:cu.esw.cn

Keypair and certificate request completed. Your files are:
req: /data/kubernetes/easy-rsa/easyrsa3/pki/reqs/cu.req
key: /data/kubernetes/easy-rsa/easyrsa3/pki/private/cu.key

[root@cu2 easyrsa3]# ./easyrsa sign-req server cu


You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.

Request subject, to be signed as a server certificate for 3650 days:

subject=
    commonName                = cu.esw.cn


Type the word 'yes' to continue, or any other input to abort.
  Confirm request details: yes
Using configuration from /data/kubernetes/easy-rsa/easyrsa3/openssl-1.0.cnf
Enter pass phrase for /data/kubernetes/easy-rsa/easyrsa3/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName            :PRINTABLE:'cu.esw.cn'
Certificate is to be certified until Mar 10 23:36:42 2027 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated

Certificate created at: /data/kubernetes/easy-rsa/easyrsa3/pki/issued/cu.crt


[root@cu2 easyrsa3]# 

这里得用签发server端证书,如果是client使用时会报错: v2 ping attempt failed with error: Get https://cu.esw.cn/v2/: x509: certificate specifies an incompatible key usage


[root@cu2 easyrsa3]# tree .
.
├── easyrsa
├── openssl-1.0.cnf
├── pki
│   ├── ca.crt
│   ├── certs_by_serial
│   │   └── 01.pem
│   ├── index.txt
│   ├── index.txt.attr
│   ├── index.txt.old
│   ├── issued
│   │   └── cu.crt
│   ├── private
│   │   ├── ca.key
│   │   └── cu.key
│   ├── reqs
│   │   └── cu.req
│   ├── serial
│   └── serial.old
├── vars.example
└── x509-types
    ├── ca
    ├── client
    ├── COMMON
    └── server

6 directories, 18 files

重新执行以下上面的步骤,配置关联比较多。https和http请求地址会有冲突。

重新配置后,把ca.cert拷贝到docker节点,然后登录、创建项目、提交项目即可。最开始有帖操作的代码,这里不重复了。

–END

Comments