Winse Blog

走走停停都是风景, 熙熙攘攘都向最好, 忙忙碌碌都为明朝, 何畏之.

Jenkins Start Guide

从原始的Eclipse右键导出打包，到后面使用maven打包，就单自己一个人使用开发部署是完全没问题的。现在的jenkins是对工具的封装、可视化和自动化，对于团队合作还是有一定的作用的，时时刻刻告诉我们代码是可运行的。

但是如果一个很久前的项目，又需要新加/修改功能，一下子还捡不起来，不放心啊还得验证一把。还有就是，测试有时刻他们自己打包，不会的还的教她们使用工具，人家烦自己也累。

jenkins是一个持续集成的工具，原来也接触过，但是都没用起来，都是搞开发，大部分时刻都能自己搞定。当下由于情况比较特殊，很多代码都直接在生产改，测试环境就不顾上了，但是测试环境不能总是旧代码啊，就想着有个自动化的东西来进行部署。

主要就是完成一个代码自动化部署的工作：自己搭建一个jenkins，从oschina上拉代码，编译后部署到tomcat并重启。

安装Jenkins

1
2

wget http://mirrors.jenkins.io/war-stable/latest/jenkins.war
nohup java -jar jenkins.war --httpPort=8081 >/var/log/jenkins.log 2>&1 &

建立到oschina的无密钥登录

由于项目是私有的，通过https需要输入密码，还是git方式无密钥登录方式便捷一些。本地linux执行ssh-keygen，然后把 id_rsa.pub 的内容拷贝到项目的公钥处进行配置。可以参考文档： http://git.mydoc.io/?t=154712。

也可以结合 本地ssh-agent 和 ssh-forward 来弄。

配置项目

第一次登录需要进行一些配置，默认创建的admin密码会保存在 ~/.jenkins/secrets/initialAdminPassword 。（在初始化页面创建新用户报错，也不知道啥原因。登录后再建吧）

新版本的按照默认安装插件还不够，需要再添加一些。登录成功后，添加如下插件：

  • Deploy to container Plugin 把war发布到容器tomcat…
  • Nexus Artifact Uploader 上传jar到私服

  • Maven Integration plugin 集成maven

  • ThinBackup 备份也是有必要的，用的越久越是必要！！

配置maven：

自己下载个maven解压后，在jenkins - Global Tool Configuration上面配置maven地址即可（把 自动安装 的勾去掉就可以填地址了）

然后配置JOB：

  • 构建一个maven项目：填任务的名称，然后点击左下角的OK
  • 源码管理git: 填写地址，然后新增Credentials - SSH Username with private key - From the Jenkins master ~/.ssh 起一个容易区分的名字
  • 构建触发器： Build periodically - 0 0 * * * 每天一次
  • Build：web/pom.xml ; clean package -Papp,dist -DskipTests 就是mvn命令的一串参数
  • Post Steps: Run only if build succeeds - Execute Shell
1
2
3
4
5
6

/opt/apache-tomcat-8.0.26/bin/shutdown.sh ; sleep 1 
rm -rf /opt/apache-tomcat-8.0.26/webapps/app.war 
cp $WORKSPACE/web/app/target/app.war /opt/apache-tomcat-8.0.26/webapps 
cd /opt/apache-tomcat-8.0.26/webapps ; ./deploy.sh 
BUILD_ID=dontKillMe nohup /opt/apache-tomcat-8.0.26/bin/startup.sh & 
sleep 3

注意：这里的BUILD_ID挺有意思的！！！

也可以配置 构建后操作 把包发布到tomcat manager（呵呵，无奈原始包webapps下的都被我删了)，就用脚本弄了。

构建

完成上面的操作后，就可以执行跑一次看看效果了。其他的还有很多功能：权限等。

多节点(集群)

如果只有一台jenkins的时刻，远程发布项目一般都scp或者使用tomcat-manager进行处理，如果把部署的机器作为jenkins node的话，就可以把部署的任务放到该节点本地跑，就不需要考虑远程部署的问题了。

配置节点： windows

1
2

用法   : 只允许运行绑定到这台机器的Job
  启动方法 ： Launch slave agents via SSH（在主机配置无密钥登录，填用户、Private key：From the Jenkins master ~/.ssh）

配置好后，在界面点击 Launch agent，会把agent拷贝到机器并启动agent。

然后任务的话，配置 Restrict where this project can be run 。

参考

–END

追生产的一次优化

注：最后就是升级到JDK8(哭笑)…

最近闲得慌啊，本来不是自己职能范围内的。但是看着一台机器每天负载50+的跑，不舒服，就想去折腾折腾把负载降下来。

进程图：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

top - 08:01:24 up 1203 days,  9:06,  4 users,  load average: 31.41, 32.97, 32.38
Tasks: 569 total,  11 running, 558 sleeping,   0 stopped,   0 zombie
Cpu(s): 20.1%us, 68.1%sy,  0.0%ni,  6.0%id,  0.1%wa,  0.0%hi,  5.7%si,  0.0%st
Mem:  49420852k total, 31831356k used, 17589496k free,   358748k buffers
Swap: 33791992k total,   519332k used, 33272660k free, 18614472k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                   
 2340 omc       20   0 29.2g 8.5g  11m S 598.1 18.1   3436:40 java                     
31349 omc       20   0 8071m 563m  11m S 348.4  1.2   1735:33 java                     
28147 omc       20   0 15.5g 1.5g  14m S 341.9  3.2   1959:42 java                     
   61 root      20   0     0    0    0 S 48.9  0.0  83728:05 ksoftirqd/14              
   73 root      20   0     0    0    0 S 48.2  0.0  82342:12 ksoftirqd/17              
    9 root      20   0     0    0    0 S 46.9  0.0  85312:03 ksoftirqd/1               
   13 root      20   0     0    0    0 S 46.6  0.0  84297:57 ksoftirqd/2               
   25 root      20   0     0    0    0 S 45.3  0.0  82811:49 ksoftirqd/5               
   89 root      20   0     0    0    0 S 45.3  0.0  84608:31 ksoftirqd/21              
   65 root      20   0     0    0    0 S 44.9  0.0  83475:48 ksoftirqd/15              
   17 root      20   0     0    0    0 R 44.6  0.0  83990:21 ksoftirqd/3               
   57 root      20   0     0    0    0 S 44.6  0.0  84625:38 ksoftirqd/13              
   33 root      20   0     0    0    0 R 44.0  0.0  80537:34 ksoftirqd/7               
    4 root      20   0     0    0    0 R 43.3  0.0  81489:54 ksoftirqd/0               
   41 root      20   0     0    0    0 R 42.0  0.0  82651:17 ksoftirqd/9               
   37 root      20   0     0    0    0 S 40.0  0.0  82636:26 ksoftirqd/8               
   85 root      20   0     0    0    0 S 39.7  0.0  84557:49 ksoftirqd/20              
   21 root      20   0     0    0    0 S 38.7  0.0  83271:24 ksoftirqd/4               
   53 root      20   0     0    0    0 R 36.1  0.0  82083:15 ksoftirqd/12              
   45 root      20   0     0    0    0 R 35.8  0.0  86230:39 ksoftirqd/10              
   93 root      20   0     0    0    0 R 35.4  0.0  86416:12 ksoftirqd/22              
   69 root      20   0     0    0    0 R 35.1  0.0  82726:46 ksoftirqd/16              
   29 root      20   0     0    0    0 S 34.8  0.0  78415:22 ksoftirqd/6               
   77 root      20   0     0    0    0 R 33.1  0.0  82419:34 ksoftirqd/18              
   81 root      20   0     0    0    0 S 30.2  0.0  80141:58 ksoftirqd/19              
   97 root      20   0     0    0    0 R 21.3  0.0  85993:03 ksoftirqd/23              
   49 root      20   0     0    0    0 S 21.0  0.0  86742:13 ksoftirqd/11              
28418 nobody    20   0  855m  32m 1144 S 20.7  0.1  72:23.66 gmetad

线程图：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

top - 08:03:20 up 1203 days,  9:08,  4 users,  load average: 31.07, 32.36, 32.23
Tasks: 940 total,  31 running, 909 sleeping,   0 stopped,   0 zombie
Cpu(s): 20.0%us, 70.0%sy,  0.0%ni,  4.6%id,  0.0%wa,  0.0%hi,  5.4%si,  0.0%st
Mem:  49420852k total, 31845576k used, 17575276k free,   358776k buffers
Swap: 33791992k total,   519332k used, 33272660k free, 18615376k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND             
28174 omc       20   0 15.5g 1.5g  14m R 59.9  3.2 307:28.86 java                
28203 omc       20   0 15.5g 1.5g  14m S 55.7  3.2 272:43.21 java                
 2416 omc       20   0 29.2g 8.5g  11m R 55.4 18.1 274:31.07 java                
31384 omc       20   0 8071m 563m  11m R 53.7  1.2 240:45.47 java                
 2409 omc       20   0 29.2g 8.5g  11m S 53.1 18.1 245:56.03 java                
28197 omc       20   0 15.5g 1.5g  14m S 52.4  3.2 279:04.35 java                
 2406 omc       20   0 29.2g 8.5g  11m R 51.8 18.1 249:00.25 java                
28208 omc       20   0 15.5g 1.5g  14m R 51.8  3.2 300:50.49 java                
 2412 omc       20   0 29.2g 8.5g  11m S 51.5 18.1 232:11.81 java                
 2415 omc       20   0 29.2g 8.5g  11m R 51.5 18.1 234:57.25 java                
 2391 omc       20   0 29.2g 8.5g  11m R 51.1 18.1 301:52.48 java                
28175 omc       20   0 15.5g 1.5g  14m R 51.1  3.2 299:18.11 java                
31383 omc       20   0 8071m 563m  11m R 50.8  1.2 242:23.43 java                
16662 omc       20   0 29.2g 8.5g  11m R 49.5 18.1   3:26.22 java                
31381 omc       20   0 8071m 563m  11m R 49.5  1.2 237:05.25 java                
   41 root      20   0     0    0    0 S 48.9  0.0  82652:00 ksoftirqd/9         
   17 root      20   0     0    0    0 S 47.9  0.0  83990:59 ksoftirqd/3         
   65 root      20   0     0    0    0 S 47.9  0.0  83476:26 ksoftirqd/15        
 2408 omc       20   0 29.2g 8.5g  11m R 47.9 18.1 249:43.27 java                
31382 omc       20   0 8071m 563m  11m R 47.9  1.2 237:07.76 java                
   49 root      20   0     0    0    0 S 47.3  0.0  86743:04 ksoftirqd/11        
   89 root      20   0     0    0    0 R 46.6  0.0  84609:14 ksoftirqd/21        
   81 root      20   0     0    0    0 S 46.3  0.0  80142:39 ksoftirqd/19        
   61 root      20   0     0    0    0 R 46.0  0.0  83728:50 ksoftirqd/14        
31376 omc       20   0 8071m 563m  11m R 45.3  1.2 306:00.66 java                
   33 root      20   0     0    0    0 R 45.0  0.0  80538:15 ksoftirqd/7         
31385 omc       20   0 8071m 563m  11m R 45.0  1.2 272:52.36 java                
   13 root      20   0     0    0    0 S 44.7  0.0  84298:42 ksoftirqd/2         
   73 root      20   0     0    0    0 S 43.7  0.0  82342:53 ksoftirqd/17        
   53 root      20   0     0    0    0 R 43.4  0.0  82083:54 ksoftirqd/12        
   97 root      20   0     0    0    0 S 43.4  0.0  85993:53 ksoftirqd/23        
   45 root      20   0     0    0    0 R 42.4  0.0  86231:24 ksoftirqd/10        
   77 root      20   0     0    0    0 S 42.1  0.0  82420:20 ksoftirqd/18        
 2407 omc       20   0 29.2g 8.5g  11m R 41.1 18.1 240:01.88 java                
 2410 omc       20   0 29.2g 8.5g  11m R 40.8 18.1 227:49.76 java                
   85 root      20   0     0    0    0 R 40.5  0.0  84558:37 ksoftirqd/20        
28196 omc       20   0 15.5g 1.5g  14m R 38.2  3.2 276:56.00 java                
   29 root      20   0     0    0    0 S 37.9  0.0  78416:08 ksoftirqd/6         
   37 root      20   0     0    0    0 S 37.9  0.0  82637:15 ksoftirqd/8         
 2411 omc       20   0 29.2g 8.5g  11m R 37.9 18.1 247:22.02 java                
 2360 omc       20   0 29.2g 8.5g  11m S 37.6 18.1 179:49.10 java                
 2413 omc       20   0 29.2g 8.5g  11m S 36.9 18.1 233:48.03 java                
   69 root      20   0     0    0    0 R 36.3  0.0  82727:24 ksoftirqd/16        
    4 root      20   0     0    0    0 R 35.6  0.0  81490:34 ksoftirqd/0         
31369 omc       20   0 8071m 563m  11m R 35.6  1.2 192:32.42 java                
   21 root      20   0     0    0    0 S 35.0  0.0  83272:02 ksoftirqd/4         
28167 omc       20   0 15.5g 1.5g  14m R 33.7  3.2 197:02.78 java                
   25 root      20   0     0    0    0 S 27.2  0.0  82812:29 ksoftirqd/5         
   93 root      20   0     0    0    0 R 25.9  0.0  86416:55 ksoftirqd/22

按照网络上的文档，查cpu时间很长的、占用很高的线程，然后拿着ID转成16进程到jstack里面去对：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

[omc@cu-omc1 ~]$ jstack 28147
2017-06-01 08:07:13
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode):
...
"Timer-0" daemon prio=10 tid=0x00007fbd84850000 nid=0x6e27 in Object.wait() [0x00007fbe31f98000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x0000000767760360> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"schedulerFactory_QuartzSchedulerThread" prio=10 tid=0x00007fbd843f6000 nid=0x6e26 in Object.wait() [0x00007fbe32099000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:311)
        - locked <0x0000000767770098> (a java.lang.Object)

"schedulerFactory_Worker-2" prio=10 tid=0x00007fbd848cd000 nid=0x6e25 in Object.wait() [0x00007fbe3219a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000007677ebb38> (a java.lang.Object)

"schedulerFactory_Worker-1" prio=10 tid=0x00007fbd848b3000 nid=0x6e24 in Object.wait() [0x00007fbe3229b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000007677ec710> (a java.lang.Object)
...
"GC task thread#17 (ParallelGC)" prio=10 tid=0x00007fbe64035000 nid=0x6e06 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007fbe6411f800 nid=0x6e0e waiting on condition 

JNI global references: 321

[omc@cu-omc1 ~]$ echo "obase=16;28203" | bc
6E2B
[omc@cu-omc1 ~]$ cat | while read id ; do echo "obase=16;$id" | bc ; done <<EOF
28174
28203
28197
28208
28175
28196
28167
EOF

6E0E
6E2B
6E25
6E30
6E0F
6E24
6E07

基本都是sleep，wait的线程占用cpu很大。并且导致了系统cpu软中断处理进程ksoftirqd占用了大部分系统资源。系统不停的在处理上下文，负载奇高：

ksoftirqd 不知道干嘛的，/proc/interrupts 看不懂，查了sleep和wait的区别，strace、iostat、jmap、jstack、jstat、vmstat、pidstat、还有看到内存补齐的一些文章，反正就是找不到北。

一开始以为是quartz的问题，对比了其他机器的quartz应用，有怀疑过版本问题（quartz-1.8.6, 2.2.0）；有试着去减少simplethreadpool的默认线程数（org.quartz.threadPool.threadCount），CPU占用是会少一点点，但是ksoftirqd还是压力很大，系统还是很大部分消耗在上下文切换，路子不对。

问题环境：

  • Red Hat Enterprise Linux Server release 6.3 (Santiago)/2.6.32-279.el6.x86_64
  • Spring + quartz-2.2.½
  • jdk1.7.0_17

完全没辙，不是功能代码的问题啊。搞到12点，困死了，回去睡个觉。今天一早起来，想想，不如换个 JDK8 试试吧（按照部署要求jdk放local目录下，要ROOT密码的昨晚就没动）。我勒个去，重启了感觉世界都变亮了。上下文切换cs 1w不到，us、sy基本忽略不计啊。

1
2
3
4
5
6
7
8
9
10
11

$ vmstat -a 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 0  0 517160 25052176 12486824 10402340    0    0     2    31    0    0  4  9 86  0  0
 0  0 517160 25052732 12486972 10401996    0    0     0  4676 4548 4806  0  0 99  0  0
 1  0 517160 25052468 12486972 10402356    0    0     0     0 4044 4419  0  0 99  0  0
 0  1 517160 25053664 12486852 10401992    0    0     0  8100 7608 5311  0  0 95  4  0
 0  1 517160 25054800 12486852 10402084    0    0     0  8228 7847 5408  1  1 95  4  0
 0  1 517160 25054924 12486852 10402380    0    0     0  8200 8075 4929  0  0 95  4  0
 1  1 517160 25054868 12486852 10402112    0    0     0  7484 7898 5754  1  1 94  4  0
 2  1 517160 25055544 12486848 10402148    0    0     0  8224 7537 4428  0  0 95  4  0

好吧，以后优化的第一步就是换JDK .__. 。就像优化数据库第一步就建索引 V.V 。应该是JDK8对object.wait调用linux系统调用进行了优化。

有点意思

us过高 当us值过高时，表示运行的应用消耗了大部分的cpu。在这种情况下，对于java应用而言，最重要的是找到具体消耗cpu的线程所执行的代码，可以采用如下方法。

首先通过linux命令top命令查看us过高的pid值

通过top -Hp pid查看该pid进程下的线程的cpu消耗状况，得到具体pid值

将pid值转化为16进制，这个转化后的值对应nid值的线程

通过jstack pid grep -C 20 “16进制的值” 命令查看运行程序的线程信息

该线程就是消耗cpu的线程，在采样时须多执行几次上述的过程，以确保找到真实的消耗cpu的线程。

java应用造成us过高的原因主要是线程一直处于可运行的状态Runnable，通常是这些线程在执行无阻塞、循环、正则或纯粹的计算等动作造成。 另外一个可能会造成us过高的原因是频繁的gc。如每次请求都需要分配较多内存，当访问量高时就导致不断的进行gc，系统响应速度下降， 进而造成堆积的请求更多，消耗的内存严重不足，最严重的时候会导致系统不断进行FullGC，对于频繁的gc需要通过分析jvm内存的消耗来查找原因。

sy过高 当sy值过高时，表示linux花费了更多的时间在进行线程切换。java应用造成这种现象的主要原因是启动的线程比较多， 且这些线程多处于不断的阻塞（例如锁等待，io等待）和执行状态的变化过程中，这就导致了操作系统要不断的切换执行的线程， 产生大量的上下文切换。在这种情况下，对java应用而言，最重要的是找出不断切换状态的原因， 可采用的方法为通过kill -3 pid 或jstack -l pid的方法dump出java应用程序的线程信息，查看线程的状态信息以及锁信息， 找出等待状态或锁竞争过多的线程。

strace -T -r -c -p pid pstack pid trace -p tid

–END

Hive on Spark预测性执行BUG一枚

为了平复难以平复的痛苦，难以掩饰的激动，把这次遇到并解决的记录下。尽管最终解决的patch是官网的: Hive on Spark gives incorrect results when speculation is on。

版本说明下：

  • hive-1.2.1
  • spark-1.3.1

在没有启动spark.speculation前，有个别任务执行非常慢，非常之讨厌。而启用预测性执行后，时不时任务会有些会失败，让人很烦躁。但是吧，也不算故障，说来也奇怪，重启下后再次查询问题就不出现了，也就没太在意。

今天数据量比较大，并且是上头检查。妈蛋，搞成了故障，没得办法，必须把原因找出来了。下来就帖日志了：

应用SQL查询报错日志：啥也看不到，就知道Hive查询报错，只能拿着时间去查Hive日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

[ERROR] 14:19:56.685 [RMI TCP Connection(7)-192.168.31.11] c.e.z.h.s.BaseHiveQueryService | Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
        at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392)
        at org.apache.hive.jdbc.HivePreparedStatement.executeQuery(HivePreparedStatement.java:109)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at com.esw.zhfx.hbase.service.BaseHiveQueryService.listIteratorInternal(BaseHiveQueryService.java:101)
        at com.esw.zhfx.hbase.service.BaseHiveQueryService.listIterator(BaseHiveQueryService.java:80)
        at com.esw.zhfx.hbase.QueryService.getAccessLogIterator(QueryService.java:140)
        at com.esw.zhfx.hbase.QueryService$$FastClassByCGLIB$$a60bf6f7.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:50)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        at com.esw.zhfx.hbase.QueryService$$EnhancerByCGLIB$$9a4ab584.getAccessLogIterator(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy22.getAccessLogIterator(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

HIVE服务日志：rename错了，但是也好像看不到啥。知道那个节点有问题了，去查节点日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

2017-05-23 14:19:20,509 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) - 2017-05-23 14:19:20,508 WARN  [task-result-getter-1] scheduler.TaskSetManager: Lost task 2199.1 in stage 2.0 (TID 4517, hadoop-slaver41): java.lang.IllegalStateException: Hit error while closing operators - failing tree: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0 to: hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_tmp.-ext-10001/002199_0.snappy
2017-05-23 14:19:20,509 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:195)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:106)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at scala.collection.Iterator$class.foreach(Iterator.scala:727)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:114)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:114)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1576)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1576)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.spark.scheduler.Task.run(Task.scala:64)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at java.lang.Thread.run(Thread.java:722)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) - Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0 to: hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_tmp.-ext-10001/002199_0.snappy
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
2017-05-23 14:19:20,510 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
2017-05-23 14:19:20,511 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  at org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:172)
2017-05-23 14:19:20,511 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -  ... 15 more
2017-05-23 14:19:20,511 INFO  client.SparkClientImpl (SparkClientImpl.java:run(569)) -

Task错误节点错误日志：这日志没啥。重名，拿名称去查namenode日志看看是啥子？

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

17/05/23 14:19:18 INFO exec.FileSinkOperator: FS[24]: records written - 0
17/05/23 14:19:18 INFO exec.FileSinkOperator: Final Path: FS hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_tmp.-ext-10001/002199_0
17/05/23 14:19:18 INFO exec.FileSinkOperator: Writing to temp file: FS hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0
17/05/23 14:19:18 INFO exec.FileSinkOperator: New Final Path: FS hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_tmp.-ext-10001/002199_0.snappy
17/05/23 14:19:19 INFO compress.CodecPool: Got brand-new compressor [.snappy]
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0 to: hdfs://zfcluster/hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_tmp.-ext-10001/002199_0.snappy
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
        at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616)
        at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
        at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
        at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
        at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:630)
        at org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:172)
        at org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58)
        at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:106)
        at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:114)
        at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:114)
        at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1576)
        at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1576)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
        at org.apache.spark.scheduler.Task.run(Task.scala:64)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

Namenode日志：有点点线索了，分配了两次，导致了第二个任务写入的时刻报错！

1
2
3
4
5
6

[hadoop@hadoop-master2 ~]$ grep '_tmp.002199' hadoop/logs/hadoop-hadoop-namenode-hadoop-master2.log.1
2017-05-23 14:19:01,591 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0. BP-1414312971-192.168.32.11-1392479369615 blk_1219124858_145508182{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ad2eac59-1e38-4019-a5ac-64c465366186:NORMAL:192.168.32.93:50010|RBW], ReplicaUnderConstruction[[DISK]DS-90c8cbe3-fd70-4ad7-938a-4248b4435df7:NORMAL:192.168.32.136:50010|RBW], ReplicaUnderConstruction[[DISK]DS-9da76df9-47f0-4e25-b375-e1bf32f4cf52:NORMAL:192.168.36.58:50010|RBW]]}
2017-05-23 14:19:14,939 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0 is closed by DFSClient_attempt_201705231411_0000_m_001585_0_1316598676_51
2017-05-23 14:19:20,368 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0. BP-1414312971-192.168.32.11-1392479369615 blk_1219125517_145508841{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4d4c90f0-1ddf-4800-b33a-e776e58dc744:NORMAL:192.168.32.61:50010|RBW], ReplicaUnderConstruction[[DISK]DS-948cd823-5a4c-4673-8ace-99f02a26522b:NORMAL:192.168.32.52:50010|RBW], ReplicaUnderConstruction[[DISK]DS-7818addb-3881-446e-abb3-2c178be6bb63:NORMAL:192.168.32.176:50010|RBW]]}
2017-05-23 14:19:20,478 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0 is closed by DFSClient_attempt_201705231411_0000_m_001345_1_1292482540_51
2017-05-23 14:19:20,480 WARN org.apache.hadoop.hdfs.StateChange: DIR* FSDirectory.unprotectedRenameTo: failed to rename /hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_task_tmp.-ext-10001/_tmp.002199_0 to /hive/scratchdir/hadoop/64801461-94aa-4e17-afee-494e77b49998/hive_2017-05-23_14-18-38_278_4858034238266422677-2/-mr-10000/.hive-staging_hive_2017-05-23_14-18-38_278_4858034238266422677-2/_tmp.-ext-10001/002199_0.snappy because destination exists

好了，看到这里，驴脑袋还没怀疑到是预测性执行导致的问题。当时想为啥会出现同一个文件名呢：SPARK ON HIVE多个stage执行导致的? 但是重启后报一样的错误，002199是哪里产生，怎么产生的？

MAP太多了000000又循环了一轮？看了执行的map数也就2600啊，不应该啊。

那么这个文件名是哪里产生的呢？然后就搞了下远程调试：没啥用，错误是在task上发生的，调试hive-driver没啥用，但是有意外收获

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

[hadoop@hadoop-master2 hive]$ DEBUG=true bin/hive
Listening for transport dt_socket at address: 8000

Logging initialized using configuration in file:/home/hadoop/apache-hive-1.2.1-bin/conf/hive-log4j.properties
hive> set hive.execution.engine=spark; '查询之前需要设置下引擎，故障得先处理。搞成默认的mr跑是成功的
hive>                                  'SQLSQLSQL...执行刚报错的SQL
Query ID = hadoop_20170523173748_7660d9fb-9683-4792-8315-a51f6dcc270b
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Spark Job = 48a8668b-1c59-4cbf-b1e2-e19612ee77d0

Query Hive on Spark job[0] stages:
0

Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
2017-05-23 17:38:15,730 Stage-0_0: 0/2609
2017-05-23 17:38:16,739 Stage-0_0: 0(+159)/2609
...
2017-05-23 17:39:23,182 Stage-0_0: 2162(+447)/2609
2017-05-23 17:39:24,188 Stage-0_0: 2167(+608)/2609
2017-05-23 17:39:25,195 Stage-0_0: 2201(+836,-1)/2609
2017-05-23 17:39:26,201 Stage-0_0: 2215(+832,-2)/2609
2017-05-23 17:39:27,207 Stage-0_0: 2227(+820,-2)/2609
2017-05-23 17:39:28,213 Stage-0_0: 2250(+797,-2)/2609
2017-05-23 17:39:29,219 Stage-0_0: 2280(+767,-2)/2609
2017-05-23 17:39:30,224 Stage-0_0: 2338(+709,-2)/2609
2017-05-23 17:39:31,230 Stage-0_0: 2350(+696,-3)/2609
2017-05-23 17:39:32,236 Stage-0_0: 2359(+684,-6)/2609
2017-05-23 17:39:33,243 Stage-0_0: 2363(+676,-10)/2609
2017-05-23 17:39:34,249 Stage-0_0: 2365(+673,-12)/2609
...

有报错了，赶紧去web页面看了下结果，好家伙，全部是Speculation的报错：

在结合前面的namenode的日志，基本就走到正道上面。然后 hive spark speculation 一股沟，没错第一条就是hive官网的bug啊。

然后就是打patch修改HivePairFlatMapFunction，验证是OK的。至少原来出错的语句完美跑完。

总结下

就是前段集成攻城狮把网络回环的问题处理了，导致网络状态好的不要不要的啊！把那些有备用10M网卡全部停了，集群的机器的网络好了N倍。第二个就是数据量实在大，其实speculation有启动，但是最先完成的还是先启动的，又没有把预测执行kill掉并且还运行完了最终还保存到同名文件。最后让我又一次体验了一把找开源软件BUG激情四射的半天。

记录聊以慰藉！！

other : SparkClientImpl LeaseExpiredException No lease on File does not exist

–END

Puppet批量自动化部署实战

断断续续使用Puppet近一年，多次体验到Puppet的强大：SSH更新、需ROOT权限批量处理等等。这次集群新上架了又爽了一把。把整个过程记录下来，方便今后参考。

运维的同事也想了解puppet，在docker容器上安装了一遍，把具体的内容附上：expect+puppet.txt

这次操作是对以前零零碎碎积累的一次检验和温习。需要用到的工具比较多：

  • RPM打包、本地YUM仓库 - RPMBUILD、CREATEREPO
  • SSH无密钥登录 - EXPECT&FOR
  • 时间同步、host配置 - SCP、SSH&FOR
  • 创建用户、新用户无密钥等 - PUPPET
  • ssh_known_hosts - PUPPETDB
  • rhel.repo、gmond、时区设置 - PUPPET

远程配置机器首先当然是进行无密钥登录的设置，这样才能进行批量操作，不然几百台机器每次都需要干预太烦人、工作量太大。无密钥登录使用原来写好的EXPECT脚本，使用FOR循环执行，等待结果即可。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

[root@hadoop-master1 ~]# cat ssh-copy-id.expect 
#!/usr/bin/expect  

## Usage $0 [user@]host password

set host [lrange $argv 0 0];
set password [lrange $argv 1 1] ;

set timeout 30;

spawn ssh-copy-id $host ;

expect {
  "(yes/no)?" { send yes\n; exp_continue; }
  "password:" { send $password\n; exp_continue; }
}

exec sleep 1;

# 用for，不要用while
for h in `cat /etc/hosts | grep -v '^#' | grep slaver | grep -E '\.36\.|\.37\.' | awk '{print $2}' ` ; do 
  ./ssh-copy-id.expect $h 'PASSWD';
done

做好无密钥登录，拷贝 /etc/hosts, /etc/cron.daily/ntp.cron, /etc/yum.repos.d/puppet.repo 到全部的新机器。这里puppet.repo是自己编译搭建的私有仓库（具体编译配置步骤查看puppet分类下的文章），通过 yum install mcollective-plugins-simple 就可以把mcolletive和puppet-agent安装好。把所有步骤封装到一个prepare.sh脚本，内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#!/bin/sh

# must be hostname!!
HOSTS="$@"
PASSWD=${PASSWD:-'root'}
PUPPETSERVER="hadoop-master1"

for h in $HOSTS ; do ./ssh-copy-id.expect $h "$PASSWD" ; done

for h in $HOSTS ; do
scp /etc/hosts $h:/etc ;
scp /etc/yum.repos.d/puppet.repo $h:/etc/yum.repos.d/ ;
scp /etc/cron.daily/ntp.cron $h:/etc/cron.daily/ ;

ssh $h '
#ntpdate cu-omc1 #着重注意
rm -rf /etc/yum.repos.d/CentOS-*
yum install mcollective-plugins-simple -y
' ;

scp /etc/puppetlabs/mcollective/server.cfg $h:/etc/puppetlabs/mcollective/
ssh $h "
sed -i '/HOSTNAME/ {
i \
HOSTNAME=$h
d
} ' /etc/sysconfig/network
hostname $h

echo -e '\n\n[agent]\nserver = $PUPPETSERVER\ncertname=$h' > /etc/puppetlabs/puppet/puppet.conf
chkconfig mcollective on
service mcollective start
"

done

然后执行 ./prepare.sh hadoop-slaver{200..500} 就可以了。

接下来重点讲讲PUPPET配置的编写。

首先根据当前需要创建的用户、组把创建用户的配置写好：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

[root@hadoop-master1 ~]# puppet resource -e group hadoop
group { 'hadoop':
  ensure => 'present',
  gid    => '501',
}
[root@hadoop-master1 ~]# puppet resource -e user hadoop
user { 'hadoop':
  ensure           => 'present',
  gid              => '501',
  groups           => ['wheel'],
  home             => '/home/hadoop',
  password         => '$6$AfnA...uIhHC9I.',
  password_max_age => '99999',
  password_min_age => '0',
  shell            => '/bin/bash',
  uid              => '501',
}

添加require、groups，然后删除uid、gid。最后需要添加 managehome => true, 否则用户目录就不会自动创建：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

# 默认不创建用户目录
[root@hadoop-slaver200 ~]# su - hadoop
su: warning: cannot change directory to /home/hadoop: No such file or directory
-bash-4.1$ 

# 创建用户配置成品
group { 'hadoop':
  ensure => 'present',
}

user { 'hadoop':
  ensure           => 'present',
  groups           => ['hadoop', 'wheel'],
  home             => '/home/hadoop',
  password         => '$6$Af...IhHC9I.',
  password_max_age => '99999',
  password_min_age => '0',
  shell            => '/bin/bash',
  managehome       => true,
  require          => Group['hadoop'],
}

添加好用户后，就是把无密钥登录也让PUPPET来弄。其实就是把 id_rsa.pub 的内容写入都行机器的 authorized_keys ，PUPPET已经自带了这个类：ssh_authorized_key。把id_ras.pub的内容（中间的内容）赋值给 key 属性即可。

1
2
3
4
5
6
7
8
9
10
11
12

ssh_authorized_key {'root@hadoop-master1':
  user => 'root',
  type => 'ssh-rsa',
  key => 'AAAAB3NzaC1y...O1Q==',
}

ssh_authorized_key {'hadoop@hadoop-master1':
  user => 'hadoop',
  type => 'ssh-rsa',
  key => 'AAAAB3Nza...IZYPw==',
  require  => User['hadoop'],
}

无密钥登录比较容易，没有涉及到收集节点信息。仅仅把公钥写入新机器还不够，还得把 known_hosts 也处理好，不然第一次连接新机器都需要输入一下yes。内容如下：

1
2
3
4

[hadoop@hadoop-slaver200 ~]$ ssh hadoop-slaver202
The authenticity of host 'hadoop-slaver202 (192.168.36.59)' can't be established.
RSA key fingerprint is fe:7e:26:c4:56:ea:f4:21:61:82:6d:9b:4a:72:93:a4.
Are you sure you want to continue connecting (yes/no)?

正如上面官网介绍的，需要用到虚拟资源，自动把新机器指纹（fingerprint）写入到机器需要PUPPETDB的支持，安装配置又需要PGSQL的配合。需要耗费一番功夫，但是还是划得来的（具体安装步骤查看puppet分类下的文章）。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

if $hostname =~ /^hadoop-/ {

  $host_aliases = [ $ipaddress, $hostname ]
  
  # Export hostkeys from all hosts.
  @@sshkey { $::fqdn:
    ensure => present,
    host_aliases => $host_aliases,
    type => 'ssh-rsa',
    key => $sshrsakey,
  }
  
  if $hostname =~ /^hadoop-master/ {
    # realize all exported
    Sshkey <<| |>>
  }
  
}

先在所有slaver机器运行一遍 puppet agent -t ，然后再在master节点把收集的指纹写入到 /etc/ssh/ssh_known_hosts 。

这里说个插曲：机器的hosts和hostname是通过 FOR&SSH 命令来统一修改的，有些可能没有配置好导致机器的主机名有重复。通过执行配置known_hosts竟然帮我找出了hostname重复的机器，意外的收获。该问题的处理我是直接登录到PGSQL改了对应表的数据处理的。

到这里机器基本能用了。主机名、hosts、时间同步、hadoop用户以及master到该用户的无密钥登录都已经配置好了。

接下来把实战过程中安装gmond的步骤帖出来：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

$$ cd /etc/puppetlabs/code/environments/production/manifests/

$$ vi change_site.sh
#!/bin/sh

## Usage:
##  ./change_site.sh nrpe.site
##

[[ $# != 1 ]] && exit 1

cd $(cd $(dirname $0); pwd)

rm -rf site.pp
ln -s $1 site.pp

$$ vi pexec.sh
#!/bin/sh

## Usage:
##   ./pexec.sh /cu-ud/ sudo_revert.site 
##

case $# in
1)
  FUNC="$1"
  HOST_PARAM=
  ;;
2)
  FUNC="$2"
  HOST_PARAM="-I $1"
  ;;
*)
  while [ $# -gt 1 ] ; do 
    HOST_PARAM="$HOST_PARAM -I $1"
    shift
  done
  FUNC=$1
  ;;
esac

cd $(cd $(dirname $0); pwd)

./change_site.sh "$FUNC"

if [[ "$HOST_PARAM" != "" && ! "$HOST_PARAM" =~ */* ]] ; then
  mco shell $HOST_PARAM run -- `which puppet` agent -t
else
  mco puppet $HOST_PARAM runall 20
fi

由于机器增加比较多，且网络环境变的复杂化。把原来的2个分组修改成4个。不同的网络段和功能分别设置不同的广播端口。

1
2
3
4
5
6
7

$ ./pexec.sh /hadoop-slaver.$/ gmond.site 

# 采集数据的节点重启后，其他发送数据的节点貌似都需要重启。
$ screen
$ for ((i=1;i<=53;i++)); do  mco shell -I /hadoop-slaver${i}.$/ run -- ' service gmond restart ' ; done 
# 这个确认搞的很麻烦，
# 想通过ganglia-web获取数据然后判断是否有数据进行重启。

Ganglia删除某节点后，如果要从rrds上去掉改节点的信息，需要：重启对应收集的gmond，对应集群的rrds目录，然后重启gmetad。或者等够一段时间，gmetad会自动去掉。

总结

现在添加机器，直接连上puppetserver机器然后执行几个命令就可以搞定；

1
2
3
4
5
6

HOST=new-host-name 
# 无密钥登录和puppet/mco
PASSWD=new-host-root-password ./prepare.sh $HOST

./pexec.sh $HOST new-hadoop.site
./pexec.sh $HOST gmond.site # 当前需要到web界面确认新节点的数据是否被采集

–END

K8s Hadoop Deploy

折磨了一个多星期，最后还是调通了。折磨源于不自知，源于孤单，源于自负，后来通过扩展、查阅资料、请教同事顺利解决。简单部署可以查看README.md 。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

yum install docker-engine-1.12.6 docker-engine-selinux-1.12.6 -y

cd kube-deploy
vi hosts
vi k8s.profile
# 把deploy同步到其他实体机，同时把k8s.profile映射到/etc/profile.d
./rsync-deploy.sh

cd docker-multinode/
./master.sh or ./worker.sh

docker save gcr.io/google_containers/etcd-amd64:3.0.4 | docker-bs load
docker save quay.io/coreos/flannel:v0.6.1-amd64 | docker-bs load

cd kube-deploy/hadoop/kubenetes/
./prepare.sh
kubectl create -f hadoop-master2.yaml
kubectl create -f hadoop-slaver.yaml

Tip：其实使用一套配置就可以启动多个集群，在 kubectl create 后面加上 -n namespace 即可。

比如：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

[root@cu2 kubenetes]# kubectl create namespace hd1
[root@cu2 kubenetes]# kubectl create namespace hd2

[root@cu2 kubenetes]# ./prepare.sh hd1
[root@cu2 kubenetes]# kubectl create -f hadoop-master2.yaml -n hd1
[root@cu2 kubenetes]# kubectl create -f hadoop-slaver.yaml -n hd1
[root@cu2 kubenetes]# ./prepare.sh hd2
[root@cu2 kubenetes]# kubectl create -f hadoop-master2.yaml -n hd2
[root@cu2 kubenetes]# kubectl create -f hadoop-slaver.yaml -n hd2

[root@cu2 kubenetes]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
hd1           hadoop-master2                          1/1       Running   0          28s
hd1           slaver-rc-fdcsw                         1/1       Running   0          18s
hd1           slaver-rc-qv964                         1/1       Running   0          18s
hd2           hadoop-master2                          1/1       Running   0          26s
hd2           slaver-rc-0vdfk                         1/1       Running   0          17s
hd2           slaver-rc-r7g84                         1/1       Running   0          17s
...

现在想来其实就是 dockerd –ip-masq=false 的问题（所有涉及的dockerd都需要加）。 还有就是一台机器单机下的容器互相访问，源IP都错也是安装了openvpn所导致，对所有过eth0的都加了MASQUERADE。

根源就在于请求的源地址被替换，也就是iptables的转发进行了SNAT。关于iptables转发这篇文章讲的非常清晰；IPtables之四：NAT原理和配置 。

所遇到的问题

没加ip-masq之前，namenode收到datanode的请求后，源地址是flannel.0的ip: 10.1.98.0。

namenode对应的日志为：

1
2
3

2017-04-09 07:22:06,920 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(10.1.98.0, datanodeUuid=5086c549-f3bb-4ef6-8f56-05b1f7adb7d3, infoPort=50075, ipcPort=50020, storageInfo=lv=-56;cid=CID-522174fa-6e7b-4c3f-ae99-23c3018e35d7;nsid=1613705851;c=0) storage 5086c549-f3bb-4ef6-8f56-05b1f7adb7d3
2017-04-09 07:22:06,920 INFO org.apache.hadoop.net.NetworkTopology: Removing a node: /default-rack/10.1.98.0:50010
2017-04-09 07:22:06,921 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/10.1.98.0:50010

一开始以为是flannel的问题，换成yum安装，然后同时flannel把backend切换成vxlan后，还是一样的问题。

最后请教搞网络的同事，应该是请求的源地址被替换了，也就定位到iptables。然后通过查看文档，其实前面也有看到过对应的文章，但是看不明白不知道缘由。

iptables的部分相关信息：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

[root@cu2 ~]# iptables -S -t nat
...
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j OUTPUT_direct
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.1.34.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-75CPIAPDB4MAVFWI -s 10.1.40.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-75CPIAPDB4MAVFWI -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.1.40.3:53
-A KUBE-SEP-IWNPEB4T46P6VG5J -s 192.168.0.148/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-IWNPEB4T46P6VG5J -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-IWNPEB4T46P6VG5J --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.0.148:6443
-A KUBE-SEP-UYUINV25NDNSKNUW -s 10.1.40.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-UYUINV25NDNSKNUW -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.1.40.3:53
-A KUBE-SEP-XDHL2OHX2ICPQHKI -s 10.1.40.2/32 -m comment --comment "kube-system/kubernetes-dashboard:" -j KUBE-MARK-MASQ
-A KUBE-SEP-XDHL2OHX2ICPQHKI -p tcp -m comment --comment "kube-system/kubernetes-dashboard:" -m tcp -j DNAT --to-destination 10.1.40.2:9090
-A KUBE-SERVICES -d 10.0.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.0.0.95/32 -p tcp -m comment --comment "kube-system/kubernetes-dashboard: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XGLOHA7QRQ3V22RZ
-A KUBE-SERVICES -d 10.0.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.0.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-75CPIAPDB4MAVFWI
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-IWNPEB4T46P6VG5J --mask 255.255.255.255 --rsource -j KUBE-SEP-IWNPEB4T46P6VG5J
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-IWNPEB4T46P6VG5J
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-UYUINV25NDNSKNUW
-A KUBE-SVC-XGLOHA7QRQ3V22RZ -m comment --comment "kube-system/kubernetes-dashboard:" -j KUBE-SEP-XDHL2OHX2ICPQHKI

在dockerd服务脚本加上 --ip-masq=false 后，-A POSTROUTING -s 10.1.34.0/24 ! -o docker0 -j MASQUERADE 这一句就没有了，也就是不会进行源地址重写了，这样请求发送到namenode后还是datanode容器的IP。问题解决，原因简单的让人欲哭无泪啊。

写yaml遇到的一些其他问题：

当然还有很多其他的问题，这篇就写这么多，优化工作后面的弄好了再写。

中间过程步骤记录

主要就是记录心路历程，如果以后遇到同样的问题能让自己快速回想起来。如果仅仅为了部署，可以跳过该部分，直接后最后的常用命令。

记录下中间 通过yum安装etcd和flanneld 的过程。物理机安装flanneld会把配置docker环境变量（/run/flannel/subnet.env）加入启动脚本。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

安装docker-v1.12
https://docs.docker.com/v1.12/
https://docs.docker.com/v1.12/engine/installation/linux/centos/

# 删掉原来的
yum-config-manager --disable docker-ce*
yum remove -y docker-ce*

sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF

https://yum.dockerproject.org/repo/main/centos/7/Packages/
[root@cu3 ~]# yum --showduplicates list docker-engine | expand
docker-engine.x86_64             1.12.6-1.el7.centos                  dockerrepo

[root@cu3 yum.repos.d]# yum install docker-engine-1.12.6 docker-engine-selinux-1.12.6


https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/

cat > /etc/yum.repos.d/virt7-docker-common-release.repo <<EOF
[virt7-docker-common-release]
name=virt7-docker-common-release
baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/
gpgcheck=0
EOF

yum -y install --enablerepo=virt7-docker-common-release etcd flannel
yum -y install --enablerepo=virt7-docker-common-release flannel

- ETCD配置
[root@cu3 docker-multinode]# 
etcdctl mkdir /kube-centos/network
etcdctl set /kube-centos/network/config "{ \"Network\": \"10.1.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"

- FlANNEL
[root@cu3 ~]# cat /etc/sysconfig/flanneld
# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://cu3:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/kube-centos/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

[root@cu2 yum.repos.d]# systemctl daemon-reload

[root@cu2 yum.repos.d]# cat /run/flannel/subnet.env

[root@cu2 ~]# systemctl cat docker
...
# /usr/lib/systemd/system/docker.service.d/flannel.conf
[Service]
EnvironmentFile=-/run/flannel/docker

测试过程中有yaml配置中启动sshd，然后启动容器后，通过手动启动namenode、datanode的方式来测试：

1
2
3
4
5
6
7

cd hadoop-2.6.5
gosu hadoop mkdir /data/bigdata
gosu hadoop sbin/hadoop-daemon.sh start datanode 

cd hadoop-2.6.5/
gosu hadoop  bin/hadoop namenode -format 
gosu hadoop sbin/hadoop-daemon.sh start namenode

后来发现问题出在iptables后，又回到原来的docker-bootstrap启动，需要删除flannel.1的网络：

1
2
3
4
5
6

# yum安装flanneld后停止 https://kubernetes.io/docs/getting-started-guides/scratch/
ip link set flannel.1 down
ip link delete flannel.1
route -n

rm /usr/lib/systemd/system/docker.service.d/flannel.conf

开了防火墙的话，把容器的端加入到信任列表：

1
2
3
4
5

systemctl enable firewalld && systemctl start firewalld

firewall-cmd --zone=trusted --add-source=10.0.0.0/8 --permanent 
firewall-cmd --zone=trusted --add-source=192.168.0.0/16 --permanent 
firewall-cmd --reload

一些有趣的命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

查看用了哪些镜像

[root@cu2 /]# kubectl get pods --all-namespaces -o jsonpath="{..image}" |\
 tr -s '[[:space:]]' '\n' |\
 sort |\
 uniq -c
      2 gcr.io/google_containers/dnsmasq-metrics-amd64:1.0
      2 gcr.io/google_containers/exechealthz-amd64:1.2
     12 gcr.io/google_containers/hyperkube-amd64:v1.5.5
      2 gcr.io/google_containers/kube-addon-manager-amd64:v6.1
      2 gcr.io/google_containers/kubedns-amd64:1.9
      2 gcr.io/google_containers/kube-dnsmasq-amd64:1.4
      2 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0
    
      
修改默认kubectl的配置

[root@cu2 ~]# vi $KUBECONFIG 
apiVersion: v1
kind: Config
preferences: {}
current-context: default
clusters:
- cluster:
    server: http://localhost:8080
  name: default
contexts:
- context:
    cluster: default
    user: ""
    namespace: kube-system
  name: default
users: {}

如果kubectl没有下载，可以从镜像启动的容器里面获取

[root@cu2 docker-multinode]# docker exec -ti 0c0360bcc2c3 bash
root@cu2:/# cp kubectl /var/run/

[root@cu2 run]# mv kubectl /data/kubernetes/kube-deploy/docker-multinode/

获取容器IP

https://kubernetes.io/docs/user-guide/jsonpath/
[root@cu2 ~]# kubectl get pods -o wide -l run=redis -o jsonpath={..podIP}
10.1.75.2 10.1.75.3 10.1.58.3 10.1.58.2 10.1.33.3

网络共用: --net

docker run -ti --entrypoint=sh --net=container:8e9f21956469f4ef7e5b9d91798788ab83f380795d2825cdacae0ed28f5ba03b gcr.io/google_containers/skydns-amd64:1.0


格式化输出

kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}"  

[root@cu2 ~]# export POD_COL="custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount,CONTAINERS:.spec.containers[*].name,IP:.status.podIP,HOST:.spec.nodeName"
[root@cu2 ~]# kubectl get pods -o $POD_COL 

kubectl get po -l k8s-app=kube-dns -o=custom-columns=NAME:.metadata.name,CONTAINERS:.spec.containers[*].name

[root@cu2 kubernetes]# kubectl get po --all-namespaces -o=custom-columns=NAME:.metadata.name,CONTAINERS:.spec.containers[*].name

kubectl get po --all-namespaces {range .items[*]}{.metadata.name}{“\t”}{end}

备份

echo "$(docker ps  | grep -v IMAGE | awk '{print $2}' )
$(docker-bs ps | grep -v IMAGE | awk '{print $2}' )" | sort -u | while read image ; do docker save $image>$(echo $image | tr '[/:]' _).tar ; done

加Label

cat /etc/hosts | grep -E "\scu[0-9]\s" | awk '{print "kubectl label nodes "$1" hostname="$2}' | while read line ; do sh -c "$line" ; done

扩容

[root@cu2 kubernetes]# kubectl run redis --image=redis:3.2.8 
[root@cu2 kubernetes]# kubectl scale --replicas=9 deployment/redis

 echo " $( kubectl describe pods hadoop-master2 | grep -E "Node|Container ID" | awk -F/ '{print $NF}' | tr '\n' ' ' | awk '{print "ssh "$1" \rdocker exec -ti "$2" bash"}' ) "

测试DNS是否成功：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

[root@cu2 kube-deploy]# vi busybox.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

[root@cu3 kube-deploy]# kubectl create -f busybox.yaml 
pod "busybox" created
[root@cu3 kube-deploy]# kubectl get pods 
NAME      READY     STATUS              RESTARTS   AGE
busybox   0/1       ContainerCreating   0          11s
[root@cu3 kube-deploy]# kubectl get pods 
NAME      READY     STATUS    RESTARTS   AGE
busybox   1/1       Running   0          1m
[root@cu3 kube-deploy]# kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local

用容器的MYSQL的做客户端

kubectl run -it --rm --image=mysql:5.6 mysql-client -- mysql -h mysql -ppassword

小结一点：日志的重要性！

1
2
3
4
5

[root@cu2 kubenetes]# docker ps -a | grep kubelet
[root@cu2 kubenetes]# docker logs --tail=200 7432da457558

E0417 11:39:40.194844   22528 configmap.go:174] Couldn't get configMap hadoop/dta-hadoop-config: configmaps "dta-hadoop-config" not found
E0417 11:39:40.194910   22528 configmap.go:174] Couldn't get configMap hadoop/dta-bin-config: configmaps "dta-bin-config" not found

监控heapster的一些错误，还没调好

1
2
3
4
5
6
7
8
9
10
11

[root@cu2 ~]# kubectl exec -ti heapster-564189836-shn2q -n kube-system -- sh
/ # 
/ # 
没pod的数据
/ # /heapster --source=https://kubernetes.default --sink=log --heapster-port=8083 -v 10

E0329 10:11:53.823641       1 reflector.go:203] k8s.io/heapster/metrics/processors/node_autoscaling_enricher.go:100: Failed to list *api.Node: Get https://kubernetes.default/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout


$heapster/metrics
$heapster/api/v1/model/debug/allkeys

其他一些配置

1
2
3
4
5
6


other_args=" --registry-mirror=https://docker.mirrors.ustc.edu.cn "

--insecure-registry gcr.io 

iptables -S -t nat

其他一些资源

statefulset

–END