Winse Blog

走走停停都是风景, 熙熙攘攘都向最好, 忙忙碌碌都为明朝, 何畏之.

Hadoop安装与升级-(2)2.2升级到2.6

升级的命令很简单，但是不要瞎整！升级就一个命令就搞定了！

部署2.6.3

1
2
3
4
5
6
7
8
9
10
11
12
13
14

[hadoop@hadoop-master1 ~]$ tar zxvf hadoop-2.6.3.tar.gz 

[hadoop@hadoop-master1 ~]$ cd hadoop-2.6.3/share/
[hadoop@hadoop-master1 share]$ rm -rf doc/

[hadoop@hadoop-master1 hadoop-2.6.3]$ rm -rf lib/native/*

#// 拷贝四个配置文件到hadoop-2.6.3
[hadoop@hadoop-master1 hadoop-2.6.3]$ cd etc/hadoop/
[hadoop@hadoop-master1 hadoop]$ cp -f ~/hadoop-2.2.0/etc/hadoop/*-site.xml ./
[hadoop@hadoop-master1 hadoop]$ cp -f ~/hadoop-2.2.0/etc/hadoop/slaves ./

[hadoop@hadoop-master1 hadoop]$ cd 
[hadoop@hadoop-master1 ~]$ for h in hadoop-master2 hadoop-slaver1 hadoop-slaver2 hadoop-slaver3 ; do rsync -vaz --delete --exclude=logs hadoop-2.6.3 $h:~/ ; done

升级（最佳方式）

直接使用upgrade选项启动dfs即可。（secondarynamenode不要单独操作来升级，反正就是执行upgrade启动dfs就好了）。

1
2
3
4
5

[hadoop@hadoop-master1 hadoop-2.6.3]$ sbin/start-dfs.sh -upgrade

// 2.2和2.6都没有这个命令
// hadoop dfsadmin -upgradeProgress status
hadoop dfsadmin -finalizeUpgrade

参考[Hadoop: The Definitive Guide/Chapter 10. Administering Hadoop/Maintenance/Upgrades]

瞎整1

1
2
3
4

# 先停集群
[hadoop@hadoop-master1 hadoop-2.2.0]$ sbin/stop-dfs.sh

[hadoop@hadoop-master1 hadoop-2.6.3]$ sbin/start-dfs.sh

直接在原来的2.2基础上启动，datanode启动没问题，但是namenode报错：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

[hadoop@hadoop-master1 hadoop-2.6.3]$ less logs/hadoop-hadoop-namenode-hadoop-master1.log 
...
2016-01-07 08:05:23,582 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: 
File system image contains an old layout version -47.
An upgrade to version -60 is required.
Please restart NameNode with the "-rollingUpgrade started" option if a rolling upgrade is already started; or restart NameNode with the "-upgrade" option to start a new upgrade.
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:232)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1022)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:741)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:538)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:597)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:764)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:748)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1441)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1507)
2016-01-07 08:05:23,583 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2016-01-07 08:05:23,585 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop-master1/172.17.0.1
************************************************************/

重新启动，使用upgrade选项启动：

1
2

[hadoop@hadoop-master1 hadoop-2.6.3]$ sbin/stop-dfs.sh
[hadoop@hadoop-master1 hadoop-2.6.3]$ sbin/start-dfs.sh -upgrade

或者还原到2.2：

1
2
3
4
5
6
7

# **所有**slaver节点的VERSION改回47
[hadoop@hadoop-slaver3 ~]$ vi /data/tmp/dfs/data/current/VERSION 
...
layoutVersion=-47

[hadoop@hadoop-master1 hadoop-2.2.0]$ sbin/stop-dfs.sh
[hadoop@hadoop-master1 hadoop-2.2.0]$ sbin/start-dfs.sh -rollback

原理

升级的时刻，首先备份原来的数据到previous目录下，升级后的放置到current目录下。namenode这样没啥大问题，但是datanode也是这样结构current和previous，那相当有问题，那数据量不是翻倍了？

查看数据后，发现一个名字的文件current和previous里面使用的是一个inode。也就是说用的是硬链接，数据只有一份！

1
2
3
4
5
6
7
8
9
10
11

[hadoop@hadoop-master1 hadoop-2.2.0]$ bin/hadoop fs -put *.txt /

[hadoop@hadoop-slaver3 ~]$ cd /data/tmp/dfs/data/

[hadoop@hadoop-slaver3 BP-1695500896-172.17.0.1-1452152050513]$ test current/finalized/subdir0/subdir0/blk_1073741825 -ef previous/finalized/blk_1073741825
[hadoop@hadoop-slaver3 BP-1695500896-172.17.0.1-1452152050513]$ echo $?
0
[hadoop@hadoop-slaver3 BP-1695500896-172.17.0.1-1452152050513]$ ls -i current/finalized/subdir0/subdir0/blk_1073741825 
142510 current/finalized/subdir0/subdir0/blk_1073741825
[hadoop@hadoop-slaver3 BP-1695500896-172.17.0.1-1452152050513]$ ls -i previous/finalized/blk_1073741825
142510 previous/finalized/blk_1073741825

–END

Comments