Winse Blog

走走停停都是风景, 熙熙攘攘都向最好, 忙忙碌碌都为明朝, 何畏之.

配置TEZ-UI

tez-ui很早就出来了，荒废了很多时间。今天才把它配置出来，效果挺不错的，和spark-web差不多。

记录了在hive-1.2.1上配置tez-0.7.0过程，配置运行hadoop2.6.3-timeline以及为tez添加tez-ui特性的步骤。

编译tez-0.7.0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

[hadoop@cu2 apache-tez-0.7.0-src]$ mvn package -Dhadoop.version=2.6.3 -DskipTests

[INFO] Reactor Summary:
[INFO] 
[INFO] tez ................................................ SUCCESS [  0.831 s]
[INFO] tez-api ............................................ SUCCESS [  6.580 s]
[INFO] tez-common ......................................... SUCCESS [  0.124 s]
[INFO] tez-runtime-internals .............................. SUCCESS [  0.676 s]
[INFO] tez-runtime-library ................................ SUCCESS [  1.378 s]
[INFO] tez-mapreduce ...................................... SUCCESS [  0.989 s]
[INFO] tez-examples ....................................... SUCCESS [  0.105 s]
[INFO] tez-dag ............................................ SUCCESS [  2.391 s]
[INFO] tez-tests .......................................... SUCCESS [  0.187 s]
[INFO] tez-ui ............................................. SUCCESS [02:23 min]
[INFO] tez-plugins ........................................ SUCCESS [  0.017 s]
[INFO] tez-yarn-timeline-history .......................... SUCCESS [  0.595 s]
[INFO] tez-yarn-timeline-history-with-acls ................ SUCCESS [  0.316 s]
[INFO] tez-mbeans-resource-calculator ..................... SUCCESS [  0.189 s]
[INFO] tez-tools .......................................... SUCCESS [  0.017 s]
[INFO] tez-dist ........................................... SUCCESS [ 16.554 s]
[INFO] Tez ................................................ SUCCESS [  0.015 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:55 min
[INFO] Finished at: 2016-01-12T19:08:50+08:00
[INFO] Final Memory: 63M/756M
[INFO] ------------------------------------------------------------------------

tez嵌入到hive

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

// 上传tez程序到hdfs
[hadoop@cu2 ~]$ cd sources/apache-tez-0.7.0-src/tez-dist/target/
[hadoop@cu2 target]$ hadoop fs -mkdir -p /apps/tez-0.7.0
[hadoop@cu2 target]$ hadoop fs -put tez-0.7.0.tar.gz /apps/tez-0.7.0/

// TEZ_CONF_DIR = HADOOP_CONF_DIR
[hadoop@cu2 ~]$ cd hadoop-2.6.3/etc/hadoop/
[hadoop@cu2 hadoop]$ vi tez-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez-0.7.0/tez-0.7.0.tar.gz</value>
</property>

</configuration>

// 本地tez jars加入HADOOP_CLASSPATH
[hadoop@cu2 apache-tez-0.7.0-src]$ cd tez-dist/target/
archive-tmp/              maven-archiver/           tez-0.7.0/                tez-0.7.0-minimal.tar.gz  tez-0.7.0.tar.gz          tez-dist-0.7.0-tests.jar  
[hadoop@cu2 apache-tez-0.7.0-src]$ cd tez-dist/target/
[hadoop@cu2 target]$ mv tez-0.7.0 ~/

[hadoop@cu2 ~]$ vi apache-hive-1.2.1-bin/conf/hive-env.sh

// 多个jline版本 http://stackoverflow.com/questions/28997441/hive-startup-error-terminal-initialization-failed-falling-back-to-unsupporte
export HADOOP_USER_CLASSPATH_FIRST=true
export TEZ_HOME=/home/hadoop/tez-0.7.0
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_HOME/*:$TEZ_HOME/lib/*

// http://stackoverflow.com/questions/26988388/hive-0-14-0-not-starting [/tmp/hive on HDFS should be writable. Current permissions are: rwxrwxr-x]
// hive.metastore.warehouse.dir  hive.exec.scratchdir
[hadoop@cu2 hive]$ rm -rf /tmp/hive
[hadoop@cu2 hive]$ hadoop fs -rmr /tmp/hive
// 或者修改权限 hadoop fs -chmod 777 /tmp/hive

启用/使用tez

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

[hadoop@cu2 hadoop]$ cat ~/hive/conf/hive-site.xml 
...
<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>

</configuration>

[hadoop@cu2 hive]$ bin/hive
...
hive> select count(*) from t_ods_access_log2;
Query ID = hadoop_20160112200359_f8be3d1c-9adc-42c0-abb9-2643dfef2cc7
Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id application_1452600034599_0001)

--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ..........   SUCCEEDED      1          1        0        0       0       0
Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0
--------------------------------------------------------------------------------
VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 20.83 s    
--------------------------------------------------------------------------------
OK
67
Time taken: 27.823 seconds, Fetched: 1 row(s)

部署/启动hadoop-timeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

[hadoop@cu2 hadoop]$ vi etc/hadoop/yarn-site.xml 
...
<property>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
</property>

<property>
  <name>yarn.timeline-service.hostname</name>
  <value>hadoop-master2</value>
</property>

<property>
  <name>yarn.timeline-service.http-cross-origin.enabled</name>
  <value>true</value>
</property>

<property>
  <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
  <value>true</value>
</property>

[hadoop@cu2 hadoop]$ for h in hadoop-slaver1 hadoop-slaver2 hadoop-slaver3 ; do rsync -vaz --exclude=logs hadoop-2.6.3 $h:~/ ; done

[hadoop@cu2 hadoop]$ sbin/yarn-daemon.sh start timelineserver

[hadoop@cu2 hadoop]$ sbin/stop-all.sh
[hadoop@cu2 hadoop]$ sbin/start-all.sh

部署tez-ui

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

// 放置tez-ui
[hadoop@cu2 target]$ cd ../../tez-ui/
[hadoop@cu2 tez-ui]$ cd target/
[hadoop@cu2 target]$ ll
total 1476
drwxrwxr-x 3 hadoop hadoop    4096 Jan 12 19:08 classes
drwxrwxr-x 2 hadoop hadoop    4096 Jan 12 19:08 maven-archiver
drwxrwxr-x 8 hadoop hadoop    4096 Jan 12 19:08 tez-ui-0.7.0
-rw-rw-r-- 1 hadoop hadoop    3058 Jan 12 19:08 tez-ui-0.7.0-tests.jar
-rw-rw-r-- 1 hadoop hadoop 1491321 Jan 12 19:08 tez-ui-0.7.0.war
[hadoop@cu2 target]$ mv tez-ui-0.7.0 ~/

// 部署tez-ui
[hadoop@cu2 ~]$ cd apache-tomcat-7.0.67/conf/
修改端口为9999
[hadoop@cu2 apache-tomcat-7.0.67]$ vi conf/server.xml 

[hadoop@cu2 apache-tomcat-7.0.67]$ cd conf/Catalina/localhost/
[hadoop@cu2 localhost]$ vi tez-ui.xml

[hadoop@cu2 apache-tomcat-7.0.67]$ bin/startup.sh 

// tez添加tez-ui功能
[hadoop@cu2 hive]$ vi ~/hadoop-2.6.3/etc/hadoop/tez-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez-0.7.0/tez-0.7.0.tar.gz</value>
</property>

<property>
<name>tez.history.logging.service.class</name>
<value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

<property>
<name>tez.tez-ui.history-url.base</name>
<value>http://hadoop-master2:9999/tez-ui/</value>
</property>

</configuration>

再运行一遍hive，查询一两个SQL。

最终效果：

参考

–END

Comments