User Tools

Site Tools


dwsc:hdfs:install

This is an old revision of the document!


HDFS install

General configuration

.bashrc:
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export ZK_HOME=/usr/lib/zookeeper

Zookeeper install

A következő 5 gépre kell feltelepíteni: mgmt1, mgmt2, stg1, stg2, stg3

apt-get install zookeeper-server

szerver inicializálás:
zookeeper-server-initialize –force
chown -R zookeeper:zookeeper /var/lib/zookeeper/

service start:
service zookeeper-server start

Zookeeper configuration

/etc/zookeeper/conf/zk-auth.txt:
digest:vodstorage01-zkfcs:<password>

chown root:hdfs /etc/zookeeper/conf/zk-auth.txt
chmod 750 /etc/zookeeper/conf/zk-auth.txt

java -cp $ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.5-cdh4.2.0.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider vodstorage01-zkfcs:<password>

Minta output (a végén a digest jelszó):

...->vodstorage01-zkfcs:Ez4VnDSTGPKnPJHGhzp+2W74=

/etc/zookeeper/conf/zk-acl.txt:
digest:vodstorage01-zkfcs:<digest password in output>:rwcda

chown root:hdfs /etc/zookeeper/conf/zk-acl.txt
chmod 750 /etc/zookeeper/conf/zk-acl.txt

Journal install

Hadoop general configuration

/etc/default/hadoop:
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

Namenode install

apt-get install hadoop-hdfs-namenode

Namenode configuration

/etc/hadoop/conf/core-site.xml

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://vodstorage01</value>
</property>

Datanode install

apt-get install hadoop-hdfs-datanode

HDFS kliens install

apt-get install hadoop-client

HDFS FUSE install

apt-get install hadoop-hdfs-fuse

with all dependency:
apt-get install hadoop-hdfs-fuse hadoop-client hadoop-mapreduce hadoop-0.20-mapreduce libssl0.9.8

libssl0.9.8 is available in squeeze, need to build one for wheezy

FUSE mount:

hadoop-fuse-dfs dfs://<namenode> <mount point>
hadoop-fuse-dfs dfs://172.28.28.11:8020 /srv/users/default/content
dwsc/hdfs/install.1364885965.txt.gz · Last modified: 2013/04/02 08:59 by ivan