ivan's wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== HDFS install ======

Kapcsolódó oldalak: \\
https://ccp.cloudera.com/display/CDH4DOC/Software+Configuration+for+Quorum-based+Storage \\
https://ccp.cloudera.com/display/CDHDOC/ZooKeeper+Installation \\
https://ccp.cloudera.com/display/CDH4DOC/HDFS+High+Availability+Initial+Deployment \\
https://ccp.cloudera.com/display/CDH4DOC/Deploying+HDFS+on+a+Cluster#DeployingHDFSonaCluster-FormattingtheNameNode


Cloudera repo csomag letöltés:
https://ccp.cloudera.com/display/CDH4DOC/CDH4+Installation#CDH4Installation-DebianPackage

dpkg -i cdh4-repository_1.0_all.deb ; apt-get update \\
apt-get install openjdk-7-jre-headless:amd64

===== General configuration =====
.bashrc: \\
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 \\
export ZK_HOME=/usr/lib/zookeeper

===== Zookeeper install =====
A következő 5 gépre kell feltelepíteni: mgmt1, mgmt2, stg1, stg2, stg3

apt-get install zookeeper-server

szerver inicializálás: \\
zookeeper-server-initialize --force \\
chown -R zookeeper:zookeeper /var/lib/zookeeper/

service start: \\
service zookeeper-server start

===== Zookeeper configuration =====
/etc/zookeeper/conf/zk-auth.txt: \\
digest:vodstorage01-zkfcs:<password>

chown root:hdfs /etc/zookeeper/conf/zk-auth.txt \\
chmod 640 /etc/zookeeper/conf/zk-auth.txt

java -cp $ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.5-cdh4.2.0.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider vodstorage01-zkfcs:<password>

Minta output (a végén a digest jelszó):
<code>
...->vodstorage01-zkfcs:Ez4VnDSTGPKnPJHGhzp+2W74=
</code>

/etc/zookeeper/conf/zk-acl.txt: \\
digest:vodstorage01-zkfcs:<digest password in output>:rwcda

chown root:hdfs /etc/zookeeper/conf/zk-acl.txt \\
chmod 640 /etc/zookeeper/conf/zk-acl.txt


===== Journal install =====
apt-get install hadoop-hdfs-journalnode

===== Hadoop general configuration =====
/etc/default/hadoop: \\
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

===== Namenode install =====
apt-get install hadoop-hdfs-namenode \\
aptüget install hadoop-hdfs-zkfc
===== Namenode configuration =====
/etc/hadoop/conf/core-site.xml
<code>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://vodstorage01</value>
</property>
</code>

Első namenode-on: \\
su hdfs -c "hadoop namenode -format" \\
su hdfs -c "hdfs namenode -initializeSharedEdits"

Második namenode-on: \\
root@mgmt1:~# rsync -av /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/ root@mgmt2:/var/lib/hadoop-hdfs/cache/hdfs/dfs/name/ \\
Bug: https://issues.apache.org/jira/browse/HDFS-3752 \\
su hdfs -c "hdfs namenode -bootstrapStandby"

===== Datanode install =====
apt-get install hadoop-hdfs-datanode

===== HDFS kliens install =====
apt-get install hadoop-client

===== HDFS FUSE install =====
apt-get install hadoop-hdfs-fuse

with all dependency: \\
apt-get install hadoop-hdfs-fuse  hadoop-client hadoop-mapreduce hadoop-0.20-mapreduce libssl0.9.8

libssl0.9.8 is available in squeeze, need to build one for wheezy

FUSE mount: \\
<code>
hadoop-fuse-dfs dfs://<namenode> <mount point>
hadoop-fuse-dfs dfs://172.28.28.11:8020 /srv/users/default/content
</code>
ivan's wiki

User Tools

Site Tools

Page Tools
