Multi Node hadoop set-up



  1. Assign host name to master,slaves and client 
  2. Install ssh
        -"sudo yum install openssh-server"
  3. Generate .ssh password less key 
        -"ssh-keygen"
        -press Enter - press Enter 
  4. Now we should add IP address and host name of the master,slave and client machine in /etc/hosts file 
  5. Share the generated key across all slave nodes
    •   "ssh-copy-id -i $HOME/.ssh/id.rsa.pub <user>@<hostname>"
  6. Setup JAVA on a Master Node
    • download JAVA from oracle and export the JAVA_HOME and PATH in .bashrc file then "source .bashrc"
  7. Setup hadoop on Master Node lets say /usr/local/hadoop is the HADOOP_HOME
    • download Hadoop from apache and export HADOOP_HOME and PATH in .bashrc file then "source .bashrc"
  8. Now we need to configure 6 files in $HADOOP_HOME/conf/ directory
    • -"sudo nano /usr/local/hadoop/conf/hadoop-env.sh"
       -add JAVA_HOME (export JAVA_HOME=/usr/local/java/jdk1.7.0_75)
    • "sudo nano /usr/local/hadoop/conf/core-site.xml"
       -add below entries, Note that NameNode  is the host name of the machine where  NameNode process of the hadoop HDFS is running. In my case NameNode is the master node Host name.
                     <configuration>
                         <property> 
                              <name>hadoop.tmp.dir</name>
                              <value>/usr/local/hadoop/tmp</value>
                       </property> 

                       <property> 
                              <name>fs.default.name</name> 
                              <value>hdfs://NameNode:54310</value>
                       </property> 
                   </configuration>
             
    • sudo mkdir -p /usr/local/hadoop/tmp
    • "sudo nano /usr/local/hadoop/conf/mapred-site.xml"
       -add below entries ,Note that NameNode  is the host name of the machine where JobTracker process of the hadoop MapReduce is running. 
       <configuration>
              <property> 
                      <name>mapred.job.tracker</name> 
                            <value>NameNode:54311</value>
              </property> 
       </configuration>
    • "sudo nano /usr/local/hadoop/conf/hdfs-site.xml"
       <configuration>
                <property>
                    <name>dfs.replication</name>
                    <value>1</value>
                </property>
                <property>
                    <name>dfs.block.size</name>
                    <value>67108864</value>
                </property>   
       <configuration>
    • Now we need to configure master file, here just we need to do is add the host name of the node where namenode and jobtracker process will run
    • Similarly add the hostnames in slaves file, so that datanode and tasktracker process will runs on those slave machine 
        9. Now we are ready with master machine so i will use the rsync command to sych up                  hadoop with the slave nodes and client node               
                  o   "rsync -avg /usr/local/hadoop/ <user>@<hosatname>/usr/local/hadoop/"
              -Note: here i followed the same directory structure in both master and slave machine
       
       10. Similarly we can also sync the JAVA from master machine with the slave machines 
o"rsync -avg /usr/local/java/jdk1.7.0_75/ <user>@<hosatname>/usr/local/java/jdk1.7.0_75/"
              -Note : here i maintained the same directory so no need of changing the JAVA_HOME        from /usr/local/hadoop/conf/hadoop-env.sh,
          o  But JAVA_HOME and HADOOP_HOME should be set properly in slave machine or we can  use rsync for .bashrc file as well.

      11. Now delete /usr/local/hadoop/tmp from all the nodes and from master node run 
               "hadoop namenode -format"
      
      12. Now we can star our hadoop muli node set-up by running "start-all.sh" from                      master machine 
      

              

































Comments

Post a Comment

Popular posts from this blog

Secondary NameNode check-pointing process

Image
Secondary namenode, whose purpose is to produce checkpoints of the primary’s in-memory filesystem metadata. The check pointing process proceeds as follows, 1. The secondary asks the primary to roll its edits file, so new edits go to a new file. 2. The secondary retrieves fsimage and edits from the primary (using HTTP GET). 3. The secondary loads fsimage into memory, applies each operation from edits, then creates a new consolidated fsimage file. 4. The secondary sends the new fsimage back to the primary (using HTTP POST). 5. The primary replaces the old fsimage with the new one from the secondary, and the old edits file with the new one it started in step 1. It also updates the fstime file to record the time that the checkpoint was taken. At the end of the process, the primary has an up-to-date fsimage file and a shorter edits file (it is not necessarily empty, as it may have received some edits while the checkpoint was being taken). It is possible for an administrator...
Read more

Failover and fencing

1.The transition from the active namenode to the standby is managed by a new entity in the system called the failover controller.  2.Failover controllers are pluggable, but the first implementation uses ZooKeeper to ensure that only one namenode is active.  3.Each namenode runs a lightweight failover controller process whose job it is to monitor its namenode for failures (using a simple heartbeating mechanism) and trigger a failover should a namenode fail. 4.Failover may also be initiated manually by an adminstrator, in the case of routine maintenance, for example. This is known as a graceful failover, since the failover controller arranges an orderly transition for both namenodes to switch roles. 5.In the case of an ungraceful failover, however, it is impossible to be sure that the failed namenode has stopped running. For example, a slow network or a network partition can trigger a failover transition, even though the previously active namenode is still runn...
Read more

Hadoop 1 Vs Hadoop 2

Limitation of hadoop 1      ü No horizontal scalability of NameNode ·         As a cluster size grows there is a bottleneck for NameNode metadata   ü Does not supports NameNode High-availability ·         If NameNode is single point of contact, if it fails then whole HDFS process is down   ü Overburdened Job Tracker ·         As Schedule job, Monitor Job, if Task Tracker fails then reschedule, Resource manage are done by this process alone   ü Does not support multi-tenancy ·         Means different(Map-reduce, Streaming job, Interactive job) jobs on same resource at a same time Hadoop 2     1.      HDFS Federation ·         We can have more than one NameNode’s each NameNode will have namespace associated with it. · ...
Read more