Hadoop 1 Vs Hadoop 2

Limitation of hadoop 1

  

  ü No horizontal scalability of NameNode

·        As a cluster size grows there is a bottleneck for NameNode metadata

  ü Does not supports NameNode High-availability

·        If NameNode is single point of contact, if it fails then whole HDFS process is down

  ü Overburdened Job Tracker

·        As Schedule job, Monitor Job, if Task Tracker fails then reschedule, Resource manage are done by this process alone

  ü Does not support multi-tenancy

·        Means different(Map-reduce, Streaming job, Interactive job) jobs on same resource at a same time

Hadoop 2

    1.     HDFS Federation

·        We can have more than one NameNode’s each NameNode will have namespace associated with it.

·        Each NameNode’s can save blocks across the DataNode’s under it’s namespace

·        In other words, NameNode-1 can access all those blocks in DataNode’s, which belongs to the namespace of NameNode-1

    2.     Multi-Tenancy –through, Capacity scheduler

·        Different types of jobs are organized in different queue

·        FIFO is followed within the queue

·        Each queue shares % of cluster

·        Priority can be set to each queue

    3.     YARN –Yet Another Resource Negotiator

·        It has Resource Manager on Mater, which has two components

§  Scheduler- 

          § Responsible for allocating resources to  various application subject to constraint of capacity

§  Application Manager-

§  Accepting job submission

§  Negotiating the Container for executing application specific application master

§  Provide service for restarting application master container on failure

·        Node Manager

§  Monitor Resource usage, CPU, Memory, Disk, Network

§  Report same to Resource Manager / scheduler

·         Application Master

§  Per application ,Application master

§  Negotiating appropriate resource container from the scheduler tracking status monitor process