Integrating LVM with Hadoop and providing Elasticity to Data Node Storage

♠ Elasticity Task

♦(A) — Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

♣(B) — Increase or Decrease the Size of Static Partition in Linux.

♦©— Automating LVM Partition using Python-Script.

Hadoop provides massive storage for any kind of data, enormous processing power, and the ability to handle virtually limitless concurrent tasks or jobs.

By default Data node share all their storage to the NameNode But we can restrict the Space the storage of the data node. By using the concept of the LVM

LVM(Logical Volume Management); This provides the flexibility to FileSystem

STEP 1:- First we have made the Partition and that partition can be converted into the physical volume

……………………………………………………………………………………….

Here I have 5 partition from which I have selected 2 for volume group

Now we can that there is two partitions as I make second similar to the first one

STEP 2:- First we have to convert the partition in the physical volume as only physical volume share storage to the Volume group . We get volume from the Volume group.

STEP 3:- Now we have to make a volume group between these two partition.

Our Volume group created

STEP 4:- Now we have to create the logical volume with have to make inode table for this we have to format finally we have to share this volume as datanode for Hadoop Cluster.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — —

— — — — — — — — — — — — — — — — — — — — — — — — — — — — —

This will create the inode (Contain Information about everything which present in the hard disk OS check the Inode table to know the location of the file in the hard disk without inode table )table and now we have to mount this partition with one of the folders before that we have to enable a driver

Now Create CONFIGURATION OF NAMENODE

UPDATE THE /etc/hadoop/hdfs-site.xml

……………………………………………………………………………………….

Now we have to the format the /namenode for the same reason as we did before (Inode table)

Now we have to start Name Node service and Data-Node service

Now we can the the space of the share by data-Node to the name node

Now we have to increase the size Dynamically

……………………………………………………………………………………..

TASK :- 3 Automating LVM Partition using Python-Script

Step 1 : First we will run python program that name is LVM_Automation.py

Step 2 : Then we will enter the name of hard disk & create physical volum

After Creating PV , then we will the PV .

Step 4 : Create Logical Volume & the name of LV is LV_name1 & th size of LV is 10GB’

Step 5 : After create LV then we want extend the size of the LV that is 3GB

…………………………………………………………………………………………………………

— — — — — — — — — — — — — — — — — — — — — — — — — — — —

Thank You So Much For Reading This Article Of Integrating LVM with Hadoop and providing Elasticity to Data Node Storage, i hope you Like it, Keep Sharing…