Integrating LVM With Hadoop
Hey guys hope you all are doing good in today's article we are going to see what is lvm, how to attach; VM storage to the Hadoop cluster so that we can increase storage on the fly as per the requirement we needed.
What is Lvm?
Lvm is also known as a logical volume manager basically we create physical devices and make one volume group of all the devices and from that volume group, we can easily create the partition as much as we want as shown below.
First, we have to attach a hard disk to the system as shown below
in the above image, I have attached two hard disk of size 8gib and 20gib respectively after attaching hard disk we have to create them into physical volume by the following command
pvcreate disk name
Once we create all disk into physical volume we can easily create our new volume group from that physical volume.
once you create physical volume you can check them by the following command.
pvdisplay disk name
To create a new volume group from physical volume use the following command.
vgcreate volume name disk name
Once you create the volume group to check volume group use the following command.
vgdisplay volumename
Once we check and verify that our volume group is created we can now make a partition from that volume group called lv.
To create a new lv from the volume group use the following command
lvcreate — — name you want to give — size you want to give volume group name
Once you create the lv from the volume group to check it use the following command.
lvdisplay volumegrp/lvname
After creating and verifying lvgroup we have to format that part so that we can mount it to a folder and that folder will be our namenode here.
To format and mount the folder to that part we use the following command
mkfs.ext4 diskpath you want to format =>for formating the partition
mount disk path folder path =>to mount the partition to folder
df -h => to check which folder is mount to which partition
Now we can see that first when we give dn1 it is in root storage that is of 20gib so Hadoop cluster has 20gib as shown below.
After attaching the data node folder to the lv partition we can see that space is reduced to 9 gib.
To extend the portion use the following command.
lvextend — — size to which you want to extend disk name
After resizing the partition we have to format the newly added partition so by using the following command.
resize2fs disk path
The result can be seen in the above picture.
Some extra commands
vgextend volume name physical device=> to extend existing volume group
Guys, here we come to the end of this blog I hope you all like it and found it informative.
Guys below is a fully automated python script for the above long process
https://github.com/guptaadi123/lvm-automation.git
Guys follow me for such amazing blogs and if have any review then please let me know I will keep those points in my mind next time while writing blogs. If want to read more such blog to know more about me here is my website link https://sites.google.com/view/adityvgupta/home.Guys Please do not hesitate to keep 👏👏👏👏👏 for it (An Open Secret: You can clap up to 50 times for a post, and the best part is, it wouldn’t cost you anything), also feel free to share it across. This really means a lot to me.
