How to contribute limited amount of storage as a slave to the cluster?

For this task, I have used the AWS cloud.

So, let’s get started…

First, we have to launch two instances using Amazon Linux image, one for the master and another for the data/slave node.

Next, we will create EBS volume of 1GiB and attach it to the instance named as “Slave”.

Now, we will remote login to the slave node and can check the EBS volume is attached to it or not using the following command:-

fdisk -l

Next, we will create partition on the EBS volume which we have attached to the slave instance earlier.

Here, I have created the partition of 512 MiB. We can see the device /dev/xvdf1 created.

We know that to solve the problem of BigData, we share the storage of Data-node to Name-Node. So, for storing any data in this partition, we have to first format it.

To format the partition use the following command:-

mkfs.ext4 /dev/xvdf1

Next, we have to mount it on the same directory of data-node we will be using in Hadoop Cluster. To mount the partition on desired directory, run the following command:-

mount /dev/xvdf1 /dn2

After mounting, we can see the size using the following command:-

df -hT

Now, we have to configure the hdfs-site.xml file and core-site.xml file in name node and data node.

In name node ,we have we have to format /nn directory we created and after that start the service of name node using hadoop-daemon.sh start namenode.

Now, we have to configure the same files in data node also in the similar way, just remember to give IP of name node in core-site.xml file.

Start the service of data node as well.

Now, let’s see the status of hadoop cluster using the following command:-

dfsadmin -report

Finally, we have set the limitation to data-node storage size.

Thanks for reading :)