HDFS - DataNode

> Database > (Apache) Hadoop > Hadoop Distributed File System (HDFS)

1 - About

A dataNode is a HDFS process that manage storage attached to the nodes that they run on.

The DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction from the NameNode.

The files are on the dataNode not on the NameNode

Advertising

3 - Management

3.1 - List

with HDFS - DFSAdmin

hdfs dfsadmin -D "fs.default.name=hdfs://10.10.6.20/"  -report
Configured Capacity: 2532916322304 (2.30 TB)
Present Capacity: 2351330590629 (2.14 TB)
DFS Remaining: 2351325233152 (2.14 TB)
DFS Used: 5357477 (5.11 MB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
 
-------------------------------------------------
Live datanodes (3):
 
Name: 10.10.6.14:30010 (wn2-ax.internal.cloudapp.net)
Hostname: wn2-.ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1785765 (1.70 MB)
Non DFS Used: 15721173083 (14.64 GB)
DFS Remaining: 785616138240 (731.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018
 
 
Name: 10.10.6.6:30010 (wn1-ax.internal.cloudapp.net)
Hostname: wn1-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1789952 (1.71 MB)
Non DFS Used: 18485723136 (17.22 GB)
DFS Remaining: 782851584000 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018
 
 
Name: 10.10.6.5:30010 (wn0-ax.internal.cloudapp.net)
Hostname: wn0-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1781760 (1.70 MB)
Non DFS Used: 18479804416 (17.21 GB)
DFS Remaining: 782857510912 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018

3.2 - State

See HDFS DataNode Admin Guide

With HDFS - DFSAdmin see the options -getDatanodeInfo <datanode_host:ipc_port> : Get the information about the given datanode. This command can be used for checking if a datanode is alive.

3.3 - Shutdown

See the options -shutdownDatanode <datanode_host:ipc_port> [upgrade] of HDFS - DFSAdmin