About
An HDFS cluster consists of:
- a single NameNode (the head node) managing the file system. The NameNode is the arbitrator and repository for all HDFS metadata.
- a number of DataNodes, usually one per node in the cluster performing IO operations on behalves of the namenode.
HDFS has then a head/worker architecture.
A typical deployment has:
- a dedicated machine that runs only the NameNode software.
- other machines that runs one instance of the DataNode software.
The system is designed in such a way that user data never flows through the NameNode.
Articles Related
Management
Print the topology
See the option -printTopology of dfsadmin. It will display a tree of racks and datanodes attached to the tracks as viewed by the NameNode.
Report
See the option -report of dfsadmin.
hdfs dfsadmin -D "fs.default.name=hdfs://10.10.6.20/" -report
Configured Capacity: 2532916322304 (2.30 TB)
Present Capacity: 2351330590629 (2.14 TB)
DFS Remaining: 2351325233152 (2.14 TB)
DFS Used: 5357477 (5.11 MB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 10.10.6.14:30010 (wn2-ax.internal.cloudapp.net)
Hostname: wn2-.ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1785765 (1.70 MB)
Non DFS Used: 15721173083 (14.64 GB)
DFS Remaining: 785616138240 (731.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018
Name: 10.10.6.6:30010 (wn1-ax.internal.cloudapp.net)
Hostname: wn1-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1789952 (1.71 MB)
Non DFS Used: 18485723136 (17.22 GB)
DFS Remaining: 782851584000 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018
Name: 10.10.6.5:30010 (wn0-ax.internal.cloudapp.net)
Hostname: wn0-ax.internal.cloudapp.net
Decommission Status : Normal
Configured Capacity: 844305440768 (786.32 GB)
DFS Used: 1781760 (1.70 MB)
Non DFS Used: 18479804416 (17.21 GB)
DFS Remaining: 782857510912 (729.09 GB)
DFS Used%: 0.00%
DFS Remaining%: 92.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Mon Apr 09 14:57:19 UTC 2018