HDFS - Block

> Database > (Apache) Hadoop > Hadoop Distributed File System (HDFS)

1 - About

File System - Block in HDFS.

The block size can be changed by file.

Block are stored on a datanode and are grouped in block pool

Advertising

3 - Management

3.1 - Info

3.2 - Location

The location on where the blocks are stored is defined in hdfs-site.xml. Example:

hdfs-site.xml
<property>
	<name>dfs.datanode.data.dir</name>
	<value>file:/hadoop/data/dfs/datanode</value>
</property>

3.2.1 - Offset

3.3 - Size

A typical block size used by HDFS is 128 MB. Thus, an HDFS file is chopped up into 128 MB chunks.

hdfs-site.xml
<property>
  <name>dfs.blocksize</name>
  <value>134217728</value>
</property>
hdfs getconf -confKey dfs.blocksize
134217728
# of 128 Mb
Advertising

3.4 - Move

See the mover hdfs sub-command to move block replicas across storage types.

hdfs mover

3.5 - Failure

3.5.1 - Under-replicated

under-replicated block

  • web ui: The overview page gives you this information.

3.5.2 - Missing