HDFS - Configuration (hdfs-site.xml)

> Database > (Apache) Hadoop > Hadoop Distributed File System (HDFS)

1 - About

HDFS follows the same configuration scheme than the whole Hadoop platform. See Hadoop - Configuration

The configuration are split between two files:

  • hdfs-site.xml, which provides default behaviors for the HDFS client.
  • core-site.xml, which sets the default filesystem name.

The hdfs configurations are stored in:

Advertising

3 - Management

3.1 - Get a value

With the hdfs utility, to get the value of a key:

hdfs getconf -confKey key

Example with the file system URI. You get the value here of a wasb azure file system

hdfs getconf -confKey fs.defaultFS
wasb://container@account.blob.core.windows.net

3.2 - Directory

hdfs envvars | grep -i HADOOP_CONF_DIR
HADOOP_CONF_DIR='/usr/hdp/2.6.2.25-1/hadoop/conf'

See HDFS - Environment Variable

3.3 - Pass a value at runtime

  • Command line: You can pass a configuration at the command line with the D generic option.
hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /
  • Code API (Java)
Configuration conf = new Configuration();
String hdfsUri = "hdfs://mycluster/";
conf.set("fs.defaultFS", hdfsUri);
FileSystem fileSystem = FileSystem.get(URI.create(hdfsUri), conf);
Advertising