HDFS - URI

> Database > (Apache) Hadoop > Hadoop Distributed File System (HDFS)

1 - About

Advertising

3 - Syntax

The URI format is [scheme://][authority]/path where:

Example:

hadoop fs -cat hdfs://namenodehost1/file1 hdfs://namenodehost2/file2

where:

4 - Management

4.1 - Scheme

The default is specified in the core-site.xml file.

List:

  • hdfs for HDFS
  • file for the local file system: file:/// (default)
  • HFTP FS,
  • S3 FS
  • wasb - Azure Blob Storage
  • adls - Azure Data Lake Storage
Advertising

4.2 - Modify

4.2.1 - Core-site.xml

If you want to modify it, add the following to Hadoop - core-site.xml

core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Get with hdfs

hdfs getconf -confKey fs.defaultFS
wasb://container@account.blob.core.windows.net

4.2.2 - Hadoop Cli

Every configuration can be change at the command line with the D option.

Example:

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /

4.2.3 - Java

Configuration conf = new Configuration();
String hdfsUri = "hdfs://mycluster/";
conf.set("fs.defaultFS", hdfsUri);
FileSystem fileSystem = FileSystem.get(URI.create(hdfsUri), conf);

with the following import

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
Advertising