Yarn - Log (Container, Application) - Tfile

> Database > (Apache) Hadoop > Yarn (Yet Another Resource Negotiator) - Hadoop Operating System

1 - About

The log of an application (ie a container when running).

Advertising

3 - Format

Application logs are not saved in text format. They are saved in a binary format called TFile.

A TFile format is a container of key value pairs, and it is not readable by humans.

4 - Management

4.1 - View

4.1.1 - Command Line

doc

yarn logs -applicationId <applicationId>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application> -containerId <containerId> -nodeAddress <worker-node-address>

where:

Code: LogsCLI.java

4.1.2 - Ui

With HortonWorks Ambari > Yarn > Quick Links:

Advertising

4.1.3 - Other tfile reader

tfile reader:

4.2 - Configuration

4.2.1 - Aggregation

Logs for all the containers belonging to a single Application are aggregated and written out to a single (possibly compressed) log file at a configured location.

yarn.log-aggregation-enable to true

If disabled, node manager will keep the logs locally

4.2.2 - log-dirs

The log should be under the yarn.nodemanager.log-dirs configuration properties

Screenshot from Ambari

/mnt/resource/hadoop/yarn/log

Advertising

4.2.3 - Location

4.2.3.1 - Application

The application log will be at: ${yarn.nodemanager.log-dirs}/application_${appid}

4.2.3.2 - Container

The Containers log will be at ${yarn.nodemanager.log-dirs}/application_${appid}/container_{$contid}

4.2.4 - Framework

Application Framework Description
Map reduce job Each container directory will contain the files stderr, stdin, and syslog generated by that container.
Other frameworks They can choose to write more or less files not the location

4.3 - Archive

For clusters with a lot of Yarn aggregated logs, it can be helpful to combine them into hadoop archives in order to reduce the number of small files, and hence the stress on the NameNode.

  • Usage
mapred archive-logs

More … Hadoop Archive Logs Guide

db/hadoop/yarn/log.txt · Last modified: 2019/01/23 16:21 by gerardnico