Yarn - Log (Container, Application) - Tfile

1 - About

The log of an application (ie a container when running).

3 - Format

Application logs are not saved in text format. They are saved in a binary format called TFile.

A TFile format is a container of key value pairs, and it is not readable by humans.

4 - Management

4.1 - Configuration

yarn.log-aggregation-enable to true

4.2 - View

4.2.1 - Command Line

doc

yarn logs -applicationId <applicationId>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application>
yarn logs -applicationId <applicationId> -appOwner <user-who-started-the-application> -containerId <containerId> -nodeAddress <worker-node-address>

where:

4.2.2 - Ui

With HortonWorks Ambari > Yarn > Quick Links:

4.2.3 - Other tfile reader

tfile reader:

4.3 - Location

The log should be under the yarn.nodemanager.log-dirs configuration properties

Definition: Where to store container logs:

  • An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}
  • Individual containers' log directories will be below this, in directories named container_{$contid}.

Each container directory will contain the files stderr, stdin, and syslog generated by that container.

Screenshot from Ambari

/mnt/resource/hadoop/yarn/log

4.4 - Archive

For clusters with a lot of Yarn aggregated logs, it can be helpful to combine them into hadoop archives in order to reduce the number of small files, and hence the stress on the NameNode.

  • Usage
mapred archive-logs

More … Hadoop Archive Logs Guide

db/hadoop/yarn/log.txt · Last modified: 2018/06/12 10:47 by gerardnico