Yarn - Queue

> Database > (Apache) Hadoop > Yarn (Yet Another Resource Negotiator) - Hadoop Operating System

1 - About

A queue is a scheduler data structure that allows scheduler implementation to categorize apps into queue.

Advertising

3 - Properties

Queue definitions and properties such as capacity, ACLs can be changed, at runtime.

4 - Capacity

4.1 - Full

The utilization of the queue may become full (100% utilization) because the rate at which new jobs get submitted is higher than the rate at which the old jobs are completed.

The following image shows a joblauncher queue at 714.4% overused.

This is acceptable so long as there is still free capacity in the default queue to borrow from. However, when the cluster is fully utilized and the YARN memory is at 100% capacity, new jobs must wait, which causes timeouts on the client side (submit)

5 - Management

5.1 - Add

Administrators can add additional queues at runtime.

5.2 - Delete

Queues cannot be deleted at runtime.

5.3 - Stop

Administrators can stop queues at runtime. Existing applications continue to completion

If a queue is in STOPPED state, new applications cannot be submitted.

Advertising

5.4 - Start

A stopped queue can be started.

5.5 - Info

Yarn - Yarn CLI

yarn queue

5.6 - List

5.7 - Status

Yarn - Yarn CLI

yarn queue -status <QueueName>

State or status can be:

  • STOPPED - Existing applications continue to completion.
Advertising

5.8 - Move app to queue

See move

db/hadoop/yarn/queue.txt · Last modified: 2019/03/25 13:36 by gerardnico