About
Accumulators can only be written by workers and read by the driver program.
They allow us to aggregate values from workers back to the driver.
Now only the driver can access the value of the accumulator for the tasks the accumulators are basically write-only.
And we can use this as an example to count errors that are seen in an RDD across workers.
- Aggregate values from workers back to driver
- Only driver can access value of accumulator
- For tasks, accumulators are write-only
- Use to count errors seen in RDD across workers
Variables that can only be “added” to by associative operations.
- Used to efficiently implement parallel counters and sums
- Only driver can read an accumulator’s value, not tasks
Types: integers, double, long, float
Articles Related
Accumulator in Action vs Transformation
In an action, each tasks update to the accumulator is guaranteed by spark to only be applied once. When you perform transformations, there's no guarantee because a transformation might have to be run multiple times if there are slow nodes or a node fails. So you should only use accumulators for debugging purposes when you have transformations.
Example
Sum
accum = sc.accumulator(0)
rdd = sc.parallelize([1, 2, 3, 4])
def f(x):
global accum
accum += x
rdd.foreach(f)
accum.value
Value: 10
Counting empty lines
file = sc.textFile(inputFile)
# Create Accumulator[Int] initialized to 0
blankLines = sc.accumulator(0)
def extractCallSigns(line):
global blankLines # Make the global variable accessible
if (line == ""):
blankLines += 1
return line.split(" ")
callSigns = file.flatMap(extractCallSigns)
print "Blank lines: %d" % blankLines.value