About
job history server
For map reduce job, see history server
It lists the following jobs type:
- incomplete
- completed
- attempts.
Articles Related
Management
Start
./sbin/start-history-server.sh
URL
web interface: http://server:18080 by default
For azure hdinsight: https://clusterName.azurehdinsight.net/sparkhistory/
Configuration
(see spark.history.provider below),
file-system provider
- Server side configuration: When using the file-system provider class a base logging directory will contain sub-directories that each represents an application’s event logs.
- Client Side configuration: The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:
spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs