About

(Transaction|Request|commit) logs are structured log file store all changes made to the data as they occur.

They permits the implementation of :

same as Transactions - Write-Ahead Logging (Rollback journal) - WAL ?

See Design Pattern - (Command|Action|Transaction)

A transaction log is made up of entries (also records), which record all changes made to data.

This log are also known as:

The truth is the log. The database is a cache of a subset of the log.

If we keep events in a transaction log, it starts to behave like a version control system for data (versus code).

Most databases write every single insert, update and delete operation to a transaction log. This serves as a source of truth to the database and in case of errors, the database state is recovered from them.

Database Usage

If have all your logs file, you can reconstruct the entire database

Distributed

Application Purpose
http://bookkeeper.apache.org/ Machine Replication (High Availability)
Kafka (Event Hub) Publish-subscribe messaging rethought as a distributed commit log

Management

Transfer them

How to transfer log files into a log manager such as Kafka

  • Kafka Connect
  • Logstash
  • Flume
  • FileBeats from Elastic
  • Log Tailers
  • Develop an Application's logger that send straight to (No files involved at all).
  • Telegraf (for metrics oriented messages)

Read them

Implementation

Documentation / Reference