What are Log Collectors (Aggregators)?

Data System Architecture

About

A log collector will collect log file, optionally transform them and deliver them to one or more destination such as:

List

Name Type Log to Log Log To Metrics Description
Vector (Rust) observability pipeline tool Yes Yes
Promtail Log Collector Yes Yes 1) Graphana tool to insert log Loki
Logstash (Java) Log Collector Yes Yes 2) Elastic log collector
Fluentd Log Collector ? ? Example: BigQuery
Flume Log Collector ? ?

Log to Metrics means extracting metrics from application logs to be exported into a metrics server





Discover More
Card Puncher Data Processing
Computer Monitoring / Operational Intelligence / Real Time Monitoring

Computer Monitoring / Operational Intelligence / Real Time Monitoring Monitoring is the process of defining metrics and alerts in order to respond to a performance degradation where the acceptable level...
Scale Counter Graph
Counter - Collector

Metrics collector query and collects metrics in order to be able to send them to a metrics server Log Collector In a instrumented application, reporter are a client piece of code which: process...
Data System Architecture
Flume - Log (Aggregator|Collector)

Apache Flume is a log collector to collect, aggregate, and move log data from many different sources to a centralized data store. aggregation = sink Apache Flume example with HTTP channel to...
Data System Architecture
Log - Logging

A log is a text file with: a timed list of message (activities) that an application has performed execution/request log: web server: web log ( - that stores the Http request error cron...
Data System Architecture
Log - Server

A log server is an application that is aimed to: receive log via a collector analyse them report on them and send alert if needed They are mostly search engine where: the words are stored...
Data System Architecture
LogStash

LogStash is: * a metrics collector * a log collector. * with pipeline ability A Logstash pipeline is composed of the following elements; * input (produce the data) * filter (optional, process...
Scale Counter Graph
Metrics Server

This page is software that can store and visualize counter (metrics. The collectors that sends data to a metrics server are: Metrics Collector Log Collector Instrumentation Reporter The...
Card Puncher Data Processing
Monitoring Platform

Monitoring platform provides one or more monitoring services such as: Metrics management Log management Trace Management and other monitoring service such as visualization and ticketing. They...
Scale Counter Graph
What are Application Metrics? ie Perfcounter, Performance Metrics, Operational data, Monitoring, telemetry

This section is about the collection and calculation of metrics in a monitoring context known as observability.
Scale Counter Graph
What are observability data pipeline tools? known as Telemetry agent

An observability data pipeline is a pipeline tool dedicated to observability data (logs, metrics, and traces). It collects them (log collector, metrics collector, ...) from multiple sources transform,...



Share this page:
Follow us:
Task Runner