Data Processing - Buffer (Batch concept in code)

Data Processing - Batch in code.

Buffers are containers of a (fixed|variable with max) amount of data or operations.

Buffers live in the memory (ie created by code structure) whereas a blocks lives in the disk.

Without buffer, each read or write request is handled directly by the underlying layer (OS, Database, …) . This can make a program much less efficient, since each such request often triggers other operations that may be relatively expensive.. For instance, for an OS layer

• disk access,
• network activity,
• or some other operation

Furthermore, if requested data were sent as single big chunks, only one user could process it at a time, which obviously would make the application very inefficient and not much fun to use. (ie only one user would download data)

Example of buffered data:

• Byte arrays
• Database: Database - Fetch Size
• Network: Network - Data unit

• (Collection|Container) Data Type (Array, List, Set, ...) (queue, …)
• Java - IO - Buffer Streams

Flushing the buffer means that you write data to an entity (mostly a file).

• Data buffer