"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;

February 26, 2020

Queues vs Logs

Queues - Someone will publish a message in the broker, Consumers can read from the queue. Long back I worked on this on SQL Server Message Broker
Logs - All the information/transactions in SQL is implemented as WAL (Write ahead logging). At some point when a checkpoint is reached the transactions are written to disk. Commands are applied and data changes saved to disk.

How logs can be used?
  • Read the transaction and replay it elsewhere (Allows multiple consumers without blocking each other)
  • Keep logs read-only and let everyone read it (persist it as long as needed)
  • Read information in logs in sequence (maintain sequence to replay it in order)
So, Logs can be read across multiple readers and it enables scaling :)
Tools have evolved but the fundamentals are the same. Kafka is similar to a log playback system (distributed log processing) which helps to scale, publish and consume data.

Happy Learning!!!

No comments: