What is the difference between at-most-once and at-least-once delivery in Kafka?

In Kafka, delivery semantics define the guarantees regarding message delivery from producers to brokers and from brokers to consumers. The primary differences lie in how they handle message loss and duplication during network failures or restarts, impacting data integrity and system performance.

At-Most-Once Delivery

At-most-once delivery guarantees that a message will be delivered zero or one time. This means that while a message might not be delivered at all (e.g., if a producer sends a message and fails before receiving an acknowledgment), it will never be delivered more than once. Producers configured for at-most-once delivery do not retry sending a message if an acknowledgment is not received or if an error occurs during transmission.

This semantic prioritizes low latency and high throughput over absolute data completeness. It is suitable for use cases where occasional data loss is acceptable, or where messages are ephemeral and re-sending them would be redundant or harmful. For instance, monitoring metrics or log data where missing a few entries won't compromise the overall analysis.

At-Least-Once Delivery

At-least-once delivery guarantees that a message will be delivered one or more times. This means that a message is guaranteed to be delivered, but there is a possibility that duplicates might occur. Duplicates typically arise if a producer sends a message, but doesn't receive an acknowledgment (e.g., due to a network timeout) and retries the send, even though the original send might have succeeded.

Kafka producers achieve at-least-once semantics by configuring 'acks=all' (or 'acks=1') and enabling retries. With 'acks=all', the producer waits for acknowledgments from all in-sync replicas before considering a message successfully written. If an ack isn't received within a timeout, the producer retries. This ensures no data is lost but introduces the potential for duplicates if a retry happens after an initial successful write whose ack was lost.

This semantic is crucial for applications where data loss is unacceptable, such as financial transactions, order processing, or critical event logging. To handle the potential for duplicates, consumers often need to implement idempotent processing logic.

Key Differences and Implications

The fundamental difference lies in the trade-off between data loss and message duplication. At-most-once favors speed and avoids duplicates at the cost of potential data loss. At-least-once guarantees no data loss at the cost of potential duplicates, requiring consumers to handle idempotency. Kafka also offers 'exactly-once' semantics (since 0.11.0) which builds upon at-least-once by adding transaction IDs to ensure no data loss and no duplicates, but it comes with higher overhead.