Trendyol Indexing Team is heavily based on an event-based system that should be consistent, eventually. So even though this word seems to give relief, it is tough to ensure that every message in the system is successfully processed.
Their eventing system relies on Apache Kafka. It gives them powerful abilities when they need performance on event streaming. Their messages carry invalidations (daily 150M+) that should apply to Trendyol contents (300M+). This invalidation could be about a stock, promotion, or one of their 30 events.
In events worlds, we need to orchestrate messages even in some inconsistent states. Retrying and delaying messages are their key strategies in this “eventual” life cycle of events. They have 10+ consumers that should apply this strategy. Therefore, they created an open-sourced Golang library called
Kafka Cronsumer for easy implementation.