Redesigning Kaizen's Cloud Data Lake for the Future
At Kaizen Gaming, data is the backbone of our decision-making process and a driving force behind our innovative services. However, as our organization has grown, the complexity and volume of data we manage have scaled exponentially. This growth revealed inefficiencies in our legacy cloud architecture, leading to escalating costs, delayed insights, and limited scalability. Our initial setup, which was designed for a much smaller organization, could no longer meet the demands of our current scale operating in 18 countries and handling over 350M daily transactions (over 1PB in storage). Shared quotas, lack of cost transparency, and operational bottlenecks had become significant barriers, preventing us from fully leveraging our data’s potential.
To tackle these challenges, we embarked on a journey to redesign our cloud architecture, focusing on improving scalability, increasing cost transparency, enhancing data accessibility, and strengthening data governance. Central to our new approach is the concept of Data Landing Zones – a scalable and modular cloud framework designed to support the needs of a growing organization. By decoupling cloud resources across teams, departments, and environments, we eliminated shared quotas, enabling divisions to scale independently and more efficiently while being accountable for their costs.
Automation has been a cornerstone of this transformation. By implementing repeatable infrastructure, we significantly reduced administrative overhead and enhanced visibility into cloud costs through a FinOps framework, enabling granular insights at the team level rather than just the company level. This automation streamlined key processes like environment provisioning and reduced lead times paving the way for self-service infrastructure, allowing us to deliver insights faster and more securely with cross governance. Additionally, centralized governance using Unity Catalog ensures that data remains accessible while adhering to compliance and security standards.
The migration to this new architecture was not without its challenges. We needed to maintain the stability of our data products, manage data transfer costs, and minimize network latencies. To address these, we adopted a phased migration strategy to ensure continuity and minimize disruptions. Unity catalog’s Delta sharing protocol helped us hide migration complexity from our end users, while Databricks Asset Bundles streamlined our multi-environment setup, cutting down on operational complexity.
The results have been transformational. By democratizing access to our data platform, departments across Kaizen Gaming are now empowered to make data-driven decisions faster and more effectively. Enhanced cost attribution and observability have provided teams with greater control over their budgets, enabling them to optimize variable cloud expenses. Furthermore, the new architecture has significantly shortened onboarding periods for new teams, fostering a culture of innovation and operational efficiency.
Through this session, we aim to share our journey of migrating to a more scalable, efficient, and transparent cloud architecture. Attendees will gain a practical understanding of leveraging modern cloud solutions to address organizational challenges and walk away with actionable insights to apply within their own environments.
Key Takeaways:
Designing cloud architectures for organizational scale
Comparison of old vs new architectures
Enhancing data governance and data security
Leveraging variable cloud costs to optimize operational expenses
Migrating with minimal service disruptions
Reducing migration effort and impact on teams involved