CapitalOne is first US bank to exist out of on-premises and moved completely on Cloud. Over this process of modernizing our application in CapitalOne Card Rewards, we developed ground up custom transactions processing application on open source technologies like Apache Spark, MongoDB, Apache Cassandra etc. This application currently processes millions of customer transactions daily providing them millions of miles, cash and points everyday. In process of building our application, we came across many challenging issues to have Spark application process data from MongoDB and Cassandra backend to serve customers. This talk focuses on tale of these 3 open source technologies on how they co-existed and solved the problems of Open source land applications mainly focussing on below problems:
How Cassandra Key sequence is important and how it impacts in querying
How Cassandra batching helps and works well with Spark partitions
Importance of Cassandra Data Modeling and its implications after MVP/Deployment
How to manage Mongo Connection (at JVM level)
Implications of using MongoSpark connector on its Partitioner