The proliferation of digital channels has made it mandatory for marketers to understand an individual across multiple touchpoints. In order to develop market effectiveness, marketers need have a pretty good sense of its consumer’s identity so that it can reach him via mobile device, desktop or a big TV screen on living room. Examples of such identity tokens include cookies, app IDs etc.A consumer can use multiple devices at the same time and so the same consumer should not be treated as different people in the advertising space. The idea of identity resolution comes with this mission and goal to have an omnichannel view of a consumer.

Identity Spine is MIQ’s proprietary identity graph, using identity signals across our ecosystem to create a unified source of reference to be consumed by product, business analysis and solutions teams for insights and activation. We have been able to build a strong data pipeline using Spark and Delta Lake, thereby strengthening our connected media products offerings for cross channel insights and activation.

This talk mostly highlights :
* The journey of building a scalable data pipeline that handles 10TB+ of data daily.
*How we were able to save our processing cost by 50%
* Optimization strategies implemented to onboard new dataset to enrich the graph