E149 ‘AI and Causal Inference to transform the user experience’ with Spotify’s Ciarán Gilligan-Lee

Today’s guest is Ciarán Gilligan-Lee, Head of Causal Inference Research Lab at Spotify. A finalist for AI Person of the year at the 2023 AI Awards, Ciarán joins host Mark Kelly on the podcast to discuss the significant role that AI and causal inference plays in enhancing user experiences at Spotify.

Ciarán explains that his team focuses on causal inference to understand cause and effect in user behaviors, rather than just correlations. This helps Spotify make more accurate recommendations for tracks, podcasts or audiobooks by determining what truly influences user enjoyment.

He highlights the importance of understanding whether users would naturally find content or need recommendations to improve their experience. The team combines causal inference techniques with deep learning to manage the vast amounts of data Spotify handles, enhancing both user satisfaction and artist exposure.

Ciarán also shares insights from his previous work in healthcare, where causal inference improved diagnostic accuracy over traditional correlational methods. He emphasises the broader applicability of causal inference across various industries for better decision-making.

The conversation touches on the challenges of avoiding biases and spurious correlations, ensuring that decisions are based on accurate causal relationships. Ciarán advocates for clear assumption documentation and rigorous testing, often involving A/B tests, to refine models and approaches.

The discussion concludes with reflections on the importance of careful data analysis and trusted sources in making informed decisions, particularly in complex fields like healthcare and technology.