Agile Data N’ Info

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern for the AgileData Activity Event Tile.   The Activity Event Tile pattern tracks a sequence of events (e.g., subscription signups, subscription payments) for a core business concept by storing minimal event data and using a hydration layer to automatically generate pre-calculated metrics as views. This allows business users and analysts to easily understand business performance and event sequencing without complex SQL, providing immediate, consistent insights into activity movements and financial impacts   An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.   If you want a copy of the pattern template head over to: https://agiledata.substack.com/i/170387919/pattern-name Discover more  AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern for the AgileData Event Tile.   The Event Tile Data Modeling pattern captures business processes and transactions within AgileData's three-layered data architecture, modeling "who does what" at a specific point in time. It achieves this by storing only a small number of core business keys in insert-only event tables. These event records are then automatically hydrated into a "consume tile" by joining with all relevant detail attributes that were current at the exact moment the event occurred, ensuring an accurate, time-staged narrative. This provides data consumers with a rich, wide, and historically accurate view of events, simplifying the querying of complex historical changes and state transitions.   An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.   If you want a copy of the pattern template head over to: https://agiledata.substack.com/i/169697974/pattern-name Discover more  AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Automated Load Patterns based on Source Data Profiles.   The Automated Load Patterns based on Source Data Profiles pattern automatically profiles incoming data to determine its optimal loading method. It classifies data as either event data (immutable, append-only) or change data (evolving records) by analysing characteristics like unique key volume and column names over time. This automated classification then dictates the load strategy: an efficient partition replace for event data or a cost-effective upsert (end-dating historical records) for change data. This capability removes the manual burden and accelerates data onboarding, providing a trustworthy and efficient process even when the data type is initially unknown.   An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.   If you want a copy of the pattern template head over to: https://agiledata.substack.com/i/168912285/pattern-name Discover more  AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns     If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Trust Rules to validate data.   The Trust Rules pattern provides automated data validation to ensure all incoming data is fit for purpose and trustworthy. It bakes in essential checks such as unique business keys and business effective dates, which run automatically upon data load or table refresh. Users can also define custom validation rules for specific columns. Results are collected, persisted, and surfaced via applications or alerts, with the system optimising validation for cost and speed through smart partitioning and clustered columns.   An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.   If you want a copy of the pattern template head over to: https://agiledata.substack.com/i/167964917/pattern-name Discover more  AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns     If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Unioning two or more tables together automatically.   The Automated Table Unioning pattern automatically combines two or more tables by intelligently looking up column names and data types to generate safe SQL under the covers. It supports disparate data sources, such as those from multiple publishers in a data clean room, and creates a view or incrementally loads the unified data into a physical table while tracking load watermarks to prevent duplicates.   An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.     If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Orchestrating Dynamic Data Flows. The Dynamic Data Flow Orchestration pattern dynamically generates and self-heals data flow manifests (DAGs) at runtime from a central context database, enabling adaptive, cost-effective, and low-latency data processing primarily for micro-batching workloads An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.   If you want a copy of the pattern template head over to: https://agiledata.substack.com/i/167144707/pattern-name Discover more  AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns     If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson as he chats with Roelant Vos about a number of Data Engine Thinking patterns and his new book Data Engine Thinking. You can get in touch with Roelant via LinkedIn or over at https://roelantvos.com/blog/ Or you can grab a copy of the Data Engine Thinking book on Amazon or at https://dataenginethinking.com/en/ If you want want to read a summary generated with GenAI, head over to: https://agiledata.substack.com/i/166295398/google-notebooklm-briefing   If you want to download the transcript for the podcast, head over to: https://agiledata.substack.com/i/166295398/transcript     Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson as he chats with Chris Gambill about a number of Data Engineering patterns. You can get in touch with Chris via LinkedIn or view his YouTube channel https://www.youtube.com/@GambillDataEngineering If you want want to read a summary generated with GenAI, head over to: https://agiledata.substack.com/i/165842333/google-notebooklm-briefing   If you want to download the transcript for the podcast, head over to: https://agiledata.substack.com/i/165842333/transcript   Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Ramona C Truta as she interviews Shane Gibson about the patterns of writing a data book (and they discuss a whole lot of other data things) You can get in touch with Ramona via LinkedIn or read more on substack https://ramonatruta.substack.com/ Or buy a copy of Shanes book "an Agile Data Guide to Information Product Canvas", and finally fix those data requirement gathering problems. If you want to download the transcript for the podcast, head over to: https://agiledata.io/podcast/agiledata-podcast/the-pattern-of-writing-a-data-book-with-shane-gibson-and-ramona-c-truta/#read   If you want want to read a summary generated with GenAI, head over to: https://agiledata.substack.com/p/the-pattern-of-writing-a-data-book Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson as he chats with Andrew Foad on his data modeling pattern "Hook" You can get in touch with Andrew via LinkedIn or read more on substack https://hookcookbook.substack.com If you want to download the transcript for the podcast, head over to: https://agiledata.io/podcast/agiledata-podcast/the-hook-data-modeling-pattern-with-andrew-foad/#read   If you want want to read a summary generated with GenAI, head over to: https://agiledata.substack.com/p/the-hook-data-modeling-pattern-with   Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson as he chats with Aaron Wilkerson on ways data teams can increase the perception of the value they add to their organisation. You can get in touch with Aaron via LinkedIn or read his newsletter on substack https://aaronwilkerson.substack.com If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/increasing-the-perceived-value-of-your-data-team-with-aaron-wilkerson/#read   Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson as he chats with Remco Broekmans about the pattern templates that are part of the Ensemble Logical Modeling (ELM) patterns. You can find more about ELM and its templates over at: https://www.elmstandards.com/ Find Remco's book on Amazon: https://www.amazon.com/Stories-Solutions-Redefining-Communication-Professionals-ebook/dp/B0DXQ8RR3X/ You can get in touch with Remco via LinkedIn or get in touch via https://www.geneseeacademy.com/ If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/elm-patterns-templates-with-remco-broekmans/#read   Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson as he chats with Dylan Anderson about the patterns required to define a Data Operating Model. This conversation was based on this article written by Dylan: https://thedataecosystem.substack.com/p/issue-13-defining-the-data-operating And specifically this pattern diagram: You can get in touch with Dylan via LinkedIn or subscribe to his newsletter at https://thedataecosystem.substack.com or get in touch via https://profusion.com If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/data-operating-models-patterns-with-dylan-anderson/#read   Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now!   Simply Magical Data Ways of Working

Join Shane Gibson as he chats with Chris Bergh on improving your teams way of working by using DataOps patterns. You can get in touch with Chris via LinkedIn or over at https://datakitchen.io If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/dataops-patterns-with-chris-bergh/#read   Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the AgileData Way of Working over at https://wow.agiledata.io/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |    Simply Magical Data

Join Shane Gibson as he chats with Patrick Cuba on combining the Data Vault data modeling pattern with the Medallion Architecture pattern. This discussion was based on the patterns described in this article: https://medium.com/the-modern-scientist/the-modern-data-vault-stack-75103102e3d2 You can get in touch with Patrick via LinkedIn or you can read his thoughts on Data Vault patterns over at https://patrickcuba.medium.com If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/merging-data-vault-and-medallion-architecture-patterns-with-patrick-cuba/#read   Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the AgileData Way of Working over at https://wow.agiledata.io/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |    Simply Magical Data

Join Shane Gibson as he chats with Eric Broda on the patterns required to create an ecosystem to support the use of Agents in enterprise organisations. You can get in touch with Eric via LinkedIn or you can read his thoughts on Agentic Mesh over at https://medium.com/@ericbroda If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/agentic-mesh-ecosystem-patterns-with-eric-broda/#read Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the AgileData Way of Working over at https://wow.agiledata.io/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |    Simply Magical Data

Join Shane Gibson as he chats with Petr Pascenko on the pattern of Reliability Engineering of AI Agents You can get in touch with Petr via LinkedIn or you can read his thoughts on Agentic Mesh over at https://evalmy.ai/ If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/reliability-engineering-of-ai-agents-with-petr-pascenko/#read Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the AgileData Way of Working over at https://wow.agiledata.io/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |    Simply Magical Data

Join two LLM generated guests as they discuss the Top Data Trends of 2025 Whitepaper published by Coalesce. This is a different episode.  Instead of a human guest, we have two robot guests.  I decided to try and experiment. My experiment was, can I upload a white paper to LLM, have it generate a podcast listen to that podcast in my daily walk and see whether that summary removes the need for me to actually read the white paper. So in this case, I have grabbed a white paper  called Top Data Trends for 2025 from Coalesce, uploaded it to the Google Notebook LLM and got it to generate a podcast with two hosts chatting about the white paper. Have a listen, let me know what you think. I'm really keen to understand, do you think this approach is useful, or is it a load of bollocks? You can download the whitepaper from the Coalesce website here: https://coalesce.io/reports/the-top-data-trends-for-2025/ If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/an-experiment-top-data-trends-for-2025-with-coalesce-and-google-notebooklm/   Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the AgileData Way of Working over at https://wow.agiledata.io/way-of-working/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility or Nigel @nigelvining on LinkedIn.   Subscribe: Apple Podcast | Spotify | YouTube | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |    Simply Magical Data

Join Shane Gibson as he chats with Andrew Jones on the pattern of Data Contracts You can get in touch with Andrew via LinkedIn or at https://andrew-jones.com/ If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/data-contracts-with-andrew-jones/#read Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the AgileData Way of Working over at https://wow.agiledata.io/way-of-working/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility or Nigel @nigelvining on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |    Simply Magical Data

Join Shane Gibson as he chats with Joe Reis on how the potential adoption of GenAI and LLM's in the way Data teams work. You can get in touch with Joe via LinkedIn or at https://joereis.substack.com/ If you want to read the transcript for the podcast head over to: https://agiledata.io/podcast/agiledata-podcast/ai-data-agents-with-joe-reis/#read Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the AgileData Way of Working over at https://wow.agiledata.io/way-of-working/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility or Nigel @nigelvining on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |    Simply Magical Data