Description:

Being on a technology team is like being in a band. It takes meticulous work for an ensemble to understand a piece of music to the level of having an intuition about how it works. The same goes for running the software systems we build, and the discoveries of complex system behavior uncovered by the practice of Chaos Engineering clarify understanding. This talk juxtaposes the disciplines of musical performance and operating resilient computer architectures, illustrating the capacity of the human mind to capitalize on intuition enhanced by shared experience and collaboration.

Abstract:

Musicians learning how to improvise music develop an intuition built around internalizing the materials and form of the genre – like scales, chord changes, or rhythmic structures. It can be directly compared to the “mental map” that engineers develop when writing software and understanding complexities. Each member of an ensemble or team have their subjective view on relevant (but overlapping) parts of the system and are challenged when relating each other’s substrate to theirs.

This is different from the more lizard-brainy concept of instinct, which can be confused with intuition as the “gut feeling” humans get from something. For example, great technology leaders develop good intuition when hiring engineers but should never rely on instinct. The best DBREs have an intuitive understanding of their platform in the context of the whole, but there’s nothing instinctual about it. Put simply, intuition is about knowledge, instinct is about not getting eaten.

The more we understand a system, the better we know how to make it resilient to unexpected events. Even so, a system becomes more complex as it grows dimensions, shrinking the capacity of any one person to comprehend the whole thing. Therefore we rely on shared and discovered knowledge. For instance, a phenomenon known as “fundamental common-ground breakdown” (FCGB) happens when accumulated individual learnings – i.e. each person’s intuition about the system – are assumed knowledge among participants, good or bad. Part of the game is learning how to harmonize these separate threads of experience, whether it be during an incident or designing a solution.

This talk seeks to unravel the terminology and methodology around how we humans operate complex systems by taking a closer look at intuition and pattern recognition, especially as it relates to other human activities like performing a musical work. It will introduce the concept of FCGB and how it contributes to our efforts to collaborate and respond to incidents, with real-world examples. A Chaos Game Day example will tear down the “gut feeling” myth to show that intuition is not an act of instinct, but a developed ability based on careful analysis and practice. By relating my own direct experiences in both performing music and running distributed systems, I will show how being inspired by software is a thing, just like playing in a band.

Takeaways: Chaos Engineering Game Day practice methods, how to avoid FCGB and improve communication, and an understanding that Chaos Engineering is not just about ‘breaking production’ but about discovering unknowns in the system through scientific experimentation.

Suggested topics: Humans in Computing, Chaos Engineering, Resiliency in Distributed Systems, Creativity in Technology, Team Building