Effective Machine Learning Teams
The story of Dana, the data scientist
It’s 9.25pm and the soft glow of Dana’s computer screen glared into her bleary eyes as she logged on to continue fixing an error — the source of a splash of red across her screen. She had made and had her dinner, done her everyday chores, but her mind wasn’t really there — it was in a few places in fact.
It’s been an intense day — her day was peppered with switches between long training runs and back-and-forth messages with the support team on customer queries on why the model denied their loan applications. She was stuck in debugging why the model’s performance just wouldn’t improve, despite various tweaks to the data and model architecture. The occasional stack traces didn’t help either.
She was tired, and the tangled heap of uncommitted code changes sitting on her local machine added to the latent cognitive load that was bubbling over in her mind. But she had to keep going — her team had already missed the initial release date by four months, and the executives’ impatience was showing. What made things worse was a fear that her job might be on the line. 1 in 10 employees in her company — several of whom she knew — were laid off in the latest round of cost-cutting measures.
Everyone on her team was well-meaning and capable, but getting bogged down every day in a quagmire of tedious testing, anxiety-laden production deployments, stepping through illegible and brittle code — that wore them down within a few months. She and her team were doing their level best, but sometimes it feels like they’re building a house without a foundation — things keep falling apart. She doesn’t know it yet, but they weren’t set up for success.
—
Challenges and better paths in machine learning
Many individuals begin their machine learning (ML) journey with great momentum and gain confidence quickly, thanks to the growing ecosystem of tools, techniques, tutorials, and community of ML practitioners. However, when we graduate beyond the controlled environment of tutorial notebooks and Kaggle competitions into the space of real-world problems, systems, and people, many inevitably struggle to realize the potential of AI in the real world due to various unforeseen traps and unanticipated detours.
When we peel back the glamorous claims of being the sexiest job of the 21st century, we often see ML practitioners mired in burdensome manual work, complex and brittle codebases, and ultimately frustration resulting from Sisyphean ML experiments that never see the light of day in production.
In 2019, it was reported that 87% of data science projects never make it to production. According to Algorithmia’s Enterprise AI/ML Trends, even among companies who have successfully deployed ML models in production, 64% of survey respondents take more than a month to deploy a new model in 2021, an increase from 56% in 2020. They also found that 38% of organizations surveyed are spending more than 50% of their data scientists’ time on model deployment.
These barriers impede — or, in some cases, even prevent — ML practitioners from applying their expertise in ML to deliver on the value and promise of AI for customers and businesses. Even amidst the growing interest in data and AI, there are, more so than before, existential questions such as: is data science a dying profession?
Is there a better way?
The good news is — it doesn’t have to be this way. In the past few years, we’ve had the privilege to work on various data and ML projects and collaborate with ML practitioners from various industries. While there are barriers and pains as we have outlined above, there are also better paths, practices, and systems of work that allow ML practitioners to reliably deliver ML-enabled products into the hands of customers.
That’s what Effective ML Teams is all about. In this book, we draw from Lean software delivery to help ML teams create structures that shorten feedback loops, experiment rapidly and reliably, focus on the voice of the customer, and iterate towards building the right thing.
As Edwards Deming once said, “A bad system will beat a good person every time”. So we will share principles and practices that will help teams create structures that improves information flow, reduce waste (e.g. handoffs, dependencies), and increase the flow of value.
If we’ve done our job right, this book will invite you to look closely at how things have “always been done” in ML and in your teams, reflect on how well they are working for you, and consider other alternatives (based on our experience delivering AI solutions).
A sneak peek of the book
Chapter 1: Challenges and Better Paths in Delivering Machine Learning Solutions is a mini distillation of the entire book. We explore high-level and low-level reasons for why and how ML projects fail, and we lay out a more reliable path for delivering value in ML solutions by adopting Lean delivery practices across five key disciplines: product, delivery, machine learning, engineering, and data.
In the remaining chapters, we describe practices of effective ML teams and ML practitioners. In Part 1: Engineering Practices, we cover practices that help ML practitioners in their day-to-day work (e.g. automated testing, refactoring, using the code editor effectively). In Part 2: Product and Delivery Practices, we elaborate on practices in other subsystems which are necessary for delivering ML solutions, such as product thinking, delivery practices, team topologies, continuous delivery, and MLOps.
Part 1: Engineering Practices
Chapters 2 and 3: Effective Dependency Management describes principles and practices — along with a hands-on example that you can code along with — for creating consistent, reproducible, secure, production-like runtime environments for running your code. Instead of getting trapped in dependency hell, the practices in this chapter will enable you and your teammates to “check out and go” and create consistent environments effortlessly.
Chapters 4 and 5: Automated Testing — Move Fast Without Breaking Things provides you with a rubric for testing components of your ML solution — be they software tests, model tests, or data tests. We demonstrate how automated tests help us shorten our feedback cycles and reduce the tedious effort of manual testing, or worse, fixing production defects that slipped through the cracks of manual testing. We also describe the limits of the software testing paradigm on ML models, and how ML fitness functions and behavioral tests can help us scale the automated testing of ML models.
Chapter 6: Supercharging Your Code Editor with Simple Techniques will show you how to configure your code editor (PyCharm or VS Code) to help you code more effectively. After we’ve configured our IDE in a few steps, we’ll go through a series of keyboard shortcuts that can help you to automate refactoring, automatically detect and fix issues, navigate your codebase without getting lost in the weeds, among others.
Chapter 7: Refactoring — Getting Out Of Our Own Way. In this chapter, we draw from the wealth of software design to help us design readable, testable, maintainable, and evolvable code. In the spirit of “learning by doing”, you’ll see how we can take a problematic, messy and brittle codebase, and apply refactoring techniques to iteratively improve our codebase to a modular, tested, and readable state.
Part 2: Product and Delivery Practices
Chapter 8: MLOps and Continuous Delivery for Machine Learning (CD4ML) will articulate an expansive view of what MLOps and CI/CD (continuous integration and continuous delivery) really entails (spoiler alert: it’s more than automating model deployments and defining CI pipelines). We lay out a blueprint for the unique shape of CI/CD for ML projects and walk through how you can set up each component in this blueprint to create reliable ML solutions and free up your teammates from repetitive and undifferentiated labor so that they can focus on other higher-value problems.
Chapter 9: A Systems Thinking Approach to Improving ML Delivery covers other subsystems (namely product and delivery) which are essential for delivering ML solutions rapidly and reliably. We discuss product discovery techniques that help us ideate, validate and eventually converge so that we set ourselves up for success by starting with the most valuable problems. We’ll also go through delivery practices that help us measure delivery progress and learn how to identify and manage project risks. We’ll address the unique challenges resulting from the experimental and high-uncertainty nature of certain ML problems, and discuss techniques such as the dual-track delivery model that help us iterate more quickly in shorter cycles.
Chapter 10: Building Blocks of Effective ML Teams. In this chapter, we switch gears to focus on team topologies and the people aspect of effective teams. We describe principles and practices that help create a safe, human-centric, and growth-oriented team. We’ll also discuss organizational dynamics, cultural modes, team topologies, and how teams can work towards the culture that they want. We lay out concrete principles and practices that you can use to nurture a culture of collaboration, effectiveness, and learning.
Chapter 11: Responsible AI — Theory and Practice will describe mental models, techniques, and practices for putting the Responsible AI framework into practice. Beyond philosophical discussions (which are important), we also provide a framework for shifting ethics left and identifying any potential risk of harm or ethical issues in the ML solutions that we create. We’ll demonstrate how automated tests and automated quality checks from earlier chapters can help us test and prevent harm.
Some Parting Thoughts
There are three things we wanted to muse on before wrapping up the preface.
First, we want to acknowledge that machine learning is more than just supervised learning. We can also solve data-intensive (and even even data-poor) problems using other optimization techniques (e.g. reinforcement learning, operations research, simulation), data engineering, and data analysis approaches. In addition, machine learning is not a silver bullet and some problems can be solved without machine learning.
Second, on the role of culture: ML effectiveness and the practices in this book are not — and cannot be — a solo effort. That’s why we’ve titled the book as Effective Machine Learning Teams. You can’t be the only person writing tests, for instance. In organizations that we’ve worked with, individuals become most effective when there is a cultural alignment (within the team, department and even organization) on these Lean and agile practices. This doesn’t mean that you need to boil the ocean with the entire organization, and it’s also saying that it’s not enough to go it alone. As Steve Jobs once said, “Great things in business are never done by one person. They’re done by a team of people.”
Finally, this book is not about productivity (how to ship as many features, stories, or code as possible), nor is it about efficiency (how to ship features, stories, or code at the fastest possible rate). Rather, it’s about effectiveness — how to ship the right features and stories reliably and responsibly. This book is about finding balance through movement — and moving in effective ways. The principles and practices in this book have consistently helped us to successfully deliver ML solutions, and we are confident that they will do the same for you as well.
Curious?
The first three chapters of Effective Machine Learning Teams are available now on O’Reilly and a new chapter will be released each month. I will also be posting excerpts on my blog and YouTube channel. Stay tuned 🖖
