Execution vs. exploration and what it means for you
Data scientists serve a very technical purpose, but one that is vastly different from other individual contributors. Unlike engineers, designers, and project managers, data scientists are exploration-first, rather than execution-first.
This isn’t a surprise when you consider the origin of data science. If you quickly look over the early history of the field, you see that things got started with academics researching the possibilities of computational statistics. This researcher-like mindset is still embedded in our DNA.
We are constantly surrounded by data that represents the business, product, and customers at scale. This allows us to see things from a 30,000-foot view, where other roles spend most of their time at ground-level, working on execution. It’s important that we realize this fact, and more important that we make the most of it.
Execution vs. exploration
Most technical ICs within established companies focus on execution. This is pretty intuitive. In order for a company to be successful, it has to get things done that provide value.
Data science roles are a little different. They vary greatly depending on the team structure and size, but generally speaking, execution isn’t where we’re at our best. Our most valuable work often comes from exploration.
When it comes to complex questions and hypotheses, execution isn’t the answer. Someone has to dive in and figure things out on a deeper level. They have to thoroughly analyze and explore the problem. Data scientists are the perfect candidates to take this on.
The act of thinking, coming up with a hunch, and then exploring that hunch is criminally underrated. When done right, not only does this work produce interesting results — it drives decision-making. This is where data scientists really thrive.
If you look at where certain roles end up on the execution-exploration spectrum, you get something like this:
This isn’t to say that data scientists can’t or shouldn’t execute. We spend a good amount of time building models, writing production code, and automating common tasks. The reality is that we have a diverse skill set that allows us to both explore and execute. This is why data scientists are hard to find, and also what makes the field so exciting and challenging.
Finding a balance
Should data scientists go completely rogue and do whatever they want? Probably not. We can’t blatantly ignore a backlog of JIRA tickets while looking into a hypothesis that came to mind at 2:00 AM the previous night. There has to be a balance here.
We have to be there for our stakeholders. This means delivering what they need in a timely manner so they can effectively make decisions and drive things forward.
However, we’re equally obligated to take advantage of our unique position and analytical skill set. We do this by taking time to ponder new ideas, generate hypotheses, and wander through the data a little bit.
But the question remains: what does this look like in practice? It’s not easy to think this way in a world of constant focus on execution. Recently, I’ve been doing three different things to stay exploration-first. I’m pretty happy with the results so far.
Block off time
First, I recommend blocking off an hour or so daily for deep thought and exploration. The best time for you will vary from person to person. I prefer first thing in the morning, but you could just as easily set aside an hour in the afternoon. It’s extremely important to schedule this time.
Create a system for success by making a recurring meeting with yourself every day. This is a meeting that you can’t afford to miss or reschedule. Hold yourself accountable. This is your time to think.
Write everything down
In case you haven’t heard, documentation is kind of important. Your thinking practice is no exception to this rule. No matter the quality of your idea, get it down somewhere. Create a running document or keep a notepad where you can allow these ideas, questions, and hypotheses to live on and be revisited.
Stay curious
As a data scientist, curiosity is your north star. Sometimes you’ll get caught up in execution-mode and forget to develop and explore your own ideas. When this inevitably happens, curiosity is what will bring you back. I highly recommend this excellent article from Multithreaded for more on the topic of curiosity in data science.
“Empower your data scientists to come up with ideas you’ve never dreamed of before.“ — Eric Colson
Shift your mindset
The execution-based work gets most of the love in data science. And can you blame us? It’s easier to quantify. You can see the results that come from building a model or pushing code to production.
It’s more difficult to see concrete results from an afternoon tinkering with a new idea. This new idea probably won’t lead to anything significant. Maybe only 10% of these bets actually turn out to be anything. Don’t let this discourage you. The 10% is worth it. The 10% is where the truly transformative work comes from — and it all starts with thinking.
