Intuition in data science is a funny thing.
On one hand, the whole point of data science is to avoid relying on the highest paid person’s opinion (HIPPO).
On the other hand, data scientists are prized for product sense, skepticism, and other attributes that sound a lot like intuition.
So what gives? Should data scientists rely on intuition? Or should they channel their inner Spock and strive to insist that logic trumps emotion?
Or perhaps we should look at Star Trek more closely. Spock is half-Vulcan and half-human, making his character a hybrid of Vulcan logic and human intuition. That’s a great model for data scientists to follow.
The two more important activities in data science are hypothesis generation and hypothesis testing.
Hypothesis testing keeps data scientists — and everyone else — honest, and there we should rely on Vulcan logic to protect us from our human weaknesses of cognitive bias.
Hypothesis generation, however, is more of an art than a science. Hypotheses often represent intuitive leaps, and hypothesis generation is where our squishy humanity shines.
So, if you are a data scientist, go forth and be like Spock. That is, don’t try to suppress your intuition, but instead channel it into hypothesis generation where it can propel you towards critical insights. But let your Vulcan side do the hypothesis testing.
And may your data science live long and prosper!