The Age Of Overfitting

The ability to create models with data points and observations to fit any theory as certainty.

As I was taking the turn to yet another visiting lecture at a university an idea dawned upon me. What if i were given three wishes? This idea got me thinking the entirety of my drive to the university.

Now I don’t mean to pretend I wake up any different to a contemporary lifestyle. Like any other human, I check my Facebook status and I do get lost in other people’s opinions, jokes, hate and so on. Looking at the daily deluge on my Facebook made me realize what I would wish first: Let us all learn to value the gift of humanity and tolerance. Let’s not put above the value of life and happiness above any beliefs.

Passing the newly done up Arcade complex made me feel happy, remembering what it looked like not so long ago. When most of my class mates and batch mates chose to migrate I took the decision to stay back in SL as an act of great bravery or stupidity. Now that the war looks like a distant memory we all can look at better times ahead. Hence my second wish: let Sri Lanka have a sense of urgency in all its dealings.

Finally it came down to what I would wish for myself. Like any other, I’d love to have wealth, health and so much more. However there is one thing, which is never on my side: Time.

“Let me be 10 years….no make that 15 years – younger than I am now.”

What I’ve realized is we live in one of the best times of human civilization. The Internet has single handedly democratized the landscape and possibilities for everyone. I recall sitting at my computer lab in university and watching Hotmail load one line at a time just 14 years ago. Now I cannot get enough of 4G speeds.

But internet, for all its power, is still in its teenage years. As all parents know, teenagers can be bit rebellious and problematic at times. The next few years will certainly define what type of internet we are likely to be with for some time. Debates rage on for net neutrality, privacy, national security and so on – I honestly do not have any idea where the final compromise will be.

On the other hand there had been a mushroom of internet /social media businesses emerging, even in Sri Lanka. Some do have an amazing potential to be the next blue chip companies in SL. On a global scale, the emerging of the most “admired” companies as a result further proves my hypothesis of internets ability to level the playing field for industries and brands alike and make value creation the sole attribute for success. The proof of this is in the diagram below that shows the average age of organizations joining the S&P index in US now.

“Our children will work for companies that do not exist in the real world and probably the founders are not even born yet.”

Just read that line again. Blows my mind each time.

However the internet itself has few characteristics in the world. Often all of us are blinded by the blockbuster and surgically targeted marketing hype which comes from U.S. about almost, well everything. A deep study of the internet will indicate the way we consume and digest the internet is fundamentally different as you move to emerging markets -meaning countries like ours. We have skipped many tech cycles and we are that part of the world where mobile rules shoulders and  heads above all other internet touch points. This is on top of the significantly different content consumption styles of emerging markets versus the developed world.

Let’s look at mobile for a bit now. The fact that we all are slaved by the device we carry around in our pockets is a phenomenon no one really saw coming. We trust this device with all our secrets and often look at it more times a day than our most loved human being. Now this is just a factual point – and one thing I will try not to do is to give anyone reading this a conclusions whether this is a good or a bad thing.That will be determined in time to come.

Now the meteoritic rise of mobile devices even in Sri Lanka creates some interesting opportunity for those who can build models around this. The mobile devices we carry around creates a trail of who, what, when and where about us. This is the type of information any marketeer would love to get hold of any way – essentially, the holy grail of targeted marketing.

But let’s come back to the marketing opportunity on a later date.

Let’s stay on with the data volumes which get created because of this constant connection we (often unknowingly) maintain. These connections create megabytes of information per second about each of us. This gets supercharged with the age old concept of IOT – Internet Of Things. (I did say age old since it was a scientist named Mark D. Weiser from Xerox PARC who put forward the novel concept of the future as back as 1988. The term given by Mr. Weiser was “ubiquitous computing”; in my mind the concept has uncomfortably close comparison to the present day IOT).

(This is also why I believe in the long run by the likes of Samsung, which target massive numbers, rather than by Apple. No offense to Apple fans.)

The creation or emergence of overwhelming volumes of data has created opportunities for many. Suddenly, even in Sri Lanka, we ‘re hearing the need to have highly capable statisticians to build business models which should evolve to be adopted for behavior prediction purposes are well underway. If any of you are now thinking “Big Data”, we are certainly flirting with that concept as we speak.

But many have overlooked the downside of this data overload. At the bottom of it what drives this is our innate humanity. We seek evidence and data that supports a decision we have already concluded. Due to Internet, mobile, IOT etc. there is now enough and more data and facts to prove any case we have already made up our minds or an argument we are already convinced.

The insistent creation of data points as a result of consistent connections and engagements driven primarily by mobile devices is creating an opportunity for many to create models for almost any purpose. We will move to do more data driven model creations for business purposes but will overlook the certain cost of errors.

The ability to create models as a result of data does not in anyway diminish the consequences or the cost of any errors as a result.

Hence to overfit is human.

Curabitur vulpatate vivera.