What is Big Data?

---

The advent of digital technology and its adoption by every sphere of life results into generation of tremendous amount of digital data. This data could be generated as part of the system's business requirements and/or as a system by-product such as logs. In any case, captured data is deemed to have significance in one or the other business use cases. At the same time, economical costing of computing devices such as PC, mobiles, etc. and the reach of internet to every nook and corner of the world has allowed mass utilization of these systems. Thus this high rate of adoption and mass usage of these systems produces a huge amount of data which could be measured in Terabyte (TB), Petabyte (PB) and even Exabyte (EB) for each one of these systems.

Figure below show how the data storage has grown significantly, shifting markedly from analog to digital after 2000 [López11]. These numbers are from 2007 and for sure by now the analog space has squeezed more.

Consider social applications such as Facebook and Twitter. These applications are used by millions of users in order to engage with each other either socially or professionally.
For instance, according to Wikipedia [WikiFB14] Facebook which has more almost one and half billion users share around 5 billion content items per day which includes status updates, wallpapers, comments, photos and videos. This leads to more than 500TB of data per day for Facebook to handle. Twitter on the other hand with more than 250M users, share 400M tweets per day [WikiTW13]. Handling the real time tweets and mining for trending topics (hashtags) at such an exceptional rate does require some kind of innovation in technology.

Tremendous amount of data is also getting generated by the sensors attached to almost every measurable thing in the world , be it temperature, wind speed, ocean movements, heart beats or variation in your body mass index. Penetration of the smart phones in our daily life has made it possible to record everything recordable such as videos, daily exercise data, location logs, reading habits, buying trends, etc. Internet of things (IoT) using M2M technology is adding to this already crowded space of data producers.

Traditional businesses have also adopted technologies to increase their productivity. These companies churn out burgeoning volume of transactional data, capturing trillions of bytes of information about their customers, suppliers, and operations. 

Finally, the last but not the least, the continuous digitalization of information which otherwise was available so far in the form of physical documents or books and e-governance initiatives by various countries are too building a database of petabyte in size. US library of Congress collected 235 terabytes of data by 2011 and 15 out of 17 sectors in the United States have more data stored per company than the US Library of Congress [WikibonBDS12].

It is found that the 90% of all the data ever created was created in last 2 years. Estimates show that world's data is going to double every two years [EMC11].

So, how are the technologies going to handle the tremendous explosion of data in the next few years? Traditional technologies that are used to manage the data had never envisioned of managing and processing such an enormous volume of data. This has resulted in a paradigm shift in data management and processing systems. Relational models of data modelling have given way to flat data models. Vertical scaling of infrastructure has fall short of capacity to process this volume of data so horizontal scaling has become the new architectural concern. In software design, data parallelism is being preferred over task parallelism. All this has led to a new paradigm called Big Data paradigm which is efficient processing of extensive datasets by distributing data systems horizontally.