Can you guess how the data generated across the globe were handled about 10-12 years ago? Relational databases were mainly used to manage data until the advent of Hadoop technology. Do you know that 95% of such data is unstructured as of now and the percentage is growing enormously? The data in the world is projected to grow 50 times in 2020 compared to the data in 2011. Now, you may ask what unstructured data is. It is nothing but video, images among many others which are produced in incredible proportions. A lot of such data is generated thanks to free and scalable social media networks. But how do top tech MNCs process this huge amount of data to derive business decisions from them?
Make way for Big data!
The technologies that are booming greatly in the recent times are Artificial intelligence, Big data, Cloud Computing among many others. Companies across the world from startups to mature tech players have already shifted to Big data analytics. The storage which is scalable to a very large proportion at an inexpensive cost is one reason why big data analytics is so popular today. Another factor for this trend is competent processing power that comes with big data that is also fault tolerant. Seeing such a growing demand in big data technologies, e-learning company like Intellipaat is providing courses on big data analytics. But how do top technology companies leverage this big data to serve their customers? Let’s see.
Did you know that Google processes about 3.5 billion search queries on single day? Do you know that each request queries about pages numbering 20 billion? Google derives such search results from knowledge graph database, indexed pages and Google bots crawling over a plethora of web pages. The user requests are processed in Google’s application servers. The application server searches results in GFS (Google File System) and logs the search queries in logs cluster for quality testing. Google uses Dremel which is a query execution engine to run almost near real-time, ad-hoc queries from search engines. This kind of advantage is not present in MapReduce. Google launched BigQuery which runs queries based on aggregation over billions row tables in a matter of seconds. Google is really advanced in its implementation of big data technologies.
Did you know that users of Facebook upload 500+ terabytes of data per day? To process such large chunks of data, Facebook uses Hive for parallel map-reduce opertions and Hadoop for its data storage. Would you believe me if I say Facebook uses Hadoop cluster which is the largest in the world? Employees also use Cassandra which is fault-tolerant, distributed storage system aiming to manage large amount of structured data across variety of commodity servers. Facebook also uses Scuba to carry out real-time ad-hoc analysis on massive data sets. Hive is used to store large data in Oracle data warehouse. Prism is used to bring out and manage multiple namespaces instead of a single one managed by Hadoop. Facebook also uses many other big data technologies such as Corona, Peregine, among many others.
Oracle
There is an explosive growth like 12.5 billion devices which doesn’t include phones, tablets and PCs. This has helped to increase the research and development in the field of Internet-of-Things and in storage requirements which in turn require database management support. Oracle users use Oracle Advanced Analytics which requires Oracle database to be loaded with data. Oracle advanced analytics provides functionalities such as text mining, predictive analytics, statistical analysis and interactive graphics among many others. HDFS data can be loaded into an Oracle data warehouse using Oracle Loader for Hadoop. This feature is used to link data and search query results from Hadoop to Oracle data warehouse. Oracle Exadata Database Machine provides scalable and high-end performance for all database applications. Oracle is leveraging big data to mainly expand its business in Database management systems.
Other giants in big data
Using Hortonworks Data Platform, big data solutions based on Hadoop is used by Microsoft. Microsoft uses big data on its components like SQL server, HDInsight to better its applications like Excel, SQL Server Reporting Services (SSRS). This is just one among many applications deployed by Microsoft. To manage 1.5 billion retail items across its 200 fulfillment centers, Amazon expertly uses big data technologies. It’s an open secret that Amazon is the unbeatable player among cloud service providers. Many companies use AWS cloud services to run big data operation. Other companies who use big data technologies are VMWare, Teradata, Splunk, IBM, Pentaho, SAP, Tableau.
The future of Big data industry
Big data industry’s market size in 2016 was $37.67. In 2017, it is projected to make $43.4 billion market. Do you know how much the market size will be in 2020? Experts and big data scientists project a staggering $60.91 billion market share. The explosion of unstructured data especially in social media networks has increased the need for big data solutions for data management. Spark seems to overtake Hadoop in big data processing as experts argue that it is 100 times faster in memory. As Google did with BigQuery, tech players are innovating new and improved big data technologies very rapidly. Big data is already combined with artificial intelligence, cloud computing and the resulting technologies from this process will be nothing short of disruptive. Companies across the globe are expertly utilizing big data analytics to drive good business to provide better service to users.
Sources:
- https://www.slideshare.net/jdchitt/how-google-does-big-data-dev-nexus-2014
- http://www.information-age.com/google-launches-big-data-analytics-service-1670768/
- https://www.quora.com/How-does-Google-use-big-data-to-give-effective-search-results
- https://www.quora.com/What-technologies-does-Facebook-use-to-store-its-Big-Data
- https://dzone.com/articles/how-is-facebook-deploying-big-data
- http://www.oracle.com/technetwork/database/options/advanced-analytics/bigdataanalyticswpoaa-1930891.pdf
- https://msdn.microsoft.com/en-us/library/dn749804.aspx
- https://datafloq.com/read/amazon-leveraging-big-data/517
- http://www.datamation.com/big-data/big-data-companies.html
- http://www.zcscompany.com/en/blog/post/855
