Efficient Big Data analytics platform in the cloud

Due to the costs and complexity involved, up until just recently only large companies were able to set up and use Big Data analytics platforms. But this has changed radically with the emergence of low-cost and flexibly scalable cloud platforms, which now also enable small and medium-size companies (SMEs) to profit from this technology. But nothing can be achieved with the analytics platform alone – attention also needs to be paid to the data acquisition.

Systematic collection and analysis of data started long before the arrival of computer technology. But that is not what we are interested in now. In terms of computer science, things started to get really interesting in this regard in the 1970s. With the advent of database management systems, it was now possible to efficiently save, retrieve and analyse data. The volume of saved data grew continuously, and in the 1990s the amount of data reached such high levels that conventional database technology was no longer able to cope. At the same time, business intelligence systems gained steadily in popularity from the late 1980s onwards, and new methods needed to be found to bring together the data from all operational systems into a single decision making platform. This was the starting point for data warehouses. Later on, the Internet got bigger and bigger – and the term “Big Data” was coined finally via data mining in the mid-2000s.

This development was driven by large companies like IBM, Microsoft and Oracle. The growth of the Internet saw the emergence of more and more Internet companies, who shaped the way data is mined and found ways to solve the challenges posed by such vast amounts of data. On the users’ side, it seemed inconceivable for a long time that a company that did not have its own data centres and Big Data specialists would ever be able to benefit from the advantages of Big Data analytics. But we are currently seeing huge changes here. In exactly the same way that every SME can now afford its own database, in the near future they will also all be able to use Big Data analytics platforms – no matter how small the company.

This has been made possible by the broad acceptance of cloud services. With offerings of Infrastructure-as-a-Service (IaaS) from cloud service providers, it is possible to set up a big data analytics platform within a short period of time without having to make any upfront investments. Customers don't have to worry about data centres, hardware, networks and operating systems. All they have to do is configure the appropriate computing and storage instances and pay a fee based on consumption. Some cloud service providers even offer the possibility of using computing capacity not used elsewhere for their own system, thus saving up to 90 percent of the usual costs.

Anyone who still finds this platform concept too complex can use special Big Data Services. Google, for example, offers the "BigQuery" service as a serverless, highly scalable and affordable cloud data warehouse with an in-memory business intelligence engine and integrated machine learning. Other cloud service providers such as Amazon and Microsoft have similar solutions in their product range.

Do you want to benefit from the potential of Big Data without having to acquire the necessary know-how? Contact us!