The recent data explosion due to sensor-enabled machines, mobile devices, social media, and cloud computing has been projected to accelerateputting the growth of the global data volume to 44 times between 2009 and 2020 (IDC).
However, traditional databases and existing scalable architectures have not been able to meet the requirements of large volume, high velocity, and wide-variety data, which has been defined in business terms as big data. Thus, it has become absolutely necessary to find innovative technological solutions to address the needs of big data challenges.
Big Data technologies have addressed the problems related to this new data revolution through the use of commodity hardware and distributed, processing architectures to make data analytics in this frenetic pace of business, a reality. The Big Data market was estimated to be $ 6.8 billion in 2012; and is projected to grow by almost 40 percent every year.
Moreover, current businesses are investing at least 10 % of all IT spending on cloud computing, a trend showing upward movement. If you take these two recent trends in combination, then service providers have a great opportunity to offer Big Data-as-a-Service (Bras). Current service providers who offer infrastructure services may actually extend the scope of their services to include data-platform services (PaaS).
The information in this post may benefit service designers and product managers in service provider organizations to help them identify Big Data-as-a-Service opportunities.
IDC predictions for big data market growth
IDC estimated the value of Big Data market to be about $6.8 billion in 2012, growing almost 40 percent every year to $17 billion by 2015. In 2012, this estimated figure indicated 10 percent market share in the total BI and analytics market. Most market research firms have concluded that this trend is steadily growing upwards.
These events have opened up tremendous opportunities for the service market, where service providers have the option of offering big-data and allied services with carefully structured service layers.
Projected size of big-data services market
The Big Data related service markets, not enough information is still available as the market is still in its infancy. But going by the percentage and volume of business workloads moving from in-premise data centers to public or private cloud platforms, you can probably guess the markets will consistently maintain an upward trend in the coming years.
Taking an average of all the figures suggested by leading big data market analysts and research firms, it is fairly safe to conclude that approximately 15 percent of all enterprise IT spending will move to the cloud-based service platforms. Between 2015 and 2021, this service market is expected to grow about 35 percent.
If IDC’s forecast for the big data market growth proves to be true, that is the market reaches the $17 billion target by 2015, then the big-data related services market will be 15 percent of that figure, or $2.55 billion. Thereafter, when the big data market reaches the projected $88 billion in 2021, the big-data related services market will be 35 percent of that, or about $30 billion.
Hadoop services market estimates
Hadoop software revenue was $209.2 million or 11 percent of the total big data software market in 2012. Apache Hadoop being open source, the revenue growth of the Hadoop software market, compared to the hardware or the services markets, is substantially smaller. The 2012 estimation was that the comprehensive Hadoop market (combined hardware, software, & services) bagged 23 percent of the big data market in 2012, which was projected to grow to 31 percent in 2013.
The IDC estimates for Hadoop-as-a-service market in 2012 was about $130 million, projected to grow by 145 percent to $318 million in 2013.
It is difficult to break down the above figures among the different layers of the big data services architecture, but the success of the market in general will largely depend on business demands, in-premises talent pool, and the technology strengths and weaknesses of service-partner ecosystem.
Layers of big-data services
Industry thought leaders have observed that big data as service can only survive as an integrated platform if the serviced are structured around a layered architecture. So here is a snapshot of supposed layers of services on the big data platform:
Big Data as a Service or BDaaS is the bottom-most layer, where typical components are Compute-as-a-Service (CaaS) and Storage-as-a-Service. In this layer, big-data specific applications are usually deployed on a service providers cloud infrastructure. This business philosophy may be seen as an extension of infrastructure as serviceproviding analytics as an extended service on customer data that is already available on the service providers infrastructure.
The data-service layer can be further segmented into data-as service and database-as-service options. The only difference between a database-as-service and data-as-service may be that in a big data scenario, the relational database model has to be scaled up to architectural components such as Hadoop, in-memory processing capabilities, and NoSQL technologies. In case of near real-time analytics services, the PaaS infrastructure will have to provide additional capabilities to collect, store, and analyze real-time data. A good example of this case is an online store that outsources its data center to a cloud-based service provider.
The next layer in the tiered, service architecture model may be Data Platform-as-a- Service or PaaS. Here the service provider is responsible for the upkeep and maintenance of a data-management infrastructure, a data processing environment, and on-demand analytics services delivery tools. In this scenario, the end users are members of the technical communitysuch as data scientists and software programmers, who would be tasked with managing dedicated analytics clusters.
The top-most layer may be the Analytics Software-as-a-Service (SaaS) layer that typically executes algorithms, scripts and queries that data scientists have already developed, or generates reports from dashboards.
While DaaS and PaaS layers are horizontal by design, SaaS may include more vertical-specific solutions. Service providers that want to cater to only the data-analytics service space must select which industry sectors they wish to target simply because an open-ended analytics service approach will not work.
The tremendous flooding of unstructured data in the business environment has consolidated the position of Hadoop as the preferred platform for big data analytics across all types of unstructured data. Now the challenge is to adapt to the fast-changing Hadoop ecosystem for enterprise data management that depends on standardizing core functionality for repeatable processes.
Typically, in a Hadoop-as-a-service scenario, data scientist store unstructured data in a Distributed File System (HDFS) and process it using MapReduce. The HDaaS model enables data scientists to focus on data analytics having to worry about time-consuming set-up, management or tuning of Hadoop cluster for optimized processing capacity. The HDaaS platform is specifically meant for technical data scientists and not for end users who basically use the dashboard features to print out glossy reports.
Additional references
- IDC Worldwide Big Data Technology and Services 2012-2015 Forecast, March 2012
- IDC Worldwide Hadoop-MapReduce Ecosystem Software 2012-2016 Forecast, May 2012
Conclusion
The current business demands for big data technologies and allied services clearly present a large opportunity for the services marketespecially for service providers who are already providing infrastructure services. Along with this large business opportunity comes the challenge of integrating new technologies and tools to provide on-demand, customized services to broad-based customers across disparate industry sectors.
