Posts

Introduction to Apache Spark

Image
What is Apache Spark? Reasons behind Apache Spark invention: • Exploding Data • Data Manipulation speed Several shortcomings of Hadoop are: • Adherence to its Map Reduce programming model • Limited programming language API options • Not a good fit for iterative algorithms like Machine Learning Algorithms • Pipelining of tasks is not easy What is Spark Apache Spark is an open source data processing framework for performing Big data analytics on distributed computing cluster. Spark Features Spark has several advantages when compared to other big data and Map Reduce technologies like Hadoop and Storm. Spark is faster than Map Reduce and offers low latency due to reduced disk input and output operation. Spark has the capability of in memory computation and operations, which makes the data processing really fast than another Map Reduce. Unlike Hadoop, spark maintains the intermediate results in memory rather than writing every intermediate outpu...
Post a Comment
Read more

Cluster Computing and why is it used in Big Data

Image
Introduction Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating and information privacy. We will talk about big data on a fundamental level and also take a high-level look at some of the processes and technologies currently being. What Is Big Data? An exact definition of "big data" is difficult to nail down because different people use it quite differently. Generally speaking,  big data  is: large datasets the category of computing strategies and technologies that are used to handle large datasets In this context, "large dataset" means a dataset too large to reasonably process or store with traditional tooling or on a single computer. This means that the common scale of big datasets is constantly shifting and may vary s...
1 comment
Read more

Breaking the Manacles...

Image
"Necessity is the mother of invention" This proverb has always caught my attention and will always be close to me. I believe necessity is what kept pushing the human kind to be where it is. The necessity to communicate brought about the invention of the telephone and the necessity to have an artificial sun lead to the invention of the electric bulb. Many instances of necessity have brought out the best of many people and lead to much pathbreaking discoveries and inventions. I could relate to this well because there was one such instance where  necessity  invoked a new passion in me which I cherish till date. This was when I got a new Android smartphone during the time it was newly launched. I was intrigued by the features it has to offer and the limitless apps in the play store which opened doors to unlimited possibilities. The only restraint I had at that time was I couldn't download the apps because I didn't have a wifi router at my home, just a bro...
1 comment
Read more