In this practical book, four Cloudera data scientists present a set of self-contained patterns for performing large-scale data analysis with Spark. The authors bring Spark, statistical methods, and real-world data sets together to teach you how to approach analytics problems by example.
You’ll start with an introduction to Spark and its ecosystem, and then dive into patterns that apply common techniques―classification, collaborative filtering, and anomaly detection among others―to fields such as genomics, security, and finance. If you have an entry-level understanding of machine learning and statistics, and you program in Java, Python, or Scala, you’ll find these patterns useful for working on your own data applications.
Patterns include:
- Recommending music and the Audioscrobbler data set
- Predicting forest cover with decision trees
- Anomaly detection in network traffic with K-means clustering
- Understanding Wikipedia with Latent Semantic Analysis
- Analyzing co-occurrence networks with GraphX
- Geospatial and temporal data analysis on the New York City Taxi Trips data
- Estimating financial risk through Monte Carlo simulation
- Analyzing genomics data and the BDG project
- Analyzing neuroimaging data with PySpark and Thunder
Sandy Ryza is a data scientist at Cloudera and active contributor to the Apache Spark project. He recently led Spark development at Cloudera and now spends his time helping customers with a variety of analytic use cases on Spark. He is also a member of the Hadoop Project Management Committee.
Uri Laserson is a data scientist at Cloudera, where he focuses on Python in the Hadoop ecosystem. He also helps customers deploy Hadoop on a wide range of problems, focusing on life sciences and health care. Previously, Uri cofounded Good Start Genetics, a next generation diagnostics company while working towards a PhD in biomedical engineering at MIT.
Sean Owen is Director of Data Science for EMEA at Cloudera. He has been a significant contributor to the Apache Mahout machine learning project since 2009, and authored its “Taste” recommender framework. He created the Oryx (formerly Myrrix) project for realtime large scale learning on Hadoop, built on lambda architecture principles, and has contributed to Spark and Spark’s MLlib project.
Josh Wills is Cloudera's Senior Director of Data Science, working with customers and engineers to develop Hadoop based solutions across a wide range of industries. He is the founder and VP of the Apache Crunch project for creating optimized MapReduce and Spark pipelines in Java. Prior to joining Cloudera, Josh worked at Google, where he worked on the ad auction system and then led the development of the analytics infrastructure used in Google+.
