Building Python Real-time Applications With Storm

Learn to process massive real-time data streams using Storm and Python―no Java required!

About This Book

• Learn to use Apache Storm and the Python Petrel library to build distributed applications that process large streams of data
• Explore sample applications in real-time and analyze them in the popular NoSQL databases MongoDB and Redis
• Discover how to apply software development best practices to improve performance, productivity, and quality in your Storm projects

Who This Book Is For

This book is intended for Python developers who want to benefit from Storm's real-time data processing capabilities. If you are new to Python, you'll benefit from the attention to key supporting tools and techniques such as automated testing, virtual environments, and logging. If you're an experienced Python developer, you'll appreciate the thorough and detailed examples

What You Will Learn

• Install Storm and learn about the prerequisites
• Get to know the components of a Storm topology and how to control the flow of data between them
• Ingest Twitter data directly into Storm
• Use Storm with MongoDB and Redis
• Build topologies and run them in Storm
• Use an interactive graphical debugger to debug your topology as it's running in Storm
• Test your topology components outside of Storm
• Configure your topology using YAML

In Detail

Big data is a trending concept that everyone wants to learn about. With its ability to process all kinds of data in real time, Storm is an important addition to your big data “bag of tricks.”

At the same time, Python is one of the fastest-growing programming languages today. It has become a top choice for both data science and everyday application development. Together, Storm and Python enable you to build and deploy real-time big data applications quickly and easily.

You will begin with some basic command tutorials to set up storm and learn about its configurations in detail. You will then go through the requirement scenarios to create a Storm cluster. Next, you'll be provided with an overview of Petrel, followed by an example of Twitter topology and persistence using Redis and MongoDB. Finally, you will build a production-quality Storm topology using development best practices.

Style and approach

This book takes an easy-to-follow and a practical approach to help you understand all the concepts related to Storm and Python.

Kartik Bhatnagar

Kartik Bhatnagar loves nature and likes to visit picturesque places. He is a technical architect in the big data analytics unit of Infosys. He is passionate about new technologies. He is leading the development work of Apache Storm and MarkLogic NoSQL for a leading bank. Kartik has a total 10 years of experience in software development for Fortune 500 companies in many countries. His expertise also includes the full Amazon Web Services (AWS) stack and modern open source libraries. He is active on the StackOverflow platform and is always eager to help young developers with new technologies. Kartik has also worked as a reviewer of a book called Elasticsearch Blueprints, Packt Publishing. In the future, he wants to work on predictive analytics.

Barry Hart

Barry Hart began using Storm in 2012 at AirSage. He quickly saw the potential of Storm while suffering from the limitations of the basic storm.py that it provides. In response, he developed Petrel, the first open source library for developing Storm applications in pure Python. He also contributed some bug fixes to the core Storm project. When it comes to development, Barry has worked on a little of everything: Windows printer drivers, logistics planning frameworks, OLAP engines for the retail industry, database engines, and big data workflows. Barry is currently an architect and senior Python/C++ developer at Pindrop Security, helping fight phone fraud in banking, insurance, investment, and other industries.