Analyzing Multimodal Data with Large Language Models

Speed up common data science tasks with AI assistants like ChatGPT and Large Language Models (LLMs) from Anthropic, Cohere, Open AI, Google, Hugging Face, and more!

Data Analysis with LLMs teaches you to use the new generation of AI assistants and Large Language Models (LLMs) to aid and accelerate common data science tasks.

Learn how to use LLMs to:

• Analyze text, tables, images, and audio files
• Extract information from multi-modal data lakes
• Classify, cluster, transform, and query multimodal data
• Build natural language query interfaces over structured data sources
• Use LangChain to build complex data analysis pipelines
• Prompt engineering and model configuration

All practical, Data Analysis with LLMs takes you from your first prompts through advanced techniques like creating LLM-based agents for data analysis and fine-tuning existing models. You’ll learn how to extract data, build natural language query interfaces, and much more.

About the technology

Large Language Models (LLMs) can streamline and accelerate almost any data science task. Master the techniques in this book, and you’ll be able to analyze large amounts of text, tabular and graph data, images, videos, and more with clear natural language prompts and a few lines of Python code.

About the book

Data Analysis with LLMs shows you exactly how to integrate generative AI into your day-to-day work as a data scientist. In it, Cornell professor Immanuel Trummer guides you through a series of engaging projects that introduce OpenAI’s Python library, tools like LangChain and LlamaIndex, and LLMs from Anthropic, Cohere, and Hugging Face. As you go, you’ll use AI to query structured and unstructured data, analyze sound and images, and optimize the cost and quality of your data analysis process.

What's inside

• Classify, cluster, transform, and query multimodal data
• Build natural language query interfaces over structured data sources
• Create LLM-based agents for autonomous data analysis
• Prompt engineering and model configuration

About the reader

For data scientists and data analysts who know the basics of Python.

About the author

Immanuel Trummer is an associate professor of computer science at Cornell University and a member of the Cornell Database Group.

Table of Contents

Part 1
1 Analyzing data with large language models
2 Chatting with ChatGPT
Part 2
3 The OpenAI Python library
4 Analyzing text data
5 Analyzing structured data
6 Analyzing images and videos
7 Analyzing audio data
Part 3
8 GPT alternatives
9 Optimizing cost and quality
10 Software frameworks

Immanuel Trummer is an assistant professor for computer science at Cornell University and leader of the Cornell Database Group. His papers have been selected for “Best of VLDB”, “Best of SIGMOD”, for the ACM SIGMOD Research Highlight Award, and for publication in CACM as CACM Research Highlight. Immanuel’s online course on data management has reached over a million views on YouTube. Over the past few years, his group has published extensively on projects that apply large language models in the context of data science.