Python Polars: The Definitive Guide

Foreword
    Preface
        Who This Book Is For
            Hanna: The Data Analyst
            Kosjo: The Data Engineer
            A Broader Audience
        Get More Out of This Book
        Conventions Used in This Book
        O’Reilly Online Learning
        How to Contact Us
        Acknowledgments
    I. Begin
    1. Introducing Polars
        What Is This Thing Called Polars?
            Key Features
            Key Concepts
            Advantages
        Why You Should Use Polars
            Performance
            Usability
            Popularity
            Sustainability
        Polars Compared to Other Data Processing Packages
        Why We Focus on Python Polars
        How This Book Is Organized
        An ETL Showcase
            Extract
            Bonus: Visualizing Neighborhoods and Stations
            Transform
            Bonus: Visualizing Daily Trips per Borough
            Load
            Bonus: Becoming Faster by Being Lazy
        Takeaways
    2. Getting Started
        Setting Up Your Environment
            Downloading the Project
            Installing uv
            Installing the Project
            Working with the Virtual Environment
            Verifying Your Installation
        Crash Course in JupyterLab
            Keyboard Shortcuts
        Installing Polars on Other Projects
            All Optional Dependencies
            Optional Dependencies for Interoperability
            Optional Dependencies for Working with Spreadsheets
            Optional Dependencies for Working with Databases
            Optional Dependencies for Working with Remote Filesystems
            Optional Dependencies for Other I/O Formats
            Optional Dependencies for Extra Functionality
            Installing Optional Dependencies
        Configuring Polars
            Temporary Configuration Using a Context Manager
            Local Configuration Using a Decorator
        Compiling Polars from Scratch
            Edge Case: Very Large Datasets
            Edge Case: Processors Lacking AVX Support
        Takeaways
    3. Moving from pandas to Polars
        Animals
        Similarities to Recognize
        Appearances to Appreciate
            Differences in Code
            Differences in Display
        Concepts to Unlearn
            Index
            Axes
            Indexing and Slicing
            Eagerness
            Relaxedness
        Syntax to Forget
            Common Operations Side By Side
        To and From pandas
        Takeaways
    II. Form
    4. Data Structures and Data Types
        Series, DataFrames, and LazyFrames
        Data Types
            Nested Data Types
            Missing Values
        Data Type Conversion
        Takeaways
    5. Eager and Lazy APIs
        Eager API: DataFrame
        Lazy API: LazyFrame
        Performance Differences
        Functionality Differences
            Attributes
            Aggregation Methods
            Computation Methods
            Descriptive Methods
            GroupBy Methods
            Exporting Methods
            Manipulation and Selection Methods
            Miscellaneous Methods
        Tips and Tricks
            Going from LazyFrame to DataFrame and Vice Versa
            Joining a DataFrame with a LazyFrame
            Caching Intermittent Results
        Takeaways
    6. Reading and Writing Data
        Format Overview
        Reading CSV Files
        Parsing Missing Values Correctly
        Reading Files with Encodings Other Than UTF-8
        Reading Excel Spreadsheets
        Working with Multiple Files
        Reading Parquet
        Reading JSON and NDJSON
            JSON
            NDJSON
        Other File Formats
        Querying Databases
        Writing Data
            CSV Format
            Excel Format
            Parquet Format
            Other Considerations
        Takeaways
    III. Express
    7. Beginning Expressions
        Methods and Namespaces
        Expressions by Example
            Selecting Columns with Expressions
            Creating New Columns with Expressions
            Filtering Rows with Expressions
            Aggregating with Expressions
            Sorting Rows with Expressions
        The Definition of an Expression
            Properties of Expressions
        Creating Expressions
            From Existing Columns
            From Literal Values
            From Ranges
            Other Functions to Create Expressions
        Renaming Expressions
        Expressions Are Idiomatic
        Takeaways
    8. Continuing Expressions
        Types of Operations
            Example A: Element-Wise Operations
            Example B: Operations That Summarize to One
            Example C: Operations That Summarize to One or More
            Example D: Operations That Extend
        Element-Wise Operations
            Operations That Perform Mathematical Transformations
            Operations Related to Trigonometry
            Operations That Round and Categorize
            Operations for Missing or Infinite Values
            Other Operations
        Nonreducing Series-Wise Operations
            Operations That Accumulate
            Operations That Fill and Shift
            Operations Related to Duplicate Values
            Operations That Compute Rolling Statistics
            Operations That Sort
            Other Operations
        Series-Wise Operations That Summarize to One
            Operations That Are Quantifiers
            Operations That Compute Statistics
            Operations That Count
            Other Operations
        Series-Wise Operations That Summarize to One or More
            Operations Related to Unique Values
            Operations That Select
            Operations That Drop Missing Values
            Other Operations
        Series-Wise Operations That Extend
        Takeaways
    9. Combining Expressions
        Inline Operators Versus Methods
        Arithmetic Operations
        Comparison Operations
        Boolean Algebra Operations
        Bitwise Operations
        Using Functions
            When, Then, Otherwise
        Takeaways
    IV. Transform
    10. Selecting and Creating Columns
        Selecting Columns
            Introducing Selectors
            Selecting Based on Name
            Selecting Based on Data Type
            Selecting Based on Position
            Combining Selectors
        Creating Columns
        Related Column Operations
            Dropping
            Renaming
            Stacking
            Adding Row Indices
        Takeaways
    11. Filtering and Sorting Rows
        Filtering Rows
            Filtering Based on Expressions
            Filtering Based on Column Names
            Filtering Based on Constraints
        Sorting Rows
            Sorting Based on a Single Column
            Sorting in Reverse
            Sorting Based on Multiple Columns
            Sorting Based on Expressions
            Sorting Nested Data Types
        Related Row Operations
            Filtering Missing Values
            Slicing
            Top and Bottom
            Sampling
            Semi-Joins
        Takeaways
    12. Working with Textual, Temporal, and Nested Data Types
        String
            String Methods
            String Examples
        Categorical
            Categorical Methods
            Categorical Examples
        Enum
        Temporal
            Temporal Methods
            Temporal Examples
        List
            List Methods
            List Examples
        Array
            Array Methods
            Array Examples
        Struct
            Struct Methods
            Struct Examples
        Takeaways
    13. Summarizing and Aggregating
        Split, Apply, and Combine
        GroupBy Context
            The Descriptives
            Advanced Methods
        Row-Wise Aggregations
        Window Functions in Selection Context
        Dynamic Grouping
        Rolling Aggregations
        Upsampling
        Takeaways
    14. Joining and Concatenating
        Joining
            Join Strategies
            Joining on Multiple Columns
            Validation
        Inexact Joining
            Inexact Join Strategies
            Additional Fine-Tuning
            Use Case: Marketing Campaign Attribution
        Vertical and Horizontal Concatenation
            Vertical
            Horizontal
            Diagonal
            Align
            Relaxed
            Stacking
            Appending
            Extending
        Takeaways
    15. Reshaping
        Wide Versus Long DataFrames
        Pivot to a Wider DataFrame
        Unpivot to a Longer DataFrame
        Transposing
        Exploding
        Partition into Multiple DataFrames
        Takeaways
    V. Advance
    16. Visualizing Data
        NYC Bike Trips
        Built-In Plotting with Altair
            Introducing Altair
            Methods in the Plot Namespaces
            Plotting DataFrames
            Too Large to Handle
            Plotting Series
        pandas-Like Plotting with hvPlot
            Introducing hvPlot
            A First Plot
            Methods in the hvPlot Namespace
            pandas as Backup
            Manual Transformations
            Changing the Plotting Backend
            Plotting Points on a Map
            Composing Plots
            Adding Interactive Widgets
        Publication-Quality Graphics with plotnine
            Introducing plotnine
            Plots for Exploration
            Plots for Communication
        Styling DataFrames With Great Tables
        Takeaways
    17. Extending Polars
        User-Defined Functions in Python
            Applying a Function to Elements
            Applying a Function to a Series
            Applying a Function to Groups
            Applying a Function to an Expression
            Applying a Function to a DataFrame or LazyFrame
        Registering Your Own Namespace
        Polars Plugins in Rust
            Prerequisites
            The Anatomy of a Plugin Project
            The Plugin
            Compiling the Plugin
            Performance Benchmark
            Register Arguments
            Using a Rust Crate
            Use Case: geo
        Takeaways
    18. Polars Internals
        Polars’ Architecture
        Arrow
        Multithreaded Computations and SIMD Operations
        The String Data Type in Memory
        ChunkedArrays in Series
        Query Optimization
            LazyFrame Scan-Level Optimizations
            Other Optimizations
        Checking Your Expressions
            meta Namespace Overview
            meta Namespace Examples
        Profiling Polars
        Tests in Polars
            Comparing DataFrames and Series
        Common Antipatterns
            Using Brackets for Column Selection
            Misusing Collect
            Using Python Code in your Polars Queries
        Takeaways
    Appendix. Accelerating Polars with the GPU
        NVIDIA RAPIDS
        Installing the GPU Engine
            Step 1: Install WSL2 on Windows
            Step 2: Install Ubuntu Linux on WSL2
            Step 3: Install Prerequisite Ubuntu Linux Packages
            Step 4: Install the CUDA Toolkit
            Step 5: Install Python Dependencies
            Step 6: Test Your Installation
        Using the Polars GPU Engine
            Configuration
            Unsupported Features
        Benchmarking the Polars GPU Engine
            Solutions
            Queries and Data
            Method
            Results and Discussion
            Conclusion
        The Future of Polars on the GPU
        Takeaways
    Index
    About the Authors