7 Pandas Tricks to Handle Large Datasets
Image by Editor
Introduction
Large dataset handling in Python is not exempt from challenges like memory constraints and slow processing workflows. Thankfully, the versatile and surprisingly capable Pandas library provides specific tools and techniques for dealing with large — and often complex and challenging in nature — datasets, including tabular, text, or time-series data. This article illustrates 7 tricks offered by this library to efficiently and effectively manage such large datasets.
1. Chunked Dataset Loading
By using the chunksize argument in Pandas’ read_csv() function to read datasets contained in CSV files, we can load and process large datasets in smaller, more manageable chunks of a specified size. This helps prevent issues like memory overflows.
|
import pandas as pd
def process(chunk): “”“Placeholder function that you may replace with your actual code for cleaning and processing each data chunk.”“” print(f“Processing chunk of shape: {chunk.shape}”)
chunk_iter = pd.read_csv(“https://raw.githubusercontent.com/frictionlessdata/datasets/main/files/csv/10mb.csv”, chunksize=100000) for chunk in chunk_iter: process(chunk) |
2. Downcasting Data Types for Memory Efficiency Optimization
Tiny changes can make a big difference when they are applied to a large number of data elements. This is the case when converting data types to a lower-bit representation using functions like astype(). Simple yet very effective, as shown below.
For this example, let’s load the dataset into a Pandas dataframe (without chunking, for the sake of simplicity in explanations):
|
url = “https://raw.githubusercontent.com/frictionlessdata/datasets/main/files/csv/10mb.csv” df = pd.read_csv(url) df.info() |
|
# Initial memory usage print(“Before optimization:”, df.memory_usage(deep=True).sum() / 1e6, “MB”)
# Downcasting the type of numeric columns for col in df.select_dtypes(include=[“int”]).columns: df[col] = pd.to_numeric(df[col], downcast=“integer”)
for col in df.select_dtypes(include=[“float”]).columns: df[col] = pd.to_numeric(df[col], downcast=“float”)
# Converting object/string columns with few unique values to categorical for col in df.select_dtypes(include=[“object”]).columns: if df[col].nunique() / len(df) < 0.5: df[col] = df[col].astype(“category”)
print(“After optimization:”, df.memory_usage(deep=True).sum() / 1e6, “MB”) |
Try it yourself and notice the substantial difference in efficiency.
3. Using Categorical Data for Frequently Occurring Strings
Handling attributes containing repeated strings in a limited fashion is made more efficient by mapping them into categorical data types, namely by encoding strings into integer identifiers. This is how it can be done, for example, to map the names of the 12 zodiac signs into categorical types using the publicly available horoscope dataset:
|
import pandas as pd
url = ‘https://raw.githubusercontent.com/plotly/datasets/refs/heads/master/horoscope_data.csv’ df = pd.read_csv(url)
# Convert ‘sign’ column to ‘category’ dtype df[‘sign’] = df[‘sign’].astype(‘category’)
print(df[‘sign’]) |
4. Saving Data in Efficient Format: Parquet
Parquet is a binary columnar dataset format that contributes to much faster file reading and writing than plain CSV. Therefore, it might be a preferred option worth considering for very large files. Repeated strings like the zodiac signs in the horoscope dataset introduced earlier are also internally compressed to further simplify memory usage. Note that writing/reading Parquet in Pandas requires an optional engine such as pyarrow or fastparquet to be installed.
|
# Saving dataset as Parquet df.to_parquet(“horoscope.parquet”, index=False)
# Reloading Parquet file efficiently df_parquet = pd.read_parquet(“horoscope.parquet”) print(“Parquet shape:”, df_parquet.shape) print(df_parquet.head()) |
5. GroupBy Aggregation
Large dataset analysis usually involves obtaining statistics for summarizing categorical columns. Having previously converted repeated strings to categorical columns (trick 3) has follow-up benefits in processes like grouping data by category, as illustrated below, where we aggregate horoscope instances per zodiac sign:
|
numeric_cols = df.select_dtypes(include=[‘float’, ‘int’]).columns.tolist()
# Perform groupby aggregation safely if numeric_cols: agg_result = df.groupby(‘sign’)[numeric_cols].mean() print(agg_result.head(12)) else: print(“No numeric columns available for aggregation.”) |
Note that the aggregation used, an arithmetic mean, affects purely numerical features in the dataset: in this case, the lucky number in each horoscope. It may not make too much sense to average these lucky numbers, but the example is just for the sake of playing with the dataset and illustrating what can be done with large datasets more efficiently.
6. query() and eval() for Efficient Filtering and Computation
We will add a new, synthetic numerical feature to our horoscope dataset to illustrate how the use of the aforementioned functions can make filtering and other computations faster at scale. The query() function is used to filter rows that accomplish a condition, and the eval() function applies computations, typically among multiple numeric features. Both functions are designed to handle large datasets efficiently:
|
df[‘lucky_number_squared’] = df[‘lucky_number’] ** 2 print(df.head())
numeric_cols = df.select_dtypes(include=[‘float’, ‘int’]).columns.tolist()
if len(numeric_cols) >= 2: col1, col2 = numeric_cols[:2]
df_filtered = df.query(f“{col1} > 0 and {col2} > 0”) df_filtered = df_filtered.assign(Computed=df_filtered.eval(f“{col1} + {col2}”))
print(df_filtered[[‘sign’, col1, col2, ‘Computed’]].head()) else: print(“Not enough numeric columns for demo.”) |
7. Vectorized String Operations for Efficient Column Transformations
Performing vectorized operations on strings in pandas datasets is a seamless and almost transparent process that is more efficient than manual alternatives like loops. This example shows how to apply a simple processing on text data in the horoscope dataset:
|
# We set all zodiac sign names to uppercase using a vectorized string operation df[‘sign_upper’] = df[‘sign’].str.upper()
# Example: counting the number of letters in each sign name df[‘sign_length’] = df[‘sign’].str.len()
print(df[[‘sign’, ‘sign_upper’, ‘sign_length’]].head(12)) |
Wrapping Up
This article showed 7 tricks that are often overlooked but are simple and effective to implement when using the Pandas library to manage large datasets more efficiently, from loading to processing and storing data optimally. While new libraries focused on high-performance computation on large datasets are recently arising, sometimes sticking to well-known libraries like Pandas might be a balanced and preferred approach for many.
