Why the Polars path is faster

Options-chain CSV ingestion — the same work, six different mechanisms.

src/csv_to_pandas.py  ·  csv_to_pandas()

The two versions

pandas — beforeeager
def filter_dates(df, col, first_date, last_date):
    first_date = pd.to_datetime(first_date)
    last_date  = pd.to_datetime(last_date)
    date_col   = df[col]

    mask = date_col.between(
        first_date, last_date, inclusive='both')
    filtered_df = df[mask]
    # the whole file already exists in RAM
    # by the time this line runs
    return filtered_df

def add_mid(df):
    df['mid'] = df[['bid','ask']].mean(axis=1)
    return df          # row-wise reduction

def add_spread(df):
    df['spread'] = (df['ask'] - df['bid']) / df['mid']
    return df

options = (
    pd.read_csv(
        csv_path,
        parse_dates=['t_date', 'expiration_date']
    )
    .pipe(filter_dates, 't_date', first_date, last_date)
    .pipe(add_mid)
    .pipe(add_spread)
)
polars — afterlazy
options = (
    pl.scan_csv(
        csv_path,
        try_parse_dates=True,
        schema_overrides={
            "strike": pl.Float32,
            "bid":    pl.Float32,
            "ask":    pl.Float32,
            ...  # 9 columns
        },
    )
    .filter(pl.col("t_date").is_between(...))
    .with_columns(((bid + ask) / 2).alias("mid"))
    .with_columns(((ask - bid) / mid).alias("spread"))
    .collect()
    .to_pandas()
)

Six reasons

1

Predicate pushdown

Largest effect

read_csv is eager: it parses every row of the file into memory, and only then does filter_dates throw most of them away.

scan_csv returns a LazyFrame. Nothing executes until .collect(). By that point the query optimizer has pushed the date predicate down into the reader, so rows outside the backtest window are discarded as they are parsed and never become part of a DataFrame at all.

pandas
10 years of CSV
→
parse ALL rows
→
materialise in RAM
→
discard 80%
→
2 years
polars
10 years of CSV
→
parse + filter in one pass
→
2 years

The more selective the date window, the bigger this gets. It is the reason the lazy API exists.

2

Parallel parsing

Large effect

pandas' CSV reader is a single-threaded C parser. One core does all the byte scanning, delimiter splitting and numeric conversion, while the rest of the machine idles.

Polars splits the file into byte ranges, parses them concurrently across every available core, and stitches the chunks together. For a large CSV — where parsing is the workload — this scales close to core count.

3

Declared schema, and half the bytes

Moderate effect

pandas infers dtypes: sampling values, sometimes promoting a column's type partway through the file, and defaulting every float to float64.

schema_overrides removes the guessing entirely — Polars is told exactly what nine columns are before it reads a byte. Declaring them Float32 also halves the memory footprint of the greeks, bid, ask and strike.

That matters more than it sounds. These operations are memory-bandwidth bound, not compute bound — adding two float arrays is limited by how fast they can be pulled through cache, not by the addition. Half the width is close to twice the throughput.

Column typepandas defaultdeclaredbytes/value
bid, ask, strikefloat64Float328 → 4
iv, delta, gamma, theta, vegafloat64Float328 → 4
4

A row-wise reduction became a column operation

Moderate effect

df[['bid','ask']].mean(axis=1) is the expensive direction in pandas. It slices out a two-column sub-frame, then reduces across columns — cutting against the columnar grain, routed through the block manager, carrying NaN-handling machinery per row.

(bid + ask) / 2 is a straight vectorised operation on two contiguous arrays.

Worth being precise about: this one is a rewrite, not an engine difference. The same change would have made the pandas version substantially faster too. It belongs on the list because it is part of why the new code is quick — but it is not evidence for Polars.
5

No block-manager copies

Smaller effect

A pandas DataFrame is a BlockManager: columns sharing a dtype are consolidated into 2-D numpy blocks. Assigning a new column can force reconsolidation — reallocating and copying an entire block. The old chain did this twice, for mid and spread, on the unfiltered frame.

Polars stores each column as an independent Arrow chunked array. with_columns appends a column; nothing existing is touched, moved or copied.

6

Arrow strings instead of Python objects

Smaller effect

In pandas, ticker and call_put land as object dtype — an array of pointers to individual Python str objects scattered across the heap. Every touch is a dereference plus a reference-count update, under the GIL.

Polars uses Arrow string arrays: one contiguous byte buffer plus an offsets array. Cache-friendly, no Python objects involved, no GIL.

The cost that was accepted on purpose

.collect().to_pandas() converts Arrow buffers back into numpy-backed pandas structures — a real copy, single-threaded, and it reintroduces object dtype for the string columns.

That is a deliberate trade. Every downstream module — option_strategy_simulator, pnl_analyzer, delta_hedger — relies on pandas semantics like .loc[start:end] and .at[]. Converting once at the boundary keeps the change contained to a single function while the savings stay intact, because the conversion runs on the already filtered frame — a fraction of the rows the old path had to build just to reach the same point.