⚡ Performance Optimization Deep Dive

1. Multi-Index Lookup: O(n) → O(1)

The Problem:

Every time you need to find options for a specific entry date and expiration date, the original code was scanning through potentially millions of rows checking if each row matches both dates.

Before (v2):

mask = (date_arr == entry_day) & (exdate_arr == expiry_day)
local_options = options.loc[mask]

This creates a boolean mask by checking every single row - if you have 5 million options contracts, you're doing 5 million comparisons every lookup.

After (v3):

lookup_key = (change_to_categorical(entry_day), change_to_categorical(expiry_day))
if lookup_key in options_indexed.index:
    local_options = options_indexed.loc[lookup_key].reset_index()

What Changed:

You pre-built a multi-index on ['date', 'exdate']. Now pandas maintains a hash table internally. When you lookup (entry_day, expiry_day), it's like using a Python dictionary - instant access.

2. Merge vs isin(): O(n×m) → O(n+m)

The Problem:

You have a large options dataset (n rows) and need to filter it to only symbols you're trading (m symbols). .isin() has quadratic-ish behavior.

Before (v3_2):

filtered_options = _options[_options['symbol'].isin(all_syms)]

What .isin() does internally:

For each of the n rows in _options, it checks if that symbol exists in all_syms (m items). Worst case: n × m comparisons.

After (v3_3):

all_syms_df = pd.DataFrame({'symbol': all_syms})
filtered_options = _options.merge(all_syms_df, on='symbol', how='inner')

What .merge() does internally:

Pandas uses a hash join algorithm:

1. Build hash table from smaller dataset (all_syms_df) - O(m)
2. Probe hash table for each row in larger dataset (_options) - O(n)
3. Total: O(n + m)

3. Vectorization: Python Loop → Pandas Batch Operation

The Problem:

You need to set a flag to 0 for specific dates. Python loops are notoriously slow because of interpreter overhead.

Before (v1):

business_days = pd.date_range(start=start_date, end=end_date, freq=BDay())

for day in business_days:
    if day in opt['cond_exit']:
        dict_options[o].at[day, 'nopnl'] = 0

What's expensive here:

• Python for loop: interpreter overhead for each iteration
• .at[day, 'nopnl']: individual cell access, no optimization
• if day in opt['cond_exit']: checking membership in each iteration

If you have 250 business days, that's 250 Python-level operations, each with full overhead.

After (v2):

business_days = pd.date_range(start=start_date, end=end_date, freq=BDay())
cond_exit_days = opt['cond_exit'].index

update_days = business_days.intersection(cond_exit_days)

dict_options[o].loc[update_days, 'nopnl'] = 0

What changed:

1. .intersection(): Set operation done in C code (pandas/numpy), not Python
2. .loc[update_days, 'nopnl'] = 0: Single vectorized assignment
   • Pandas identifies all rows matching update_days in one pass
   • Updates entire column slice in optimized C code
   • Memory is written contiguously, cache-friendly

4. CSV Loading: Eager pandas → Lazy Polars

The Problem:

Loading the raw options CSV with pd.read_csv reads every row and every column into memory on a single thread, parses all dates, then filters to your date window afterward. On a multi-gigabyte options file, you pay the full I/O and parse cost before you've discarded a single row you don't need.

Before (pandas):

options = (
    pd.read_csv(csv_path, parse_dates=['t_date', 'expiration_date'])
    .pipe(filter_dates, 't_date', first_date, last_date)
    .pipe(add_mid)
    .pipe(add_spread)
)

Read everything → then mask. Single-threaded parse, float64 everywhere, and the date filter runs only after the entire file is already in RAM.

After (Polars lazy scan):

options = (
    pl.scan_csv(
        csv_path,
        try_parse_dates=True,
        schema_overrides={
            "strike": pl.Float32, "bid": pl.Float32, "ask": pl.Float32,
            "pxunder": pl.Float32, "iv": pl.Float32, "delta": pl.Float32,
            "gamma": pl.Float32, "theta": pl.Float32, "vega": pl.Float32,
        },
    )
    .filter(
        pl.col("t_date").is_between(
            pl.lit(first_date).str.to_datetime(),
            pl.lit(last_date).str.to_datetime(),
        )
    )
    .with_columns(((pl.col("bid") + pl.col("ask")) / 2).alias("mid"))
    .with_columns(((pl.col("ask") - pl.col("bid")) / pl.col("mid")).alias("spread"))
    .collect()
    .to_pandas()
)

What Changed — six distinct mechanisms:

1. Predicate pushdown Largest effect

read_csv is eager: it parses every row of the file into memory, and only then does filter_dates throw most of them away. scan_csv returns a LazyFrame — nothing executes until .collect(). By that point the query optimizer has pushed the date predicate down into the reader, so rows outside the backtest window are discarded as they are parsed and never become part of a DataFrame at all.

The more selective the date window, the bigger this gets. It is the reason the lazy API exists.

2. Parallel parsing Large effect

pandas' CSV reader is a single-threaded C parser. One core does all the byte scanning, delimiter splitting and numeric conversion, while the rest of the machine idles. Polars splits the file into byte ranges, parses them concurrently across every available core, and stitches the chunks together. For a large CSV — where parsing is the workload — this scales close to core count.

3. Declared schema, and half the bytes Moderate effect

pandas infers dtypes: sampling values, sometimes promoting a column's type partway through the file, and defaulting every float to float64. schema_overrides removes the guessing entirely — Polars is told exactly what nine columns are before it reads a byte. Declaring them Float32 also halves the memory footprint of the Greeks, bid, ask and strike.

That matters more than it sounds. These operations are memory-bandwidth bound, not compute bound — adding two float arrays is limited by how fast they can be pulled through cache, not by the addition. Half the width is close to twice the throughput.

Column type	pandas default	Declared	Bytes/value
bid, ask, strike	float64	Float32	8 → 4
iv, delta, gamma, theta, vega	float64	Float32	8 → 4

4. A row-wise reduction became a column operation Moderate effect

df[['bid','ask']].mean(axis=1) is the expensive direction in pandas. It slices out a two-column sub-frame, then reduces across columns — cutting against the columnar grain, routed through the block manager, carrying NaN-handling machinery per row. (bid + ask) / 2 is a straight vectorised operation on two contiguous arrays.

5. No block-manager copies Smaller effect

A pandas DataFrame is a BlockManager: columns sharing a dtype are consolidated into 2-D numpy blocks. Assigning a new column can force reconsolidation — reallocating and copying an entire block. The old chain did this twice, for mid and spread, on the unfiltered frame. Polars stores each column as an independent Arrow chunked array; with_columns appends a column, and nothing existing is touched, moved or copied.

6. Arrow strings instead of Python objects Smaller effect

In pandas, ticker and call_put land as object dtype — an array of pointers to individual Python str objects scattered across the heap. Every touch is a dereference plus a reference-count update, under the GIL. Polars uses Arrow string arrays: one contiguous byte buffer plus an offsets array. Cache-friendly, no Python objects involved, no GIL.

The cost that was accepted on purpose: