⚡ Performance Optimization Deep Dive

1. Multi-Index Lookup: O(n) → O(1)

The Problem:

Every time you need to find options for a specific entry date and expiration date, the original code was scanning through potentially millions of rows checking if each row matches both dates.

Before (v2):

mask = (date_arr == entry_day) & (exdate_arr == expiry_day)
local_options = options.loc[mask]

This creates a boolean mask by checking every single row - if you have 5 million options contracts, you're doing 5 million comparisons every lookup.

After (v3):

lookup_key = (change_to_categorical(entry_day), change_to_categorical(expiry_day))
if lookup_key in options_indexed.index:
    local_options = options_indexed.loc[lookup_key].reset_index()

What Changed:

You pre-built a multi-index on ['date', 'exdate']. Now pandas maintains a hash table internally. When you lookup (entry_day, expiry_day), it's like using a Python dictionary - instant access.

2. Merge vs isin(): O(n×m) → O(n+m)

The Problem:

You have a large options dataset (n rows) and need to filter it to only symbols you're trading (m symbols). .isin() has quadratic-ish behavior.

Before (v3_2):

filtered_options = _options[_options['symbol'].isin(all_syms)]

What .isin() does internally:

For each of the n rows in _options, it checks if that symbol exists in all_syms (m items). Worst case: n × m comparisons.

After (v3_3):

all_syms_df = pd.DataFrame({'symbol': all_syms})
filtered_options = _options.merge(all_syms_df, on='symbol', how='inner')

What .merge() does internally:

Pandas uses a hash join algorithm:

1. Build hash table from smaller dataset (all_syms_df) - O(m)
2. Probe hash table for each row in larger dataset (_options) - O(n)
3. Total: O(n + m)

3. Vectorization: Python Loop → Pandas Batch Operation

The Problem:

You need to set a flag to 0 for specific dates. Python loops are notoriously slow because of interpreter overhead.

Before (v1):

business_days = pd.date_range(start=start_date, end=end_date, freq=BDay())

for day in business_days:
    if day in opt['cond_exit']:
        dict_options[o].at[day, 'nopnl'] = 0

What's expensive here:

• Python for loop: interpreter overhead for each iteration
• .at[day, 'nopnl']: individual cell access, no optimization
• if day in opt['cond_exit']: checking membership in each iteration

If you have 250 business days, that's 250 Python-level operations, each with full overhead.

After (v2):

business_days = pd.date_range(start=start_date, end=end_date, freq=BDay())
cond_exit_days = opt['cond_exit'].index

update_days = business_days.intersection(cond_exit_days)

dict_options[o].loc[update_days, 'nopnl'] = 0

What changed:

1. .intersection(): Set operation done in C code (pandas/numpy), not Python
2. .loc[update_days, 'nopnl'] = 0: Single vectorized assignment
   • Pandas identifies all rows matching update_days in one pass
   • Updates entire column slice in optimized C code
   • Memory is written contiguously, cache-friendly

4. CSV Loading: Eager pandas → Lazy Polars

The Problem:

Loading the raw options CSV with pd.read_csv reads every row and every column into memory on a single thread, parses all dates, then filters to your date window afterward. On a multi-gigabyte options file, you pay the full I/O and parse cost before you've discarded a single row you don't need.

Before (pandas):

options = (
    pd.read_csv(csv_path, parse_dates=['t_date', 'expiration_date'])
    .pipe(filter_dates, 't_date', first_date, last_date)
    .pipe(add_mid)
    .pipe(add_spread)
)

Read everything → then mask. Single-threaded parse, float64 everywhere, and the date filter runs only after the entire file is already in RAM.

After (Polars lazy scan):

options = (
    pl.scan_csv(
        csv_path,
        try_parse_dates=True,
        schema_overrides={
            "strike": pl.Float32, "bid": pl.Float32, "ask": pl.Float32,
            "pxunder": pl.Float32, "iv": pl.Float32, "delta": pl.Float32,
            "gamma": pl.Float32, "theta": pl.Float32, "vega": pl.Float32,
        },
    )
    .filter(
        pl.col("t_date").is_between(
            pl.lit(first_date).str.to_datetime(),
            pl.lit(last_date).str.to_datetime(),
        )
    )
    .with_columns(((pl.col("bid") + pl.col("ask")) / 2).alias("mid"))
    .with_columns(((pl.col("ask") - pl.col("bid")) / pl.col("mid")).alias("spread"))
    .collect()
    .to_pandas()
)

What Changed:

1. Lazy scan + predicate pushdown: scan_csv builds a query plan, so the date .filter is pushed down into the reader. Rows outside [first_date, last_date] are dropped during parsing, not after.
2. Multithreaded parsing: Polars parses the CSV across all cores; pd.read_csv is single-threaded. Usually the dominant win on wide files.
3. Narrow dtypes: Forcing Float32 on prices and Greeks halves memory vs pandas' default float64 — less allocation, better cache behavior.
4. Fused column math: mid and spread are computed inside the same query plan rather than as two separate pandas passes.