Building high-performance financial analytics pipeline with Polars: lazy evaluation, advanced expression and SQL integration
In this tutorial, we dig into the advanced data analytics pipeline used aspecta lightning fast data framework library designed for optimal performance and scalability. Our goal is to show how to leverage Porars’ lazy evaluation, complex expressions, windowing capabilities, and SQL interfaces to efficiently process large financial datasets. We first generate a synthetic financial time series dataset and then move step by step through end-to-end pipelines, from functional engineering and rolling statistics to multidimensional analysis and ranking. Throughout the process, we demonstrate how Polars enables us to write expressive and performance data transformations while keeping memory usage low and ensuring fast execution.
import polars as pl
import numpy as np
from datetime import datetime, timedelta
import io
try:
import polars as pl
except ImportError:
import subprocess
subprocess.run(["pip", "install", "polars"], check=True)
import polars as pl
print("🚀 Advanced Polars Analytics Pipeline")
print("=" * 50)We first import the basic library, including poles for high-performance data framework operations, and NUMPY for generating synthetic data. To ensure compatibility, we added the back-up installation steps for Porars in case the list is not installed yet. Once set up, we signal the beginning of the advanced analytics pipeline.
np.random.seed(42)
n_records = 100000
dates = [datetime(2020, 1, 1) + timedelta(days=i//100) for i in range(n_records)]
tickers = np.random.choice(['AAPL', 'GOOGL', 'MSFT', 'TSLA', 'AMZN'], n_records)
# Create complex synthetic dataset
data = {
'timestamp': dates,
'ticker': tickers,
'price': np.random.lognormal(4, 0.3, n_records),
'volume': np.random.exponential(1000000, n_records).astype(int),
'bid_ask_spread': np.random.exponential(0.01, n_records),
'market_cap': np.random.lognormal(25, 1, n_records),
'sector': np.random.choice(['Tech', 'Finance', 'Healthcare', 'Energy'], n_records)
}
print(f"📊 Generated {n_records:,} synthetic financial records")We generated a rich synthetic financial data set using numpy and performed 100,000 records using numpy and simulated daily inventory data for major stocks such as AAPL and TSLA. Each entry includes key market functions such as price, quantity, bid spread, market caps, and industry. This provides a realistic basis for presenting advanced PORARS analysis on time series datasets.
lf = pl.LazyFrame(data)
result = (
lf
.with_columns([
pl.col('timestamp').dt.year().alias('year'),
pl.col('timestamp').dt.month().alias('month'),
pl.col('timestamp').dt.weekday().alias('weekday'),
pl.col('timestamp').dt.quarter().alias('quarter')
])
.with_columns([
pl.col('price').rolling_mean(20).over('ticker').alias('sma_20'),
pl.col('price').rolling_std(20).over('ticker').alias('volatility_20'),
pl.col('price').ewm_mean(span=12).over('ticker').alias('ema_12'),
pl.col('price').diff().alias('price_diff'),
(pl.col('volume') * pl.col('price')).alias('dollar_volume')
])
.with_columns([
pl.col('price_diff').clip(0, None).rolling_mean(14).over('ticker').alias('rsi_up'),
pl.col('price_diff').abs().rolling_mean(14).over('ticker').alias('rsi_down'),
(pl.col('price') - pl.col('sma_20')).alias('bb_position')
])
.with_columns([
(100 - (100 / (1 + pl.col('rsi_up') / pl.col('rsi_down')))).alias('rsi')
])
.filter(
(pl.col('price') > 10) &
(pl.col('volume') > 100000) &
(pl.col('sma_20').is_not_null())
)
.group_by(['ticker', 'year', 'quarter'])
.agg([
pl.col('price').mean().alias('avg_price'),
pl.col('price').std().alias('price_volatility'),
pl.col('price').min().alias('min_price'),
pl.col('price').max().alias('max_price'),
pl.col('price').quantile(0.5).alias('median_price'),
pl.col('volume').sum().alias('total_volume'),
pl.col('dollar_volume').sum().alias('total_dollar_volume'),
pl.col('rsi').filter(pl.col('rsi').is_not_null()).mean().alias('avg_rsi'),
pl.col('volatility_20').mean().alias('avg_volatility'),
pl.col('bb_position').std().alias('bollinger_deviation'),
pl.len().alias('trading_days'),
pl.col('sector').n_unique().alias('sectors_count'),
(pl.col('price') > pl.col('sma_20')).mean().alias('above_sma_ratio'),
((pl.col('price').max() - pl.col('price').min()) / pl.col('price').min())
.alias('price_range_pct')
])
.with_columns([
pl.col('total_dollar_volume').rank(method='ordinal', descending=True).alias('volume_rank'),
pl.col('price_volatility').rank(method='ordinal', descending=True).alias('volatility_rank')
])
.filter(pl.col('trading_days') >= 10)
.sort(['ticker', 'year', 'quarter'])
)
We load the synthetic dataset into a Porars LazyFrame for deferred execution, allowing us to effectively link complex transformations. From there, we enriched the data using time-based features from there and applied advanced technical metrics such as moving averages, RSI and Bollinger bands. We then perform group aggregation through stocks, year and quarters to extract key financial statistics and indicators. Finally, we rank the results by quantity and volatility, filter out insufficient fragments, and sort the data explored intuitively, while leveraging Porars’ powerful lazy evaluation engine to take advantage of its strengths.
df = result.collect()
print(f"n📈 Analysis Results: {df.height:,} aggregated records")
print("nTop 10 High-Volume Quarters:")
print(df.sort('total_dollar_volume', descending=True).head(10).to_pandas())
print("n🔍 Advanced Analytics:")
pivot_analysis = (
df.group_by('ticker')
.agg([
pl.col('avg_price').mean().alias('overall_avg_price'),
pl.col('price_volatility').mean().alias('overall_volatility'),
pl.col('total_dollar_volume').sum().alias('lifetime_volume'),
pl.col('above_sma_ratio').mean().alias('momentum_score'),
pl.col('price_range_pct').mean().alias('avg_range_pct')
])
.with_columns([
(pl.col('overall_avg_price') / pl.col('overall_volatility')).alias('risk_adj_score'),
(pl.col('momentum_score') * 0.4 +
pl.col('avg_range_pct') * 0.3 +
(pl.col('lifetime_volume') / pl.col('lifetime_volume').max()) * 0.3)
.alias('composite_score')
])
.sort('composite_score', descending=True)
)
print("n🏆 Ticker Performance Ranking:")
print(pivot_analysis.to_pandas())Once the lazy pipeline is finished, we collect the results into the data frame and immediately view the first 10 quarters based on the total amount of total USD. This helps us determine periods of intense trading activity. We then take the analysis a step further by grouping the data through stocks to calculate higher-level insights such as lifetime trading volume, average price volatility, and custom composite scores. This multidimensional summary can be compared not only by the original quantity, but also by momentum and risk-adjusted performance, thereby providing a deeper understanding of overall stock behavior.
print("n🔄 SQL Interface Demo:")
pl.Config.set_tbl_rows(5)
sql_result = pl.sql("""
SELECT
ticker,
AVG(avg_price) as mean_price,
STDDEV(price_volatility) as volatility_consistency,
SUM(total_dollar_volume) as total_volume,
COUNT(*) as quarters_tracked
FROM df
WHERE year >= 2021
GROUP BY ticker
ORDER BY total_volume DESC
""", eager=True)
print(sql_result)
print(f"n⚡ Performance Metrics:")
print(f" • Lazy evaluation optimizations applied")
print(f" • {n_records:,} records processed efficiently")
print(f" • Memory-efficient columnar operations")
print(f" • Zero-copy operations where possible")
print(f"n💾 Export Options:")
print(" • Parquet (high compression): df.write_parquet('data.parquet')")
print(" • Delta Lake: df.write_delta('delta_table')")
print(" • JSON streaming: df.write_ndjson('data.jsonl')")
print(" • Apache Arrow: df.to_arrow()")
print("n✅ Advanced Polars pipeline completed successfully!")
print("🎯 Demonstrated: Lazy evaluation, complex expressions, window functions,")
print(" SQL interface, advanced aggregations, and high-performance analytics")We end the pipeline by demonstrating Polars’ elegant SQL interface, running summary queries to analyze the post-2021 stock performance of familiar SQL syntax. This hybrid feature allows us to seamlessly blend expressive disk transformations with declarative SQL queries. To emphasize its efficiency, we print key performance metrics that emphasize lazy evaluation, memory efficiency, and zero replica execution. Finally, we demonstrate how to easily export results in various formats such as Parquet, Arrow, and JSONL, which makes this pipeline both functional and ready for production. Therefore, we used Porars to complete the full-loop high-performance analytical workflow.
In short, we have seen how Polars’ lazy API optimizes complex analytical workflows that would otherwise be sluggish in traditional tools. We have developed a comprehensive financial analytics pipeline, from raw data ingestion to rolling metrics, aggregation of groupings and advanced scoring, all performed at high speeds. Not only that, we also mined Polars’ powerful SQL interface to run familiar queries seamlessly within the data range. This dual ability to write functional expressions and SQL makes Polars a flexible tool for any data scientist.
Check Paper. All credits for this study are to the researchers on the project. Also, please stay tuned for us twitter And don’t forget to join us 100K+ ml reddit And subscribe Our newsletter.
Sana Hassan, a consulting intern at Marktechpost and a dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. He is very interested in solving practical problems, and he brings a new perspective to the intersection of AI and real-life solutions.
