When120"Profitable"TradingStrategiesFailedMyAudit

“A null result is still a result. The restraint is the point.”

Every quant portfolio contains the same project.

“I built a strategy with a Sharpe Ratio of 2.3.”

Unfortunately, almost everyone knows what usually comes next. It doesn't survive.

After reading years of quantitative finance literature, I noticed something strange. Researchers kept publishing new market anomalies- value, momentum, profitability, accruals, investment, seasonality- but almost nobody asked the uncomfortable question:

If we rerun these strategies today using modern statistical standards, how many actually survive?

So I decided to build an audit instead of another trading strategy. That became AlphaAudit.

# The problem nobody likes talking about

Imagine testing 500 different strategies. Even if markets were completely random, a handful would still look spectacular purely because of chance. This is exactly the same multiple-testing problem seen in medicine and genomics.

The more experiments you perform, the easier it becomes to accidentally “discover” something that never really existed. Traditional backtests rarely account for this- they usually optimize until something looks profitable.

I wanted to measure the opposite: how much of published alpha is actually statistical luck?

# Building an audit instead of a strategy

Rather than creating another prediction model, I built an evaluation pipeline. Every anomaly passes through progressively stricter filters.

Instead of asking “Can this strategy make money?” I ask “Has this strategy earned the right to be believed?” That philosophical difference shaped the entire project.

# The statistics most backtests ignore

Building AlphaAudit forced me into some fascinating research papers. Instead of relying on ordinary Sharpe Ratios, I implemented several techniques directly from the academic literature.

Deflated Sharpe Ratio (DSR)

A Sharpe Ratio of 0.7 sounds impressive unless you tried 300 strategies first. The Deflated Sharpe Ratio adjusts for:

• Multiple comparisons
• Non-normal returns
• Sample length
• Selection bias

In other words, it asks: “Is this Sharpe Ratio actually impressive?” Or is it just the best-looking result from hundreds of failed experiments?

Probability of Backtest Overfitting (PBO)

This became one of my favorite parts of the project. Instead of splitting data once into train/test, I implemented Combinatorially Symmetric Cross Validation (CSCV): every possible combination of in-sample and out-of-sample blocks gets evaluated.

False Discovery Rate

Suppose 120 published anomalies all claim statistical significance. Should we trust all 120? Probably not. So the framework automatically applies:

• Bonferroni correction
• Holm correction
• Benjamini-Hochberg False Discovery Rate

# Markets change. Strategies don't.

Another idea that fascinated me was regime dependence. Some anomalies work only during low volatility, rising interest rates, or pre-2018 markets. Others survive everything.

So I built a regime-conditioning layer that evaluates every anomaly across exogenous macroeconomic environments rather than averaging everything into a single Sharpe Ratio.

A strategy that only works during one economic regime isn't necessarily bad. It just shouldn't be advertised as universal.

# Creating data with known ground truth

This was probably the hardest engineering decision. Real financial markets never tell you the correct answer. So how do you validate an auditing framework?

I built a calibrated synthetic universe. Every generated anomaly secretly belonged to one of four categories:

Because I knew the hidden labels beforehand, I could measure whether AlphaAudit correctly recovered the truth. That transformed the project from “I hope these statistics work” into “I can quantitatively verify that they work.”

# The moment that surprised me

Running the complete pipeline produced one statistic that immediately stood out. Starting with 120 published anomalies…

The surprising part wasn't that strategies decayed- researchers have known that for years. The surprising part was where they failed.

Not because returns disappeared. Not because markets changed. Most failed because once you properly accounted for selection bias, their statistical evidence simply wasn't strong enough anymore. That was the real bottleneck.

# Engineering challenges

Beyond statistics, AlphaAudit became a surprisingly enjoyable software engineering project. The entire framework is deterministic. One command…

…recreates everything:

• Datasets
• Statistical analyses
• Figures & reports
• JSON outputs
• Interactive dashboard

Every experiment is reproducible from a single configuration file. That reproducibility mattered just as much as the statistical models themselves.

# My favorite lesson

Before building AlphaAudit, I believed good quantitative research was about finding profitable strategies. Now I think it's about rejecting bad ones.

Negative results are uncomfortable. They rarely go viral. But they're often the most scientifically valuable outcomes. One sentence from the project eventually became its guiding philosophy:

A null result is still a result. The restraint is the point.

And I think that's exactly what quantitative finance and machine learning more broadly needs more of.

# What's next?

AlphaAudit currently validates its methodology using a synthetic market with known ground truth before applying the same pipeline to real anomaly datasets. The next steps are extending the framework to emerging markets like the NSE, building predictive models for anomaly decay, and incorporating realistic transaction-cost sensitivity to understand how many surviving signals remain economically viable.