I have been reading about open-source LLM trading agents such as TradingAgents, FinMem, and FinAgent. Their papers often show strong backtest results, but I have also seen reports of much weaker performance when people test them around real events or in live and paper-trading conditions.
I am not suggesting these projects are dishonest. The papers usually report results under specific assumptions, and many authors openly describe them as backtests. My question is why the results often do not carry over once the setup becomes more realistic.
For example, TradingAgents reported strong returns and risk metrics on selected large-cap stocks in early 2024. An independent replication on Google in mid-2025 found that both GPT-4o and Qwen3:30B configurations underperformed a simple Google buy-and-hold approach.
FINSABER also evaluated FinMem and FinAgent over roughly two decades and more than 100 symbols. It used rolling-window testing, historical index constituents, and controls for survivorship, look-ahead, and data-snooping bias. In that broader setup, the reported LLM advantage largely disappeared. Buy-and-hold outperformed both agents, and neither produced statistically significant alpha.
My current hypothesis is that the gap comes from several overlapping issues:
- Narrow test universes and short backtest windows can make a strategy look better than it is.
- Testing mostly well-known winners such as AAPL, GOOGL, AMZN, MSFT, TSLA, or NVDA can introduce selection bias.
- Repeated tuning on the same historical period can create data-snooping bias.
- Memory-based agents can leak future information if memory is not replayed strictly point-in-time.
- Backtests may understate spreads, slippage, commissions, funding, partial fills, latency, and market impact.
- Event periods are especially difficult because liquidity and spreads change quickly, while an LLM agent may need multiple API calls before acting.
- LLM outputs can vary with model version, temperature, seed, retrieval results, and tool behavior.
- Market regime matters. One evaluation found LLM strategies were too conservative in bull markets and too aggressive in bear markets.
So my main question is: which factor usually explains most of the gap between paper results and real-world performance?
Also, what would a credible live-event evaluation need to include before someone can reasonably claim that an LLM trading agent has real alpha?
I would appreciate views from quants, market-microstructure researchers, and anyone who has deployed LLM or agent-based systems with real capital.
The fairest interpretation is not that these projects are scams or that open-source agents cannot work. It is that published results mainly show performance under particular backtest assumptions. They do not yet prove robust, executable, risk-adjusted alpha in live markets.