Is there an efficient method to having thousands of stocks in your pipeline without causing backtests and your algorithm to lag? I am looking to create an algorithm that will need to minutely poll as many stocks as possible.
When I created an algorithm that had minimal screening on the pipeline so that it can return 5000+ stocks, the backtest lagged so much that it could not complete.
Note: The reason this is necessary is because I do not know what stocks to trade at the start of the day, and need a particular trigger to occur during the day. This trigger can happen to any security in the stock universe and could be effective on the security regardless if it's in the Q500 or a thinly traded stock, so I need to cast a very wide net.