A first answer is explained in Market microstructure knowledge needed for controlling an intra-day trading process that is Chapter 21 of Fouque, Jean-Pierre, and Joseph A. Langsam, eds. Handbook on systemic risk. Cambridge University Press, 2013.
If you know some characteristics of your arbitrage opportunities as a function of time (for instance that you have usually more opportunities after the opening than around 2pm), you can tell that to your optimiser. This chapter tells you how to include them in an Almgren-Chriss (ie mean-variance) like optimisation scheme:
Another (and more generic) way to answer to your question is to explain that stochastic control is about planning; to plan you need to have a temporal structure. The simplest one is the one used in the chapter mentioned earlier.
You can have more structure, for instance to formulate your signal(s) as Ornstein-Uhlenbeck processes with different time scales. It is done in C-A L and Eyal Neuman. "Incorporating signals into optimal trading" Finance and Stochastics 23 (2019): 275-311. You will see that it is not easy... The worst is that usually when you trade your signal you influence them, hence you should model a closed loop between the signal and your own impact.