Suppose I have trend following strategy(on close to close data) that is not getting acceptable returns for some time. When should I start thinking about shutting it down?
There are really a few issues here:
1) When do I turn off a model because I believe the model is invalid?
This is a subjective call and depends on many things such as how strong the economic reasoning behind the model is, how crowded the space is, and how poorly the model is performing relative to backtest. With regard to the last point, as a rule of thumb I expect to live through a drawdown twice as bad as the worst drawdown in my backtest, since the backtest is overfit. If you have a Sharpe 5 model and realize a Sharpe 0, it is an easy call to shut down. But with trend following, you are probably getting a Sharpe around 1, so you can easily have poor performance for a few years without the model being statistically invalid.
2) How do I time turning off a model using past performance or some other variable, hoping to turn it back on later.
I have not had much luck with model timing. You are layering a tenuous prediction on top of a tenuous prediction. The only timing models that I know of are conditioning some high freq strats on volatility because volatility tends to be autocorrelated and the economics of high vola regimes rewards liquidity providers.
3) How do I size my portfolio to preserve capital?
You can either size down or stop trading when you start losing too much money. There are many algorithms for this. This is like buying a put option, so expect to pay for it. Or you can size your strat to be able to handle your expected worst performance (such as 2x mdd) comfortably and keep a constant size.
You ought to have pre-determined "kill" switches, like a maximum allowable drawdown or time-from-high. Ideally you should get an idea of what these values would be from your backtests.
When you do shutdown a model, don't just throw away the code. The strategy might not be working at the moment, but it could come back in the future. I just heard that models turned-off years ago are now working as of this week because of the higher volatility. Of course it helps if you can do automated paper trading with reasonable accuracy.
You can use a equity based model. Stop trading when your equity drops below your "X-day" equity moving average, and resume trading when your equity crosses above the "X day" equity moving average. You could also do this by measuring the slope of the curve, and not trading when the slope is statistically below 0. I like this method because it does not tie your decision to whether or not the trend following strategy does not work, or even I there is really a trend, but is purely based upon monetary concerns.
I typically have several tests.
Like Ralph Winters said, one calculation to use is the equity curve. I use "...equity curve slope must be greater than x1..." to continue using a model. Another test is, "...%wins must be greater than x2...". Another is, "...average %return per winning transaction must be greater than x3...". Another is "...%return per losing transaction must be less than x4...". Other tests depend on what phase the economy is in, level of various volatilities, etc.
The idea is to make models that can be compared with each other, older models, newer models, and the level of risk that is acceptable.