Unsupervised learning and in out of sample

Assume we are given $N$ samples, let's say small timeseries of 1 hour resolution daily exchange rates - for the sake of argument. Each sample is a $24$ element vector $x$.

Then we proceed to do clustering using our favourite unsupervised learning algorithm, say K-means. Assume we use $k$ classes.

Afterwards, we observe the mean return values of the following days conditioned on the class. Meaning, for the days in class $k=1$ we have on average returns $\mu_1$ on the following day.

The statement then is the following, if a day looks like class $c$ then tommorrow you can expect returns $\mu_c$.

Here is my question:

Have we broken any rule in this process? Should we have stored some of the $N$ days for out of sample performance? It seems to me that even talking about out of sample and in sample is meaningless, since the algorithm only uses the vectors $x$ and is absolutely agnostic about the returns on the following day.

• Indeed there's no data snooping IMO. It's just that you'll find out how well your model generalises only when going live. Having split the $N$ data points into a training + test set (or having performed cross-val) would allow you to have an estimate of this generalisation error before going live... which may be interesting! That said, since you don't cluster using the next day's return you'd be lucky to capture a relationship between the belonging to a class and the average next day's return IMO. It's a bit like if you were trying to do supervised learning using unsupervised learning, no? Commented Aug 11, 2017 at 10:03
• This is my understanding as well but I am very cautious with intuition where statistics are involved. RPG's answer below gives me food for thought. Commented Aug 11, 2017 at 10:56

You form your clusters on in-sample data. How well new data conforms to these clusters going forward can only be determined from out-of-sample data. So you are estimating the generalisation of your clusters.

If you don't separate the data then you might find clusters which only appeared in the future, and any associated forward returns would have been linked to those clusters.

• Could you provide a minimum ingredients example, to direct me to one? Commented Aug 26, 2017 at 7:41