If you add a bunch of zeros, then your covariance matrix will be singular, which could lead to problems depending on what you're doing (anything that involves inverting a matrix will have problems).
If you have to have a covariance matrix that includes the risk-free rate, then you need to provide a variance for the risk-free rate (i.e. you're no longer strictly assuming it's risk-free) at a minimum. This is like assuming that you're investing in a short-term floating rate security (and you're implicitly making assumptions about the process of the rates). For instance, if you have a 6 month horizon, you could assume that there is no risk of default in any given month, but that the rate might re-set each month. You still aren't risking default, but you do have some interest rate risk.
If you are using a fixed term risk-free security, like a 1 month T-bill, and your fixed term matches your investment horizon, 1 month in this case, then it will have zero variance at the horizon. In this case, you probably shouldn't be using the risk-free rate in a covariance matrix. Separate out the decisions. This becomes a little bit trickier if you're trying to account for transactions (b/c you have to include both decisions in the optimization then).
If you have a longer horizon, then you can make assumptions about the dynamics of the risk-free rate, such as mean-reversion or its relationship to inflation.