To fill in the details of what "John" just explained above:
Say that you have stock portfolio for several years: $t_0, t_1, \ldots, t_m$.
Say that you have $n$ stocks, so that stock $i$ has a vector of prices $X_i$. The length of each price vector is $m$ because there are $m$ years.
Then, for the first year $t_1$:
Calculate the $n$ different arithmetic returns for each stock. This means that stock $i$ will have an arithmetic return during the first year as $r_i = (X_i(1) - X_i(0)) / X_i(0)$.
Add all these stock returns, which gives you the total sum of all the arithmetic returns for the first year $t_1$ like this
$r(1) = r_1 + r_2 + ... + r_n$.
This means, for the first year, you have added returns of the first stock, plus returns of the second stock, ..., plus returns of the last stock $n$. This gives you the total return for all stocks, for the first year.
For year $t_2$:
Do the same. This will give you a total sum of all the arithmetic stock returns during year 2, call it $r(2)$.
$\vdots$
Up to year $t_m$: which gives you total arithmetic return $r(m)$, summing all stock returns during year $t_m$.
Now, to deduce the total portfolio returns for the entire time period, you want to sum the returns for every year:
$r(1) + r(2) + ... + r(m)$ (*)
BUT! You cannot sum the aritmetic returns for all years. You must add the logarithmic returns instead, to calculate the total portfolio returns. Because when you add returns for different years, and you want to pass through time, you must use the logarithmic returns as they are time invariant.
So you must transform $r(1)$ to logarithmic returns like this:
$r_{log}(1) = \ln (1 + r(1))$
So to calculate the total returns for the entire portfolio for all years, do expression (*) like this instead
$\ln (1 + r(1)) + \ln (1 + r(2)) + ... + \ln (1 + r(m))$
and let us call this sum for $R$. This is a sum of logarithmic returns. To transform $R$ back to the normal simple returns, you need to do like this:
$e^R - 1$
and this is your answer, i.e. the total returns for the entire portfolio, during years $t_0,\ldots,t_m$.
So you use arithmetic returns to calculate the stock returns within a given year (because the arithmetic returns preserve the weights of each stock). But when you want to add the returns traveling through time, one year to the next, you must add the logarithmic returns.