The interviewer meant that he's smart. Quoting Senior VP of People operations at Google,
On the hiring side, we found that brainteasers are a complete waste of
time. How many golf balls can you fit into an airplane? How many gas
stations in Manhattan? A complete waste of time. They don’t predict
anything. They serve primarily to make the interviewer feel smart.
Putting that aside, one possible approach would be to invoke Von Neumann-Morgernstern expected utility to construct a certainty equivalent value for the gamble based upon your level of risk aversion.
Utility functions are used to define a total order over possible outcomes and hence can represent complete, transitive preferences: outcome $X$ is preferred to $Y$ if and only if the utility function assigns $X$ higher utility. Expected utility extends classic utility theory to stochastic outcomes by defining the overall utility $U$ of a stochastic outcome $X$ as the expectation of a bernoulli utility function $u$ whose curvature $-\frac{u''}{u'}$ formalizes a notion of risk aversion.
$$ U(X) = \mathbb{E}[u(X)]$$
(Small note: the curvature of $u$ here is extremely important, representing risk aversion, while the curvature of $U$ is irrelevant: any monotonic, increasing transformation of an overall utility function $U$ represents the same preferences.)
A nice Bernoulli utility function $u$ to use is power utility. In a special case this is simply log utility: $u(x) = \log(x)$.
Let $w$ be a scalar representing your wealth. Let $Z$ be payoff from the dice roll (i.e. 1 dollar if dice rolls 1 etc...) Let $c$ be the certainty equivalent of the gamble. The certainty equivalent gives you the same expected utility as your gamble hence $c$ solves the equation:
$$u(w + c) = \mathrm{E}[ u(w + Z) ] $$
With log utility:
$$ \log(w + c) = \frac{1}{6}\sum_{i=1}^6 \log(w + i) $$
If we have log utility and a wealth of one million dollars ($w = 1,000,000$), then I compute the certainty equivalent of the gamble as $c = 3.49999854$. So it's not 3.5 dollars, but really, it's basically the same unless you pump up your risk aversion or scale up the gamble. (And that wealth is probably dramatically too low if you take into account the present value of all future wages.)
This analysis of course doesn't take into account the value of the time wasted talking about this gamble. A few dollar bet is almost certainly too small to be worth meaningful analysis.