I'm currently pricing American and European options on an underlying with sparse data (interpolated), high annual volatility and returns over the last year around 300%. The product isn't similar to anything quoted in the market at the moment, nor does the underlying have anything particularly correlated to it. Does anyone have any suggestions on how to go about this please? I've started with Heston's stochastic volatility model but as there's no option data available and calibrating it on price data doesn't seem to work due to the high sensitivity of the parameters probably so won't give such an accurate price. I realise this will be hard to get a good price on but need a decent idea of what it should be. Thanks!
I would take one step back and focus on the underlying.
Dissecting the underlying to identify components that you can find replacements with liquidity, that can be used in your pricing model. This will give you a baseline, with more data points. Since this sounds like some really exotic stuff, add a nice liquidity premium (as also suggested by Antoine Conze, in the comments of the question). Doing it this way obviously require good monitoring of your model over time, as the model fit might slide as the liquidity and 'characteristics' change over time.
Once you have a good / better pricing source, your options opens up and it's time to start the option modeling (pardon the pun).
In general dealing with modeling, there's the saying: Garbage in, garbage out.