I assume that by "this machinery breaks down" you mean that when it breaks down as theory, but not as a practical tool. 

I would say that the exact point where risk neutral pricing approach fails is when the payoff is no more attainable. There exist a precise mathematical characterization for attainable payoffs (see the book of Hans Föllmer, Alexander Schied, "Stochastic Finance: An Introduction in Discrete Time"). And as far as I know, there is no standard approach to valuation and hedging non-attainable payoffs ([for example][1]). Note also that there are many reasons for payoff to be non-attainable and it's hard to define them all precisely.   


  [1]: http://books.google.ch/books?id=cHSdBaYmsCcC&lpg=PA306&ots=7VAIUypFzD&dq=hedging%20non-attainable%20payoff&pg=PA305#v=onepage&q=hedging%20non-attainable%20payoff&f=false