I want to hedge a barrier option, say a knock-out call with strike K and barrier B out-of-the-money. My idea was to start from the payoff diagram of this option, and try to accomodate it with vanilla options, as it can be done for instance in the (approximate) replication of a digital call by means of vanilla call options. However in this case it seems that this method fails simply because ofr such an option the payoff diagram is simply the same of a vanilla call... so what can be done??
there are a number of ways to do this. You do have to make some modelling assumptions, however. eg continuity, BS model holds, or log stock price process is independent of level.
The most common way is to take the pay-off and geometrically reflect in the barrier. (i.e. pass to log coordinates and reflect). i.e. write the function as $f(x)$ where $x= \log S_t,$ now find a function $g$ such that $$ g(x) = f(x) $$ for $x>\log B$ and $$ g(x) = -f(2B-X) $$ for $x < \log B.$
Then statically replicate the $g$ behind the barrier at maturity. (Look up put-call symmetry.)
The resulting contract has close to zero value at the barrier. You then take contracts wholly paying off behind the barrier with varying maturity eg puts struck at $B$ and use them to cancel the value on the barrier iteratively starting at the end and stepping back towards 0. Note this is highly model dependent and requires continuity, however.
There is extensive discussion of these topics in my book the Concepts and Practice of Mathematical Finance.