I am currently doing the question on pricing the option with payoff:
$$\max (S(S-K),0).$$
On the relevant question section, it's asked why would a bank be reluctant to sell such option? I can't really think of a convinced answer for this so would appreciate any thoughts here.
For large values of the spot S, this payout goes to infinity like the square of S. However, the hedging instruments available are vanilla options, which go like S to the first power. Mathematically, the payout can be replicated from a continuous portfolio of vanilla options, and this is what a bank would try to do. However, the weights of the vanilla options might become very large, and in fact there is a perfectly reasonable vanilla option market where the arbitrage-free price of this payout (obtained by replication from vanillas) is infinite. This happens when the smile goes like the famous Roger Lee bounds asymptotically. The outcome all depends on the asymptotic behaviour of the vanilla smile. The upshot is that risk can become unstable and change a lot for small changes in the implied volatility smile, and the bank has no way of knowing the true value.
I suspect this is because, conditional on being in-the-money, the payoff of your option is convex in stock price $-$ whereas for a vanilla call, the payoff is linear. As a consequence, the delta $\Delta$ and gamma $\Gamma$ hedge ratios are larger, in particular gamma becomes much more sizeable.
Let us assume that rates are null to lighten notation. Then your payoff can be priced under the stock measure $\mathcal{S}$, see for example this answer, such that: \begin{align} V(t,S_t) &=E^\mathcal{Q}\left(S_T(S_T-K)^+|\mathscr{F}_t\right) \\ \tag{1} &=S_tE^\mathcal{S}\left((S_T-K)^+|\mathscr{F}_t\right) \end{align} As you can see in the linked question, under its own measure the stock price is still distributed like a Geometric Brownian Motion, but with drift $r+\sigma^2=\sigma^2$ due to the null rates assumption. Black-Scholes formula applies to $(1)$ and we get: $$V(t,S_t)=S_tf(t)$$ where $f(t):=f(t,S_t,T,K)$ is the Black-Scholes pricing formula for a vanilla call option but such that the stock price has drift $\sigma^2$. Therefore: \begin{align} \Delta_V(t,S_t)&=f(t)+S_t\Delta_{BS}(t,S_t) \\[6pt] \Gamma_V(t,S_t)&=2\Delta_{BS}(t,S_t)+S_t\Gamma_{BS}(t,S_t) \end{align} These hedge ratios should be much larger than for a plain vanilla call, in particular due to the $S_t$ factor: for example for a stock price of \$10 that might yield hedge ratios more than 10 times larger than for the vanilla call.
