Is it possible to replicate the option of a custom index? Yes and you can find OTC market-makers who will make a price. They use portfolio replication to mimic the payoff of the option with a position in the underlying (Black-Scholes, '73). Even though the underlying custom index is not traded it can be perfectly constructed via its traded constituents. So once you sell the option to your customer, you reconstruct the payoff of a long option by dynamically adjusting a position in the underlying. The underlying position is kept delta matched to the option. Delta matched positions are PL neutral over small changes, eg if the custom index changes price by 1USD, an at-the-money call option with a face value of 1,000,000USD will see about equivalent changes in value as a custom index holding of around 500,000USD (excuse the hand-waving).
Here the delta is based on your multivariate price model of the underlying basket. You know the volatilities of the single stocks from the options market. You have to input a correlation matrix, which is the hard part. The upper limit of this correlation matrix is where all the stocks move together, which of course will result in a high price for the option. However the price history of the stocks will generally show some negative correlation, which is what you see under normal conditions.
The problem with baskets it that this negative-correlation-under-normal-conditions makes them seem far less volatile than single assets, leading to people holding massive positions in baskets and their derivatives. However when extreme events occur, correlation can approach 1 and at the same time constituent volatility spikes. Extreme events though are rare (!), so people usually do not have the historic data to be prepared for extreme movements in baskets. In fact the underestimation of basket price volatility is the source of some spectacular financial failures. Nick Leeson bankrupted Barings Bank by selling massive options on Japanese stock indices, the positions blew up when an earthquake toppled Asian markets. The Global Financial Crisis was principally caused by the belief that baskets of credit risks were less risky than they were.
So make the positions small, and know that you are unlikely to have enough price history to confidently predict future outcomes, so account for that in your price.
BTW not sure where the idea that transaction costs prohibit dynamic replication strategies. The options markets of the 80's, 90's boomed because of dynamic hedging, and today automated trading has only reduced transaction costs further.