I don't know if these are the most commonly traded or most popular (for your definition of popular) but here are a few exotic products that I recall being supported by the flagship product at my former employers.
- Asian - Strike price is dependant on average price throughout the deal, not just at expiry
- Bermudan - So called because it's halfway between a European and an American option i.e. can be called at fixed intervals before the maturity date.
- Digital - Payoffs are a fixed amount payable iff the underlying reaches an agreed price at maturity.
- Barrier - A barrier can be a pre-condition that must be met before an option will payout or can be a condition that renders the option null and void. It takes the form of a price level that the underlying must (or must not) attain. For example, say that you're selling a 1y call option with a strike price of 120 for an underlying stock with a spot price of 100. You introduce a barrier that says the option will not payout if the spot price reaches 140 before maturity. Such a barrier is called an up and out. This limits your exposure but reduces the value of the option. Other barriers are up and in: option will only payout if the spot price goes above a certain value before maturity, down and out: option will not payout if the spot price drops below a certain value before maturity and down and in: option will only payout if the spot price drops below a certain value before maturity.
- Window Barrier - As above except that multiple different barriers can be defined throughout the lifetime of the option e.g. for the first year an up and in barrier is defined, for the second year a down and in etc.
A good example of an IR hybrid is a Convertible Bond. The trade owner buys a corporate bond that they can elect to convert into equity of that company at a later date.
The most exotic products though are collectively referred to as structured products, those whose payoffs contain some element of logic and state. Such products have generally been tailored to meet very specific hedging/speculation scenarios and owing to their complex nature have a significant quantitative pricing and sensitivity overhead.
One class of structured products is the Snowball: a coupon payment is dependent not just on some market condition but also whether the previous coupon payment was made. Coupon payments therefore "snowball".
Another class is Mountain Ranges. These are options on baskets of underlyings e.g. a Himalayan option pays out based on the best performing asset in the basket.
A couple of references can be found here and here.