Some years ago I actually looked at how capital markets developed and new instruments appeared in various frontier and emerging markets that became independent and/or moved to "free-er market" models in 20th century. I can't find my extensive notes right now, so I will write it from memory, and will amend this answer if I find my notes.
This assumes a local fiat currency. If the country uses a regional currency like XAF or XCD, or if it's pegged to a hard currency, then there are small differences.
The local government almost always sells bonds in local currency. Even some "socialist" regimes sell bonds to the population, sometimes even forcing its people to buy bonds. As soon as they decide to have a "free market" economy, they definitely sell bonds. Typically they start with short maturities (up to 1 year, sometimes as short as 1 week) and zero or fixed coupon, and as the markets develop, sell longer maturities.
There are some variations: some countries impose capital controls intended to prevent foreigners from buying the local bonds, which can be circumvented using local access instruments, while others don't mind foreign investors. Sometimes the local equivalent of the Department of Treasury / Ministry of Finance issues some debt, and the local Central Bank also sells some debt, but it's the same yield curve. Many, but not all, sell floaters whose coupon is reset from the results of the most recent auction selling new bonds. Usually the bond program is motivated by the government's need to borrow some money to pay its bills, i.e. fiscal policy, but sometimes the issuance and early repayment of bonds are used as instruments of monetary policy as well.
Many believe that creating a liquid govvy yield curve, or extending an existing one with longer tenors, would be examples of "good things" that good governments are supposed to do. Cynical libertarians might argue that government bureaucrats just create work for themselves. I thought I had an Alexander Hamilton quote about it, but can't find it right now. You may find this example APEC study insightful: https://www.apec.org/docs/default-source/Publications/1999/9/Compendium-of-Sound-Practices-Guidelines-to-Facilitate-the-Development-of-Domestic-Bond-Markets-in-A/99_fmp_soundpractice.pdf . There are many others, just google "initiative on the Development of Domestic Bond Markets". See also this IDB report https://issuu.com/idb_publications/docs/book_en_66518 and these IMF papers https://documents1.worldbank.org/curated/en/334621468740683230/pdf/Developing-government-bond-markets-a-handbook.pdf , https://papers.ssrn.com/sol3/papers.cfm?abstract_id=882875
Many countries only have the primary market - the government sells new bonds, typically at auctions organized in a variety of ways, while others also have secondary market - investors sell existing bonds to each other, over the counter or on exchanges.
As the country engages in foreign trade, investors start to trade FX forwards and cross-currency swaps (local currency fixed), and later various FX options versus USD or EUR. Depending on the capital controls, they use physical delivery or non-delivery. Very few currencies have liquid interest rate options.
Along this path, local currency fixed v float interest rate swaps invariably show up later, if at all: definitely after government bonds, and usually after corporate bonds and cross-currency swaps. I can't think of a currency these days where anyone would want something more complicated than spot FX, and where there isn't already some govvy yield curve. I don't see any theoretical reasons why this can't happen, e.g. if some libertarian-like government chooses not to issue bonds, but I don't believe this currently happens in practice.
One of the challenges, not mentioned in your question, is finding a credible benchmark or index that could be used to reset the coupons on the local currency floating leg. When interest rate swaps first appear in a market, you often different participants using 3-4 different "competing" indices, and eventually one wins out or the regulators encourage the use of one choice.
You also questioned who ends up holding the interest rate risk. Typically/generally, the end users hedge their unwanted market risk by trading with banks, who also have little appetite for holding this risk, so they intern hedge with the people who do have such appetite - typically/generally certain hedge funds and re-insurance companies, who often do have much appetite for almost anything decoupled / exotic / uncorrelated with the rest of the world.
Somewhat related: Question on Xccy swaps curve observability
