arguably the entire worth of the dollar as a currency comes from it being what taxpayers pay in, so yeah it kind of does come from taxpayers

Not at all. The dollar’s valuation comes primarily from:

1. Its backing by a stable economic and governmental system and the projected power of the military controlled by that government

2. Its status at the global reserve currency

Tokyo, Lisbon, and Santiago don’t hold debt and equity in USD because Phil from Bozeman paid the IRS in that currency.