If you found that the fluid was black and cloudy when you drained it, you are right that the former owner never had the it changed. The automatic transmission fluid (ATF) has probably been there since the vehicle rolled out of the showroom 15 years ago. With that said, though, and with the information that fourth is starting to act up a bit and that you found some rather large metal shards captured by the magnetic plug/bolt, it is also more than a good bet that something is amiss in the bell housing.
Since the Outback is shifting normally in three of the four gears and fourth is acting a little funny, it’s a good bet that you’ll find the clutch pack or band assembly may have had been damaged recently. The damage is still minor enough so that the transmission is working but you will likely have to have your shop take a look at the transmission much sooner than later. The fact also is that if you act now your chances are good of keeping the damage minimum are also excellent. If you wait a bit then the damage will likely be much more extensive.
So, here’s a recommendation: take the Outback to a local transmission specialist, unless you feel competent that you can do the teardown yourself (interestingly you’ll find some rather good training videos on YouTube — just follow them step-by-step and you can do the work, although you will need some specialized tools and a lift or hydraulic floor life), and have the transmission torn down and inspected. Ask for a report on what the specialist finds. The chances are pretty good that you will find any damage has been limited to the clutch packs/bands or the gears (to enable the various gear choices, your transmission uses a number of planetary gears, activated by solenoids and other electronics, to work properly).
If the damage has been limited to these particular systems you are in luck because repairing them will cost you far less than replacing the transmission (actually a tranxaxle). With the inspection and report and repairs, you will still be looking at an investment in the $1,200 to $1,400 range (maybe less), but it is far less than the $3,000 or so you would spend replacing the entire transmission.