Here's a neat guide from Mike Thorne from Coins Magazine, reprinted by Numismaster, on "19th Century Type Coins Worth Owning." In the article, Thorne lists 13 different US coins from the 1800s. "Not only are they affordable at $100 or less apiece, they’re also attractive and interesting. You can’t ask for much more than that," he says.
Personally, I collect coins that were minted during or immediately before the Civil War, as I spent my years as a full-time historical researcher mainly focused on the Civil War. I enjoy holding coins that may have passed through the hands of some of the soldiers whose letters and diaries I've held.