Part of the cost goes to consular services. So if you need a passport to go to Boston, once, you will pay the exact same fee as globetrotters. How does that make sense?
It will make even less sense next year when you will need a passport to drive or walk across the border.
So I say abolish the passport fee. It should be paid via taxes.