The hit is not on the server; it is for the user. Most browsers are not actually that hot on caching web fonts, and it's only because Google does most of the work for you that you don't realise how messed up the situation really is (Google has to serve 4 different *types* of font file depending on the browser receiving it), and in some cases I've observed both Firefox and Chrome not caching at all and forcibly downloading it every page load - which often adds 200KB+ per page load.
And if you use privacy mode, you're definitely going to be incurring more hits on it.
200KB per page load doesn't sound much but it's significant on mobile phones (which, surprisingly enough, will NOT cache things; I can't speak for Android browsers, but on iOS devices, items over 25KB are not internally cached, or at least weren't a short while ago - and I don't see this being any different), for users on lower bandwidth (especially metered bandwidth) and it is noticeably different even on users with higher bandwidth capacity.
Also it's unclear whether Google will rank you lower for having an external dependency on the page that needs to be resolved.
If anything, the hit on the *server* is less than a portal or shoutbox, though if either were properly designed even that shouldn't really be a problem either.
Let's also hope you don't want to use SSL as you won't be able to use SSL safely with this unless you're hosting the files yourself...
