Very interesting discussion over at Tom Woods, "When Did I Sign This 'Social Contract'?"
Update (21 Aug 12) - My comment on that thread:
Tom, when you received what may be considered stolen goods in obtaining your property which was most likely unjustly expropriated from a number of others who held claim to it before you, you entered into the contract to protect your rights to that property.
NAP is simply a principle, not a contract. It may be a basis behind certain contracts, but no more or less so than a contract based on a principle of mutually assured destruction. It's merely mutually assured non-destruction. MAD vs MAN. :)
Libertarianism seems to fail to consider that rights are nothing more than a conceptual fiction. Nothing is inalienable, except through contract that all sides endeavor to uphold. Rights do not exist in nature, only constant struggle.