We often forget the "duties" part........

 But in itself morality consists in the "side constraints," to use Nozick's expression, that make agreement rather than coercion into the foundation of our social conduct.  These side constraints are embodied in a system of rights and duties: around each individual is a wall of rights that protect him or her from unjust coercion, and on every individual is imposed a set of duties by which those rights are guaranteed.

-Roger Scruton, On Human Nature