The problem with restricting speech, just like the problem with restricting ANY rights, is the fact that you must have someone with the power to do the restricting, and the deciding of what gets restricted. That power will (sooner rather than later) be abused. It always is, it always will be. Therefore, not giving someone that power is always preferable, IMO.
I'm speaking strictly about the government here, obv.