Whether it's worth it depends on whether your group has a way to take advantage of the disadvantage via grappling, traps or speech checks or whatever.