I think bullet design would also have a significant effect since hollowpoints are far more damaging than ball rounds to unarmoured targets. Frangible rounds particularly, such as the glaser safety slugs, appear to make mulch out of soft targets at least according to the videos on youtube.
All other things being equal though, I agree that the larger slower round would be more likely to knock people over.
When you have 19mm balls being fired at you, it's going to hurt.
I was reading an account of the Pennisular War where it was sometimes reported that the musket balls were amputating hands and other appendages if the ball hit a lucky (or unlucky, depending on your point of view) spot.