You took a willing -4 to not hit your allies, I considered that enough caution to negate fumble effects.

I often will use fumble effects when it flows with what is happening.