Think about it this way. The Empire is a militaristic state where everyone receives some level of combat training. Noah is so bad at physical stuff he got an actual medical exemption from gym class for both his and others safety.
And his first arc was basically people forcing him to train his magic.