This bothered me for some time too, Michael! As a martial artist with dozens of matches under my name myself (of course none of them to the death), I think what distinguishes Musashi from a cold-blooded psychopath was that those duels were mutually agreed upon.
He sought out fights, yes, but all of his opponents were willing participants. In fact, many of them challenged him themselves. A pretty brutal way to live, but that’s the way of the warrior a couple of hundred years ago I suppose.