I admit that I was surprised to learn that we aren’t allowed to say “half-witted”. But equally, on what authority are you to say that “we” say it “all the time”? I can’t remember the last time I heard anyone say it.
Some people’s conversations must surely lack the beauty of Shakespeare if so much of their speech is comprised of “half-witted”!

Very well put. My Irish mother would many times, when losing her patience with one of her five children, say “it would take two of you to make a halfwit”. We thought she was just witty and pretty on point!