Chihuahua - Harness or Collar?

I have a 5 lb. Chihuahua who I have used a harness for for probably the last year or so. I left the harness on at all times because he made it so hard to get on him, that he would sometimes not make it outside to go potty and instead go on me while I was putting on the harness. I started noticing that his fur beneath the harness was damaged so I went back to using a collar... but the collar only chokes him. It seems neither method works well for me. I know his trachea is very delicate and I don't want to injure him by using a collar. But I also don't want him to be uncomfortable in his harness. What should I do? Can you buy softer harnesses that don't damage the fur and are more comfortable? If so, where?