By saying no to your toddler, then ultimately giving in and letting him have whatever it is that he wants, you’re teaching him that "no" means "you’ll get your way eventually if you keep pushing." If you really want your toddler to listen to you when you say “no,” you have to stick to it. Consistency is crucial, or he'll learn that challenging you is an effective strategy — not a road you want to go down.
The expression and tone you use will also make a big impact on whether he thinks you don’t care, as you put it. The reason why you set limits for your toddler is to keep him and your family safe, happy, and healthy. You know that — so communicate limits in a warm way that conveys that.