Saying Yes all the time does nothing good for you or the people you are saying yes to. Whether you are saying yes to doing work for someone or encouraging someone to continue doing something that you don’t actually feel “good” about, nobody is benefiting from it.
If you need to say no, be transparent, honest and upfront. It will serve you better in the future.
You aren’t a horrible person for saying “No”.