I try to be a good person but have ended becoming a doormat for troubled people in the past.
This seems like a good balance to me. It's the level toward which I try. There's plenty of mental illness in my clan, too - on both parents' sides. I have my own issues, too, and I try to keep myself healthy. Sometimes that requires keeping away from certain relatives. This does not equal hating them or even not caring about their well-being.
I'm very sorry for the recent loss of your sister, OP. If you had no compassion for her I don't believe you would've started this thread.
Sometimes it's just too difficult for a family member to care for the ill one. That's why public services have been created. That's why our tax dollars are well-spent in supporting such services. If the seriously mentally ill persons could have a choice, they would not choose to live with these illnesses. If family members had a choice they would not inflict these illnesses on the sufferers. We as a society have an obligation to care for them in a humane way. This does not mean self-sacrifice of anyone so unfortunate as to have such illness in the family.
Yes, it's political. Yes, there are public needs for which we all must pitch in and pay taxes. That's what it means to live in a civilized society. No one's ill family member should be cast out into the streets with no supports, despite the family's inability to shoulder the burden of care alone.
I used to go into the state hospital to observe and assist with the discharge planning for individuals going back into the community. My agency worked with our county government in order to create proper services for people. I will never forget the elderly couple who believed they were caring for their mentally ill son in the best possible way. They were wealthy people but couldn't get around much themselves anymore. They refused public services in the larger community for their son, including a residential placement, job skills assistance, etc., because they figured they could afford it and it was their responsibility to care for him. The poor guy, in his fifties and miserable, kept cycling back into the state hospital and then back again to the parents' fine home in the suburbs. All he had to do all day was sit around the house, lounge by the pool, watch tv, and the like. The psychiatrist tried to explain to the parents that a placement in the city to a less affluent area would give the guy a much "richer" existence and hope of recovery to some degree, with greater social integration. The parents were too old to drive him for vocational/social training daily so he just didn't get to go. There was no public transportation service to their area. He just festered in their home, becoming more and more seriously ill until he would bounce back into the hospital.
I wish you well, OP. I don't believe you've done anything wrong by your sister.