Unfortunately many people are ignorant of issues around disability, as they see the disability, but not the person.
They believe they are caring, but have no understanding of how care should be offered.
No one should assume that care is required, without contacting the person concerned. Just because a person has a disability does not mean their feelings and other senses are disabled also.
Everybody, no matter who they are wish to retain their own independence and therefore everyone should be respectful of others.
Before you act, think and consider how to proceed for no one should touch another without the expressed permission of that persons. This is also extended to any aids or adaptions that the person may have.
It is assault to make contact with a person without their permission and the act of touching a persons wheelchair or any other items of theirs is also assault, which is deemed a criminal act or should be as it is abuse
Do not ignore people, but treat them as you, yourself would wish to be treated, for you do not know the person like they know themselves.
The act of touching is further compounded when they are telling you not to do what you have done without their expressed permission.
They are not in the wrong, you are.
Ignorance is never any excuse.