I feel your pain. It hurts even more when an insult comes from a family member than it does from a bully at school, because it is more personal. I bet you feel at times completely helpless and frustrated so much it hurts your gut. My advice to you is to start reacting. If you have tried talking with your mother and she does not do anything, then it is time for you to take matters in your own hands.
1st-Next time it happens, look your brother square in the eye and tell him you have had enough. That his insults are draining you of your energy, and that you do not have the time or the patience to put up with his sadistic qualities. Yes, when someone, even your brother, is getting pleasure out of some one else's pain that is considered sadistic. OR you could suggest you two do something together, like go to the movies, on a bike ride, to the mall-there are so many things you could do to bond.
2nd-If talking to him doesn't work, go back to your mom and say to her- I mean it, I can't and won't take it anymore. You have to say something to him, he is cutting me down, insulting me and it's not right. And honestly, she does have to say something. She has to step in, and do her job as parent, because one of her children is being hurt. Thats her job!
3rd-If neither of the two work, you are going to have to make some noise!! For God's sake, its two boys we are talking about. Plus, you are the older one. Kick his butt- in gentle, yet firm way. No hospital visits! But seriously, pull his hair, slap him, throw him on the ground-whatever it takes. The problem is, you are not a threat to him.
Fact of the matter, you do not deserve to have the dignity verbally beaten out of you by some snot-nose, little eighth or nineth grade kid. When he does so much that it is hurting your self esteem, that is considered abuse. If he is really putting that much effort into making you miserable, he probably doesnt have any extra-curricular going on, such as sports, art, music etc. He needs a distraction, because his focus, his intent, is on hurting you.
Honestly, I hope this works.