I hate to tell you dude, you're definitely not the only one who has felt this way, but there's always so much you can do/be able to do.
You've pretty much explained my dad except he was an alcoholic for most of my childhood and so the fighting was always definitely louder and more physical than not. Can't tell you how many times we've had to have the cops over, and even to this very day he's still unhappy with his life, constantly says he wants a way out, threatens to abandon, etc. And on top of that I have two other suicidal brothers that I've had to deal with for my whole life, one in particular I can't even talk to anymore so we've become estranged, but my other brother is moved out and still has issues but we're definitely cool now (Honestly it's moving away that probably helped him the most).
I remember being your age and younger and always wishing that he would leave but it never actually happened. I remember being probably 6-7, packing stuff up ready to leave when he was in jail one night and then we were back a day later.
I also remember being a younger teen and usually ending up fighting back, in the end it solves nothing, just makes you slightly accomplished for a minute or two because you've been waiting to do that for yeaaarrrrsss.
Now to this day all that we have is a bunch of bad memories (always a few moments of good, though), i'm pretty much estranged from my dad (as is everyone else unfortunately) and of course my brother, and sometimes I wish everything could have been different, but it's just the way the cards fall into play. He has been doing better, and to the best of my knowledge has quit drinking and all that stuff and definitely isn't as violent, but it took until I was 18 years old for that to happen dude, it just doesn't seem fair does it? A whole life time of shit if you ask me. And not even that though, but just because he quit drinking doesn't mean that he's not still verbally abusive a lot to my mom, and even when I still see it I have to be on edge because I know how quickly things can go sour.
Doesn't mean I hate him for it, but just like you instead of a father, I had a dad. I walked alone, I wasn't taught anything from him, we didn't go to ball games, didn't have father-son time, it just didn't happen.
I'll always be grateful because of the financial part. I mean we've always ate canned foods and had water from the tap but I can be appreciative now for what I had because I'm older and see from a little bit of a different perspective, but I'll never forget all those images that have burned into my mind, and I promise you I'm not going to be treating my children that way.
Tl;dr - My best advice for you is to stay strong and stick with your mother. Don't be afraid to stand up for what's right, even if it's you that gets knocked down to size. Never ever abandon your family just because one bad egg.