So I had a question about learning how someone would do this kind of thing for themselves (that is, take Tribot and make this style of crack for it). Thought I would post it here, in case anyone else was wondering about it.
-----
Learning this kind of stuff is a little weird, since normally you don't just need like... "learn SQLi" or anything, but rather you need a small amount of knowledge of all web stuff. So like, for this, knowing basic web security isn't really enough, since I needed to figure out how to find SSL certificates that would authenticate (which is largely a Java specific thing), then the knowledge for capturing and decoding encrypted requests, etc etc.
So basically, it's hard to give a single resource, but working with lots of different systems for a lot of times gives you a lot of experience, which is exactly what you need. You don't need to be particularly awesome at everything, just being able to kind of thing about problems and recognize things you could do to help, almost.
However, I can recommend a few places to learn the kind of thinking you end up using for anything like this (and more generally, it applies to every kind of programming and security). Places like
http://io.smashthestack.org/ are amazing for learning how to think (and also if you're interested in security on the client side, no web stuff). Stripe,
https://stripe-ctf.com/, ran the most amazing web CTF I've ever seen (the 2.0) one, which was awesome. There are lots of other places that run kinds of CTFs which are a great way to learn both specific skills, and the thinking you need to approach problems like this. Just pick some that seem fun.