The bot nuke worked for bots that either change the client or directly grab data from the client without using the screen. Nearly any change that they do to the client will break all of these bots until the makers update. The bot nuke was this just being taken to an extreme in the hopes that it would take too much effort to update the bots for the new client. Obviously that did not work.
Bot detection works in many many many ways. The best public bot that has ever been made is more than likely the robotindisguise mage training arena bot circa 2007. It was good enough that JMods were spending hours in the arena observing players and trying to catch them. If there had been a bot detected on your IP previously they banned your mage training arena account. With technical solutions alone it was basically impossible to detect.
Those technical solutions basically involve collecting tons of data about the inputs that you are giving the game. If you setup an auto clicker that clicks the exact same spot in the exact same intervals, it is easy to tell that you are botting. Bot makers try to introduce randomness into their bots but generally that amount is severely limited in scope. For instance the bot may click a random spot on each inventory slot to drop the entire inventory. However, if you collect all of the clicks over a few days you will see that each possible location to be clicked in each spot are clicked at roughly the same rate. A human would be unable to do that and more likely than not you click in a similar spot in each inventory slot, likely in the middle but maybe closer to a corner or side.
Jagex is also able to detect when your client is modified which gives them a strong indication that they should look into your account further. That is when they would turn on the more intensive data collection on your account. Then it comes down to statistics.
