Only Jagex knows their methods with any certainty.
Still, I assume their most reliable method is probably 'marking' known RWT'ers, and following a few of their trades before issuing bans to everyone that person interacted with.
Another method would likely be sorting through random samples of unusually large, unbalanced trades/stakes.