Well you could get rough estimation, look at how many people on every day at a certain peak hour time. say 10pm. Then note that down, lets say average for Sun-Wensday is 100,000 people. Then lets say bot updates (major) update on thursday at same time it goes down to 85,000. Do this for a month or two when you yourself notice not many players are on due to bot updates. Then try to get an average. So week 1: 20K player drop, week 2:10K player drop...etc
Look at average player drop @ that time in a non-update week to be 1-3K varible. Take that away from the average and you could get an ok estimation of botters on Runescape.