You can go there any time, but if you want to dominantly secure a spot and get a good amount of exp, you should not have below 80 attack and strength.
You could always try one of the bandit tents though if the bar is too packed and you aren't doing as well there, its a bit more 'relaxed' than the bar as the bandits spawn slower and usually you only have one on you at a time, with the rest lined up to attack you afterwards.