All Runescape User Highscores Downloader!

[SuF]

I have made a program that should download and dump the contents of every Runescape username's lite highscores into a single text file which can be parsed later. I am currently testing it, and hoping Jagex will not IP block me. I will post the code if all goes well.

Ok, Here is my fail code. I may try and make it less fail, but it seems that there are too many damned usernames to make this work well... Lol...

package suf;

import java.net.*;
import java.io.*;
import java.util.*;

public class AllUserDownloader
{
	static Scanner s = new Scanner(System.in);
	public static String[] letters =
	{"", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "1", "2", "3", "4",
			"5", "6", "7", "8", "9", "0", " " **;

	public static int[] hold =
	{ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 **;

	public static void main(String[] args)
	{
		System.out.print("Welcome to SuF's All User Stats Downloader! \nWhen you are ready to start the process type 'Start':");

		while (true)
		{
			if (s.next().equalsIgnoreCase("Start"))
			{
				System.out.print("Starting! \nThis may take awhile, so please leave the program open untill it says it is done!");
				break;
			**
			else
			{
				System.out.print("Type 'Start' when you are ready to begin the process.");
			**
		**
		while (true)
		{
			hold[0]++;
			if (hold[0] == 37)
			{
				hold[0] = 1;
				hold[1]++;
				
				if (hold[1] == 37)
				{
					hold[1] = 0;
					hold[2]++;
					if (hold[2] == 37)
					{
						hold[2] = 0;
						hold[3]++;
						if (hold[3] == 37)
						{
							hold[3] = 0;
							hold[4]++;
							if (hold[4] == 37)
							{
								hold[4] = 0;
								hold[5]++;
								if (hold[5] == 37)
								{
									hold[5] = 0;
									hold[6]++;
									if (hold[6] == 37)
									{
										hold[6] = 0;
										hold[7]++;
										if (hold[7] == 37)
										{
											hold[7] = 0;
											hold[8]++;
											if (hold[8] == 37)
											{
												hold[8] = 0;
												hold[9]++;
												if (hold[9] == 37)
												{
													hold[9] = 0;
													hold[10]++;
													if (hold[10] == 37)
													{
														hold[10] = 0;
														hold[11]++;
														if (hold[11] == 37)
														{
															break;
														**
													**
												**
											**
										**
									**
								**
							**

						**
					**
				**
			**
			String player = letters[hold[0]] + letters[hold[1]] + letters[hold[2]] + letters[hold[3]] + letters[hold[4]] + letters[hold[5]] + letters[hold[6]]
					+ letters[hold[7]] + letters[hold[8]] + letters[hold[9]] + letters[hold[10]] + letters[hold[11]];
			try
			{
				write(player,connect(new URL("http://hiscore.runescape.com/index_lite.ws?player=" + player)));
			**
			catch (MalformedURLException e)
			{
			**
		**
		System.out.println("DONE!");
	**

	public static String connect(URL site)
	{
		String data = "";
		while (true)
		{
			try
			{
				String html;
				BufferedReader in = new BufferedReader(new InputStreamReader(site.openStream()));
				
				while ((html = in.readLine()) != null)
				{
					if (data.equals(""))
					{
						data = html;
					**
					else
					{
						data = data + "," + html;
					**
				**
				return (data);
			**
			catch (MalformedURLException exc)
			{
				return "-2";
			**
			catch (IOException exc)
			{
				return "-2";
			**
		**
	**
	public static void write(String name, String stats)
	{
		try
		{
			if(stats.equalsIgnoreCase("-2"))
			{
			**
			else
			{
			PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("Dump.txt", true)));
			out.print(name + "," + stats + ",");
			out.close();
			**
		**
		catch (IOException e)
		{
		**
	**
**

EDIT: I am getting scared I will use up my entire hard drive space doing this, but o well... If someone feels like calculating it for me, go ahead as I am to lazy. I am using 26 letters, 10 numbers and space.

EDIT 2: Ok, I have fixxed a few minor errors, and have decided not to write the info to the file if it is not on the RS highscores. So I am starting again.

EDIT 3: Fixxed a few errors that fixxing the other errors caused. Time to test again...

EDIT 4: This is going to take like 2 weeks to do. Lol... Thats just a bit too much fail....

 

[Supah Fly]

ok so heres what you do

make it add +21 from wherever you are starting.

also

int i;
int z;

for (i = 0; i => whateverthemaxnumberofstats; i++) {
// i forget how to download files via java
for (z = 0; z => 200000; z+21) {
downloadfile("http://hiscore.runescape.com/overall.ws?rank=" & z &"&table=2&scroll=true&category_type=" & i);
println(userparse);
}
}

btw my java sucks

		while (ss_between($bbzs, '<td class="alL"><a href="hiscorepersonal.ws?user1=', '">') != '') {
			$vbtz[$z] = ss_between($bbzs, '<td class="alL"><a href="hiscorepersonal.ws?user1=', '">');
			$bbzs = str_replace('<td class="alL"><a href="hiscorepersonal.ws?user1='.$vbtz[$z].'">', '', $bbzs);
			echo($vbtz[$z].'
');
			$z++;
		}

and the php tags fail -.- l2highlight_string()
Maybe you can convert that to java? It parses it out perfectly, you just need to put it in a loop.

 

[SuF]

Wtf are you blathering about?

 

[Supah Fly]

yours looks alot harder then it really is. >.>

first loop from first range until it reaches the end
second loop in first loop until there is nothing left to parse

load the url and parse the data
remove the parsed info so it isnt picked up again

end second
next of first loop

like i did in php.

 

[SuF]

Yours is fail. You will be IP blocked, and have all your RS guys banned if you do it that way. I used the lite highscores and looped through it with all the names possible in RS. You did not include any download function, which makes my code much longer, and you did not include any input to ask when to start ect. My huge amount of if statements could be done another way, but that was easy and I was too lazy to think of another way to do it.

The majority of my code is my if statement shit, and the other major part is the downloading code which you would need too.

Plus, what I am doing is faster to load than what you are doing, and I am writing them to a file. My code is fail, but yours is Ãœber fail.

 

[Supah Fly]

Yours is fail. You will be IP blocked, and have all your RS guys banned if you do it that way. I used the lite highscores and looped through it with all the names possible in RS. You did not include any download function, which makes my code much longer, and you did not include any input to ask when to start ect. My huge amount of if statements could be done another way, but that was easy and I was too lazy to think of another way to do it.

The majority of my code is my if statement shit, and the other major part is the downloading code which you would need too.

Plus, what I am doing is faster to load than what you are doing, and I am writing them to a file. My code is fail, but yours is Ãœber fail.

i dont run php on my computer. its called vps >.>

and i didnt even include all my code

i also think crawling the highscore pages via a proxy or something (if youre worried about getting banned) is much better then what you are doing

After you download the highscore page, it parses out the usernames, removes them from the html (so it doesnt go through them again),
adds +21 to the last page's ranks, does it again, up to 2,000,000

then after 2,000,000 it goes on to the next skills which would be (Overall = 0, attack = 1, etc.) so you just keep looping until you get to the end.

Looks like yours is just guessing each user hoping there is an account where this is certain and would take less time.

ive tried to do this in several languages, all ending up the same, they take a loooong time.

 

[SuF]

I am downloading every single possible runescapes names highscores. I am using the lite page since it is faster loading, and you won't get banned for it. It also shows any skills above 30, even if it is not on the highscores. Its faster to parse later on, and does not require to be parsed to remove html which makes it even faster. Plus, you do not need to go through the trouble of parsing the info again to put all the usernames together with all 21 skills.

I am not guessing usernames, I am going through ever possible username. There are so many it takes forever, but if i used diffrent threads it would be faster. If I divided up the work and had like 20 friends help, I could have a complete list in less and a week...

I was also thinking of using a proxy, but I don't need to my way.

Unless you use like 1000 proxys, it will take your way a lot more time than me since your ip will be blocked quite quickly and your results will be wrong.

 

[Supah Fly]

ever think jagex will look at your IP and see that you have downloaded over 9000000 lite highscore pages without slowing down for a second.

instead of loading every possible username (which could take a very very very long time) just do it this way and itll save you from going to over 9000000 pages.

max of 1,000,000 or so pages compared to that.

26 letters + numbers

my friend made a md5 hash cracker, checking all possible combinations up to 5 characters takes more then 5 hours.

 

[Swan]

Woah calm down guys, I don't want to have to do anything to this thread.

 

[SuF]

Jagex tells you to use the lite page if you are automatically collecting huge amounts of information. They do not IP block you if you do, since the page is so small. They do IP block you after about 5 minutes of constant loading of the normal highscores, meaning you can not do it your way unless you have a bunch of proxies lined up. I know mine takes forever, but if I make it so that the usernames can be split up somehow, I could have friends help and I could have a complete list pretty quickly.

The lite page loads faster than the normal highscores page too, since its just a few numbers and such. Since you have to use proxies, that would slow down loading times much more. Yours also has to parse a lot more... I made mine so I did not have to use proxies, or parse at all before it wrote the data to the file. Well, it parses out usernames that do not have any skills, but thats really easy.

I could make a few threads to do the same thing, and I would think that would speed things up quite a bit, but I don't feel like working on this right now... :/

And I would have used the way your doing it, except I did not feel like fucking around with proxies so I did not bother with it...

 

[lord.gilles]

Well even if you have the fastest computer in the world

i bet there are over 10,000,000,000 runescape accounts (all banned, account that are now, all marco accounts).

If you may remember, people used to have script to create runescape accounts like hee000 => hee999 which is 1000 accounts already.

So gl trying to find them all Would be funny if you could list them all, but your Notepad file would be like 70 GIG to download/open

 

[SuF]

12 spaces and 37 diffrent letters i think. use a calculator and figure it out...


and it does not matter how many accounts with mine, i am going through them all... then we can find out how many people have 30+ in a skill or something... would be nice... but it would be faster if i had like 20 or 30 people having their comps run it for a diffrent section of the usernames...

 

[Supah Fly]

your just loading a shitload more pages then you would need to

which would take longer
which would slow down your internet alot if it was on different threads, and slow down your computer a bit.

time is the main thing really. you could do it this way, but then you are loading a bunch of pages that MAY be highscore pages, when if you parse out a bunch of html pages, you are certain and you are done quicker.

 

[SuF]

your just loading a shitload more pages then you would need to

which would take longer
which would slow down your internet alot if it was on different threads, and slow down your computer a bit.

time is the main thing really. you could do it this way, but then you are loading a bunch of pages that MAY be highscore pages, when if you parse out a bunch of html pages, you are certain and you are done quicker.

Downloading less than 1kb takes like .01 second. You seem to be avoiding the point that you NEED to use proxies for your way to work, so that slows down your internet speed, and since you need to parse more, your takes more CPU load, which is not a lot but it is still more. I want a complete list of all users that have skills above 30, which your way would not do.

Each of our ways has pros and cons, and I think my ways pros outweigh its cons.

 

[d 7 x]

I admire you for trying. It's too much for me. :]

 

[SuF]

I admire you for trying. It's too much for me. :]

Lol, making it very good is too much for me too.


EDIT: 1300 Posts... WOOT