• Log in or sign up for an account here: https://www.sythe.org/login

Grand Exchange Data Scraping?

Status
Not open for further replies.
B

basterache3

Member
Joined
Jul 11, 2007
Sythe Gold
0
Messages
86
Has anyone else ever done one of these projects to scrape data from the GE?

I have an idea to make a BETTER version of the GE site than the one Jagex Presents that is more useful to people trying to track trends in the market so they can predict what to merchant successfully alot more often.

I wrote a program that can datascrape the GE when you enter and item id number. It will return all relevant information on that item.

This program is the basis of what I am trying to do. I have another program that can generate a complete list of all item ids that are currently in use that I wrote. In conjunction with a modified version of this program that I am posting here I hope to start making daily database dumps and collecting data, and then designing an interface people can use over the web to analyze it the way they want.

Would you consider such a site useful?

I know that various fansites track prices, but their updates are done manually and are sometimes several days old.

Here is a link to my scraper if you want to try it out:

http://getracker.co.cc/gedump.exe

Its very basic right now but should turn into something much more complex and useful.

EDIT: HERE IS THE CSV DUMP OF THE WHOLE GE.

Some Problems:

13 Items do not have the correct item names. They say ItemName Broken.

Some (Very Few) Items do not have the correct description. They Say ItemDescription Broken.

http://getracker.co.cc/ItemCSVData_6_6_2010.csv
 
you have to download sumthing from the link called "gedump.exe"
 
could you upload source code? so we see its legit
 
Describe how you 'datascrap' from here the GE on Jagex website?

Very sketchy program...100% caution
 
Whats so sketchy about datascraping?

http://en.wikipedia.org/wiki/Data_scraping

And no you cannot have the source at this time, except I will tell you that it uses inet controlls to load the complete source of a webpage into a string, then parses the string using methods I have written.

Don't use it if you don't want to. I plan on having the dump of the whole GE price database posted on that site in plaintext for everyone in a day or two at the latest.

I guess you can wait until then if you are worried, as you will see the output it produces.
 
Lawrence
Pokémon Trainer
Doge
Extreme Homosex
Two Factor Authentication User
UWotM8?
I tried it, it works okay but you could work on it more ;)
 
You should try making this into a website, so that people don't have to download anything.

It would be 10x more popular if you did so.
 
I plan on making it into a website. Iknow it is possible to use PHP or ASP to scrape data and store it to a database but I didn't know how while I knew how to do it in Visual Basic.

I agree this is pretty useless right now, the exe that is up there. But I am working on a new version that correctly does the dumping and item finding automatically.

scraper1.gif

scraper2.gif


2,Cannonball,Ammo for the Dwarf Cannon.,313,330,347,-7.0%,-14.9%,-0.9%
6,Cannon base,The cannon is built on this.,120800,127100,133500,+7.6%,+6.4%,-1.4%
8,Cannon stand,The mounting for the multicannon.,122800,129300,135700,+8.5%,+12.5%,-1.4%
10,Cannon barrels,The barrels of the multicannon.,122300,128700,135200,+8.6%,+9.3%,-5.4%
12,Cannon furnace,This powers the multicannon.,122500,129000,135400,+6.2%,+11.5%,-1.6%
36,Candle,A candle.,172,181,190,+5.8%,+12.4%,+0.5%
39,Bronze arrowtips,I can make some arrows with these.,9,10,10,+0%,+0%,+0%
40,Iron arrowtips,I can make some arrows with these.,17,18,19,-5.2%,-5.2%,+5.8%
41,Steel arrowtips,I can make some arrows with these.,34,36,36,+0%,+0%,+2.8%
42,Mithril arrowtips,I can make some arrows with these.,64,67,68,+4.6%,+1.5%,+17.5%
43,Adamant arrowtips,I can make some arrows with these.,152,160,160,+0%,+0%,+1.2%
44,Rune arrowtips,I can make some arrows with these.,437,460,460,+0%,+0%,+0%
45,Opal bolt tips,Opal Bolts,1,1,2,+0%,+0%,+0%
46,Pearl bolt tips,Pearl Bolt Tips,23,24,25,+41.1%,+14.2%,+242.8%
47,Barb bolttips,I can make bolts with these.,38,40,42,+8.1%,+14.2%,+0%
48,Longbow (u),I need to find a string for this.,12,12,13,+0%,+0%,+0%
50,Shortbow (u),I need to find a string for this.,4,4,5,+0%,+0%,-20.0%
52,Arrow shaft,A wooden arrow shaft.,4,5,6,+0%,+66.6%,+25.0%
53,Headless arrow,A wooden arrow shaft with flights attached.,20,21,22,-27.5%,+10.5%,-19.2%
54,Oak shortbow (u),An unstrung oak shortbow; I need a bowstring for this.,10,10,11,+0%,+0%,+0%
56,Oak longbow (u),An unstrung oak longbow; I need a bowstring for this.,16,16,17,+0%,+0%,+0%
58,Willow longbow (u),An unstrung willow longbow; I need a bowstring for this.,32,32,34,+0%,+0%,+0%
60,Willow shortbow (u),An unstrung willow shortbow; I need a bowstring for this.,20,20,21,+0%,+0%,+0%
62,Maple longbow (u),An unstrung maple longbow; I need a bowstring for this.,88,93,98,+12.0%,-7.0%,+27.3%
64,Maple shortbow (u),An unstrung maple shortbow; I need a bowstring for this.,47,50,53,+25.0%,+25.0%,+25.0%
66,Yew longbow (u),An unstrung yew longbow; I need a bowstring for this.,408,430,452,+0.2%,+0.2%,-2.0%
68,Yew shortbow (u),An unstrung yew shortbow; I need a bowstring for this.,164,173,182,-4.9%,-14.3%,+16.1%
70,Magic longbow (u),An unstrung magic longbow; I need a bowstring for this.,1175,1237,1299,-1.4%,-3.8%,-1.9%
72,Magic shortbow (u),An unstrung magic shortbow; I need a bowstring for this.,708,745,782,-2.8%,-11.3%,-12.0%
91,Guam potion (unf),I need another ingredient to finish this guam potion.,706,743,780,-16.9%,-7.2%,-0.9%
93,Marrentill potion (unf),I need another ingredient to finish this marrentill potion.,246,259,272,+96.2%,+87.6%,+58.8%
95,Tarromin potion (unf),I need another ingredient to finish this tarromin potion.,1012,1065,1118,+4.6%,-11.1%,+22.1%
97,Harralander potion (unf),I need another ingredient to finish this harralander potion.,997,1050,1103,-3.1%,-33.5%,-24.6%
99,Ranarr potion (unf),I need another ingredient to finish this ranarr potion.,8275,8711,9147,-1.3%,-28.5%,-23.8%
 
Are you being serious about saying there is something wrong with it? Are you joking, or just stupid?

1. Vmware is an actual program that is used to emulate operating systems. This program has nothing to do with VMware at all.

2. I had no idea what anti-anubis is, but by googling it the only thing I found under that name was a snippet of code on how to kill an anubis process.

3. sandbox - I doubt it could be infected with a sandbox. A sandbox is an isolasted test enviroment, which could be run under vmware.

These things are not infections.

Please link me to the scanner you used or explain how you scanned it.
 
I'll wait for someone to check it out
(:
 
Once this thing is legit, or made into a website, I'll try it out.
 
You guys dont have to check it, the program up doesn't do much.

It is only the top part of the the program and can only look up single items at a time.

I am currently working on running it through the itemids 1-20000. I keep finding little errors due to Jagex's stupid formatting of their GE data. Ive had to write several methods to parse out stupid abbreviations and commas in number fields.

I plan to post a complete csv file once its done full of the info gathered. I am pretty sure I can't infect a csv file with a program used to emulate an operating system, lol.
 
basterache3 said:
Are you being serious about saying there is something wrong with it? Are you joking, or just stupid?

1. Vmware is an actual program that is used to emulate operating systems. This program has nothing to do with VMware at all.

2. I had no idea what anti-anubis is, but by googling it the only thing I found under that name was a snippet of code on how to kill an anubis process.

3. sandbox - I doubt it could be infected with a sandbox. A sandbox is an isolasted test enviroment, which could be run under vmware.

These things are not infections.

Please link me to the scanner you used or explain how you scanned it.
Click to expand...

Lol you're so dumb.

1. Vmware is a program that let you run programs safely. So he mean that it disables this.

2. Anubis is also a safety program. It disable's anubis.

3. And anti-sandbox means that it disables sandboxie. yes your right its a isolated test enviroment but I disable it when u open it with that program. So i could easily be infected with a logger or RAT. I suggest not to downlaod this.
 
Status
Not open for further replies.
Runescape Gamblingrunescape gambling
Back
Top