Excel Grand Exchange Project

I'm currently working on my next big project which is an excel plugin that downloads/updates a local copy of the entire GE database.
I've already covered the basics, but now I'm looking for volunteers to help me program extra features (such as checking for price changes or updating only a set of items instead of the whole database), clean up the interface, discover bugs in different scenarios and other things I forgot to mention.

What is working

• Downloading the entire database, existing copy will be overwritten
• Downloading item icons, existing icons will be skipped (this is entirely optional, only for esthetic purpose)
• Apply table formatting that looks almost identical to the official GE database
• Ability to filter and/or sort by name, guide price, price change, free/member item status or item ID
• Formula's that convert abbreviated numbers (for example 100k becomes 100 000) so they can be used in excel calculations

What is not working

• Multithreading
• Simultaneous downloading of item icons
• Cancel button
• hide icons when their respective item row is hidden (at the moment they jump to a different row and overlap)
• keep icons on the row they belong to when overwriting the database

Planned features

• Check for price changes since the last database download
• Doubleclick items to open an item detail panel which includes graphs
• Ability to download only a given set of items
• High Alch lookup (I'll DEFINITELY need help with that)
• More will be added to this list later

The main focus at the moment is speed. Downloading price information alone is actually not that slow (seems to be anywhere between 3 to 10 minutes for a full database update), but icon files take painfully long (2-3 hours for all 3415 icons).

Screenshots (only some items have icons in the screenshots because it takes a good hour or two to download all 3415 of them)
Table with icons
Data sorted A-Z
Data sorted Z-A
Sorted expensive to cheap
Sorted cheap to expensive
Biggest price rises
Biggest price drops

Download a sample here (Static prices. You need the plugin to update it.)

Are you a (visual basic) programmer and interested in this project? PM me or leave a reply. Motivational replies are also highly appreciated.


EDIT:
The original plan was to rely completely on the official grand exchange only, but I find it very difficult/inefficient to programmatically gather the data from HTML pages. I'm now working on a complete rewrite in PHP and instead of downloading the data straight to excel, the database will be stored online in a MySQL database. The benefit is faster data access, more flexibility and there won't be so much stress on the grand exchange (I believe IRC scripts and other web crawlers are already wasting lots of jagex's server resources, we don't want to make things worse).

The drawback will be the time delay after every price change. It takes time to detect a change in price and to reflect those changes in the MySQL copy of the database. But I don't really see how this could be an issue.
Other than that, I'm a dummy when it comes to PHP and MySQL. Expect a couple more weeks before this is finished (although there will probably be a beta).

For those who still have no clue how this could be usefull, imagine this:
you desperately want 99 prayer, but your bank is worth only a sad 40M. The wise thing to do would be to find out the cheapest way to train prayer. How do you find out? First you make a list of all bones and ashes and find out the amount of XP you can get from each. Time spent: 10-20 minutes. Now you need to know their prices, so you start looking through the grand exchange database for current guide prices. Time spent: another 10-20 minutes. "BUT WAIT! The price changes every day! That's 10-20 minutes PER DAY" I hear you say. That's where this little thing here comes into play. With this, you can download all current prices with the touch of a button and be able to tell the cheapest way to train at that time, for any number of items in any number of excel sheets.

Now excel wouldn't cost the crapload of money that it does if you couldn't do other stuff with it, so you could probably make your own graphs, price history, skill planner, ... Just be creative (I myself have skill planners that show me how many items I have to make and how much it will cost me to reach a level goal). If you are a good merchanter (which I'm not), you could probably even cook up something that predict which items will rise or drop.

 

Can't help myself but it does sound like a useful project, hope to see it finished soon!

 

It looks great but I really have to ask....
Why?

 

Well obviously Excel provides a much more flexible interface than jagex provides with their database

You could for example make a list of all bones and ashes linked to the prices from that table (with vertical search), let the plugin run a price update and you would immediatley be able to tell the cheapest way to train prayer. Other than that, it's also a pretty great oversight because it has all tradeables on 1 page that you can sort and filter any way you like.

 

I kind of see this as more complicated

 

which part?

 

This is very useful for future bots. A merch bot could be made to retrieve these excel files and check which item's price has been rising the most in a certain number of days then merch with that item if it's worth it.

I could definitely see this project succeeding if it is "sold right" to the right people.

BTW, I downloaded the file and it had tons of bugs.
1) Once I clicked on one of the buttons to arrange it in a certain order, the names were written next to the wrong images.

2) It wrote on some of then '#Value!'.

Anyways, add me on MSN and I'll assist you in anything that I'm able to assist someone with.
MSN: [email protected]

 

I've seen something like this for herblore. Its really useful if its laid out in a sensible fashion. So best of luck!

 

Sounds like it could have been my project, where did you see it?

 

Sounds like a LOT of work. Good luck, I'd like to see this finished.

 

That's a lot work, good luck

 

Quick update to let you know that I'm (finally) making some good progress; I have some code that makes a 1:1 copy of the entire grand exchange in MySQL and/or updates the prices at scheduled intervals.

Does anyone else here have experience with databases? I'm still thinking about how I can store the latest price information without overwriting the old data (so we can get a nice price history going on). Any thoughts?

 

This will be amazing.

 

If you can pull this off and dumb it down for some people make it simple to use
then your in the clear