PixxelJunk
Member
- Joined
- Jun 4, 2011
- Sythe Gold
- 0
- Messages
- 57
I'm currently working on my next big project which is an excel plugin that downloads/updates a local copy of the entire GE database.
I've already covered the basics, but now I'm looking for volunteers to help me program extra features (such as checking for price changes or updating only a set of items instead of the whole database), clean up the interface, discover bugs in different scenarios and other things I forgot to mention.
What is working
What is not working
Planned features
The main focus at the moment is speed. Downloading price information alone is actually not that slow (seems to be anywhere between 3 to 10 minutes for a full database update), but icon files take painfully long (2-3 hours for all 3415 icons).
Screenshots (only some items have icons in the screenshots because it takes a good hour or two to download all 3415 of them)
Table with icons
Data sorted A-Z
Data sorted Z-A
Sorted expensive to cheap
Sorted cheap to expensive
Biggest price rises
Biggest price drops
Download a sample here (Static prices. You need the plugin to update it.)
Are you a (visual basic) programmer and interested in this project? PM me or leave a reply. Motivational replies are also highly appreciated.
EDIT:
The original plan was to rely completely on the official grand exchange only, but I find it very difficult/inefficient to programmatically gather the data from HTML pages. I'm now working on a complete rewrite in PHP and instead of downloading the data straight to excel, the database will be stored online in a MySQL database. The benefit is faster data access, more flexibility and there won't be so much stress on the grand exchange (I believe IRC scripts and other web crawlers are already wasting lots of jagex's server resources, we don't want to make things worse).
The drawback will be the time delay after every price change. It takes time to detect a change in price and to reflect those changes in the MySQL copy of the database. But I don't really see how this could be an issue.
Other than that, I'm a dummy when it comes to PHP and MySQL. Expect a couple more weeks before this is finished (although there will probably be a beta).
For those who still have no clue how this could be usefull, imagine this:
you desperately want 99 prayer, but your bank is worth only a sad 40M. The wise thing to do would be to find out the cheapest way to train prayer. How do you find out? First you make a list of all bones and ashes and find out the amount of XP you can get from each. Time spent: 10-20 minutes. Now you need to know their prices, so you start looking through the grand exchange database for current guide prices. Time spent: another 10-20 minutes. "BUT WAIT! The price changes every day! That's 10-20 minutes PER DAY" I hear you say. That's where this little thing here comes into play. With this, you can download all current prices with the touch of a button and be able to tell the cheapest way to train at that time, for any number of items in any number of excel sheets.
Now excel wouldn't cost the crapload of money that it does if you couldn't do other stuff with it, so you could probably make your own graphs, price history, skill planner, ... Just be creative (I myself have skill planners that show me how many items I have to make and how much it will cost me to reach a level goal). If you are a good merchanter (which I'm not), you could probably even cook up something that predict which items will rise or drop.
I've already covered the basics, but now I'm looking for volunteers to help me program extra features (such as checking for price changes or updating only a set of items instead of the whole database), clean up the interface, discover bugs in different scenarios and other things I forgot to mention.
What is working
- Downloading the entire database, existing copy will be overwritten
- Downloading item icons, existing icons will be skipped (this is entirely optional, only for esthetic purpose)
- Apply table formatting that looks almost identical to the official GE database
- Ability to filter and/or sort by name, guide price, price change, free/member item status or item ID
- Formula's that convert abbreviated numbers (for example 100k becomes 100 000) so they can be used in excel calculations
What is not working
- Multithreading
- Simultaneous downloading of item icons
- Cancel button
- hide icons when their respective item row is hidden (at the moment they jump to a different row and overlap)
- keep icons on the row they belong to when overwriting the database
Planned features
- Check for price changes since the last database download
- Doubleclick items to open an item detail panel which includes graphs
- Ability to download only a given set of items
- High Alch lookup (I'll DEFINITELY need help with that)
- More will be added to this list later
The main focus at the moment is speed. Downloading price information alone is actually not that slow (seems to be anywhere between 3 to 10 minutes for a full database update), but icon files take painfully long (2-3 hours for all 3415 icons).
Screenshots (only some items have icons in the screenshots because it takes a good hour or two to download all 3415 of them)
Table with icons
Data sorted A-Z
Data sorted Z-A
Sorted expensive to cheap
Sorted cheap to expensive
Biggest price rises
Biggest price drops
Download a sample here (Static prices. You need the plugin to update it.)
Are you a (visual basic) programmer and interested in this project? PM me or leave a reply. Motivational replies are also highly appreciated.
EDIT:
The original plan was to rely completely on the official grand exchange only, but I find it very difficult/inefficient to programmatically gather the data from HTML pages. I'm now working on a complete rewrite in PHP and instead of downloading the data straight to excel, the database will be stored online in a MySQL database. The benefit is faster data access, more flexibility and there won't be so much stress on the grand exchange (I believe IRC scripts and other web crawlers are already wasting lots of jagex's server resources, we don't want to make things worse).
The drawback will be the time delay after every price change. It takes time to detect a change in price and to reflect those changes in the MySQL copy of the database. But I don't really see how this could be an issue.
Other than that, I'm a dummy when it comes to PHP and MySQL. Expect a couple more weeks before this is finished (although there will probably be a beta).
For those who still have no clue how this could be usefull, imagine this:
you desperately want 99 prayer, but your bank is worth only a sad 40M. The wise thing to do would be to find out the cheapest way to train prayer. How do you find out? First you make a list of all bones and ashes and find out the amount of XP you can get from each. Time spent: 10-20 minutes. Now you need to know their prices, so you start looking through the grand exchange database for current guide prices. Time spent: another 10-20 minutes. "BUT WAIT! The price changes every day! That's 10-20 minutes PER DAY" I hear you say. That's where this little thing here comes into play. With this, you can download all current prices with the touch of a button and be able to tell the cheapest way to train at that time, for any number of items in any number of excel sheets.
Now excel wouldn't cost the crapload of money that it does if you couldn't do other stuff with it, so you could probably make your own graphs, price history, skill planner, ... Just be creative (I myself have skill planners that show me how many items I have to make and how much it will cost me to reach a level goal). If you are a good merchanter (which I'm not), you could probably even cook up something that predict which items will rise or drop.
