How do I teach a java program how to see?

I just finished a formal course on introductory Java and I want to learn more. So, I've decided to attempt to make botting programs that will play Runescape so that I can gain more experience with Java programming.

I used to make scripts for SCAR. I remember that SCAR provided me with already written code that could recognize images with a certain level distortion. I want to do this in Java from scratch using nothing more than the Java API.

So my question is actually in two parts:

1) How do I get Java to see the Runescape applet? (solved)

Or for that matter, how do I get Java to see what is being displayed on my monitor?

EDIT: I just found out about java.awt.Robot.createScreenCapture(Rectangle).

2) How do I get Java to recognize images that are slightly distorted?

For example, I give Java an image of the bank icon that appears in the minimap. However, if you notice, Jagex has made it so that icons on the minimap are all slightly different each time you play Runescape. The color of the icon may be tinted slightly. Also, depending on orientation of the map, the icon may be slightly mis-shaped.

 

Well, a pretty simple, but shitty way of doing that is capturing an image of where the bank icon is and comparing each pixel of that image to each pixel of the image you provide it. You would have to set up ranges (percentage) that the color could be off by for it to count as the same. Then, you have to count up how many pixels are the same and how many are different and use a simple percentage.

Alternatively, you could get images of all the different icons that appear on the mini map and run these through a program that finds pixels that are distinct in each image and saves them. Then, you could more quickly check if any image is a certain image by just checking a few of the pixels that make it unique.

 

May I ask why you're choosing to do it like this? SCAR was using very basic techniques of botting and to be honest they're quite outdated.

It would be more worth your time to research reflection and injection, these techniques along with image recognition (what you plan on using) are the driving force behind bots of today.

The reason for this development is exactly the point that you mentioned of how Runescape changes various elements of the graphics to reduce the use of this elementary technique. To create a program that would actual manage these changes would require you to further your studies, as there are entire fields dedicated to this type of work.

Just in case you didn't know, reflection makes use of 'looking down' at the program, identifying 'hooks' that essentially IDs of everything you have in Runescape. Injection 'looks up' and basically deals on a lower level of altering the actual game code on the client side, which is more efficient, but easily detectable.

P.S. You could use the Robot package but... I have a feeling you won't get very far with that approach.

Good luck in any case.

 

This is a good read in that even beginners should be able to understand the concepts and you might be able to find your way after that. Reading the whole thing would be recommended but only the parts starting from "How OCR works" really talk about comparing distorted/differing images.