Threading / Concurrency

I have a java program I wrote to scrape a certain set of webpages.

For example, I have a list of numerical IDs that need to be scraped. The base url is all the same.

for example here are very simplified methods

Code:

public void prgm(){
     ArrayList<String> scrapedData = new ArrayList<String>();
     for(int i=0; i=100; i++){
          scrapedData.add(scrapePage(i));
     }
}
public String scrapePage(int id){
     //// do work here
     ///return a string
     return "abc";
}

Right now My program will scrape one page, finish, then scrape the next page, finish, and on and on

It only will scrape one page at once.

I want to change it so I can scrape 5 or 10 pages at once

any suggestions?

 

Are you asking how to do it or what the best way of going about it would be?

 

I want to make 10 concurrent calls to the method scrapePage and need to know how to do it

 

You'd want to use a thread.

Basically, this is how you would go about it:

Start by creating a class that extends thread

Code:

public class Scraper extends Thread{
}

If you're working in an IDE it will complain about invoking methods, this is because a thread needs a run method (the thing that you're actually threading) inside of your class that extends thread.

Code:

public void run(){
}

So basically put the code for scrapping inside of the run, then as you pop the "Scapers" off of the array list you just say scraper.start();

Once you start them the run method will execute, hope this helps.