[Tutorial] PHP Scraping 101

[Viral_]

I am going to show all of you the basics on how to scrape with PHP. For the demonstration I am going to show you how to scrape stats from the runescape website.

Now the basics of this is all going to be through CURL. If someone would like a tutorial on CURL I would be happy to give you the run down, but for this demonstration we are going to use a library

GitHub - FriendsOfPHP/Goutte: Goutte, a simple PHP Web Scraper

I have been very fond of this library. I have built my own but for an example of usability I will be using this library.

Requirements

• Webserver with LAMP installed
• Composer

If you do not have a webserver. You can go to this thread and follow my tutorial on how to set one up.
[Tutorial] [Free] LAMP web server

If you need help setting up composer go to this link
Composer


Tutorial

Run

composer require fabpot/goutte

In the folder where you are putting your php file to scrape.

Start your php file

<?PHP


?>

Now depending on if you are pulling a platform in or if you are starting from scratch. You always want some sort of debugging to see if there is any errors.

<?PHP

$debug = 1;

if($debug == 1) {
  error_reporting(E_ALL);
  ini_set('display_errors', 1);
}

?>

This will now allow us to track the errors. Now lets add the autoloader to make sure we are pulling in the goutte library.

<?PHP
$debug = 1;

if($debug == 1) {
  error_reporting(E_ALL);
  ini_set('display_errors', 1);
}

require_once('vendor/autoload.php');

?>

Now to instantiate the Goutte Client we need to put the use properties of it.

<?PHP
$debug = 1;

if($debug == 1) {
  error_reporting(E_ALL);
  ini_set('display_errors', 1);
}

require_once('vendor/autoload.php');

$username = 'zezima';

$client = new Client();

 $post_values = array('user1' => $username, 'submit' => 'Search');

 $crawler = $client->request('POST', 'http://services.runescape.com/m=hiscore_oldschool/hiscorepersonal.ws', [], [], ['HTTP_CONTENT_TYPE' => 'application/x-www-form-urlencoded'], http_build_query($post_values));

?>

Lets start the client now define our post parameters and the do our crawl. Now I found the post parameters by going to the hiscores lookup and then typing a username in. I was in the inspect element console for chrome and then I just looked at the request and what the parameters were. If you need more details on this a tutorial can be made.

Now lets go ahead and build a each loop on the element #contentHiscores tr. We want to loop through all the tr and find all the values. The syntax is kind of like Jquery.

<?PHP
$debug = 1;

if($debug == 1) {
  error_reporting(E_ALL);
  ini_set('display_errors', 1);
}

require_once('vendor/autoload.php');

$username = 'zezima';

$client = new Client();

 $post_values = array('user1' => $username, 'submit' => 'Search');

 $crawler = $client->request('POST', 'http://services.runescape.com/m=hiscore_oldschool/hiscorepersonal.ws', [], [], ['HTTP_CONTENT_TYPE' => 'application/x-www-form-urlencoded'], http_build_query($post_values));

 $acc_array = array();
 $crawler->filter('#contentHiscores tr')->each(function($node, $k) use (&$acc_array){
    if(!in_array($k, array(0,1,2,3)) && $node->children()->count() == 5) {
      $acc_array['skills'][strtolower(trim($node->children()->eq(1)->text()))] = array('rank' => $node->children()->eq(2)->text(), 'level' => $node->children()->eq(3)->text(), 'xp' => $node->children()->eq(4)->text());
    }
  });
?>

Lets now find the username and add it to the $acc_array

<?PHP
$debug = 1;

if($debug == 1) {
  error_reporting(E_ALL);
  ini_set('display_errors', 1);
}

require_once('vendor/autoload.php');

$username = 'zezima';

$client = new Client();

 $post_values = array('user1' => $username, 'submit' => 'Search');

 $crawler = $client->request('POST', 'http://services.runescape.com/m=hiscore_oldschool/hiscorepersonal.ws', [], [], ['HTTP_CONTENT_TYPE' => 'application/x-www-form-urlencoded'], http_build_query($post_values));

 $acc_array = array();
 $crawler->filter('#contentHiscores tr')->each(function($node, $k) use (&$acc_array){
    if(!in_array($k, array(0,1,2,3)) && $node->children()->count() == 5) {
      $acc_array['skills'][strtolower(trim($node->children()->eq(1)->text()))] = array('rank' => $node->children()->eq(2)->text(), 'level' => $node->children()->eq(3)->text(), 'xp' => $node->children()->eq(4)->text());
    }
  });

$acc_array['username'] = preg_replace('/Personal(.*?)('.$username.')$$/', '$2', $crawler->filter('#contentHiscores tr td')->children()->eq(0)->text());
?>

To finish it off to show you the results lets do a print_r


FINAL EXAMPLE:

<?PHP
$debug = 1;

if($debug == 1) {
  error_reporting(E_ALL);
  ini_set('display_errors', 1);
}

require_once('vendor/autoload.php');

$username = 'zezima';

$client = new Client();

 $post_values = array('user1' => $username, 'submit' => 'Search');

 $crawler = $client->request('POST', 'http://services.runescape.com/m=hiscore_oldschool/hiscorepersonal.ws', [], [], ['HTTP_CONTENT_TYPE' => 'application/x-www-form-urlencoded'], http_build_query($post_values));

 $acc_array = array();
 $crawler->filter('#contentHiscores tr')->each(function($node, $k) use (&$acc_array){
    if(!in_array($k, array(0,1,2,3)) && $node->children()->count() == 5) {
      $acc_array['skills'][strtolower(trim($node->children()->eq(1)->text()))] = array('rank' => $node->children()->eq(2)->text(), 'level' => $node->children()->eq(3)->text(), 'xp' => $node->children()->eq(4)->text());
    }
  });

$acc_array['username'] = preg_replace('/Personal(.*?)('.$username.')$$/', '$2', $crawler->filter('#contentHiscores tr td')->children()->eq(0)->text());

  echo '<pre>';
  print_r($acc_array);
?>

Hope this tutorial was helpful. If you have any questions feel free to reply and I will respond in a prompt manner or Pm me.

 

[kmjt]

Cool tutorial. User can also set their debugging in config file if they have access to php.ini (shared hosting doesn't always grant this access though)

PHP: List of php.ini directives - Manual

 

[Hamtower]

Thanks for this, very cool!

 

[Viral_]

Glad you guys liked it