How would I make this web scraper for a Facebook page?

[kmjt]

I think this would be pretty straight forward. I want to write a scraper for Facebook pages (a page can be accessed even when you are not logged in). Basically it will iterate through every post in the page's lifespan and gather user's who have liked a post. The first problem I see is pagination. When a page is scrolled to reveal more posts the URL does not change. How can I get around this? Not really sure where to start.

 

[Covey]

I haven't programmed for a long time, but follow these rough steps and it should get the info you're after....but you'll have to do a lot of parsing.

1) Load the facebook page you're scraping
2) Retrieve the raw html from the browser and use some parsing functions to scrap what you need.
3) Force the scroll bar to the bottom so it loads the next section
4) Repeat step 2 & 3 (you'll probably have to make a function to detect when the page has finished loading so you know when to pull the raw html)

Depending what language you're using the browser should have some form of function within itself that will tell you when it's finished loading.

 

[kmjt]

I haven't programmed for a long time, but follow these rough steps and it should get the info you're after....but you'll have to do a lot of parsing.

1) Load the facebook page you're scraping
2) Retrieve the raw html from the browser and use some parsing functions to scrap what you need.
3) Force the scroll bar to the bottom so it loads the next section
4) Repeat step 2 & 3 (you'll probably have to make a function to detect when the page has finished loading so you know when to pull the raw html)

Depending what language you're using the browser should have some form of function within itself that will tell you when it's finished loading.

Thanks man! I would like to do this in java. So for 3 I can probably just use the Robot class. I am not really sure how to detect that new section of page reloaded though. I could probably detect screen colour but i'm sure there is a much more efficient way... Anyone have an idea what the java "browser function" is?

 

[70i]

Never done something like this before, but couldn't you just keep selecting the "See More" link.

 

[CompileTime]

Thanks man! I would like to do this in java. So for 3 I can probably just use the Robot class. I am not really sure how to detect that new section of page reloaded though. I could probably detect screen colour but i'm sure there is a much more efficient way... Anyone have an idea what the java "browser function" is?

That would be really inefficent. You can load webpages without actually loading up a browser like chrome or Mozilla. Just make some HTTP requests, parse the results and look through it, of course I'm oversimplifying here, but using the robot class just seems silly.

 

[Virtual]

Could you post/pm me the actual link, I might be able to help

 

[kmjt]

Never done something like this before, but couldn't you just keep selecting the "See More" link.

Where on the page is that? I don't see it

That would be really inefficent. You can load webpages without actually loading up a browser like chrome or Mozilla. Just make some HTTP requests, parse the results and look through it, of course I'm oversimplifying here, but using the robot class just seems silly.

That is my main problem. I don't understand how the pagination works. I don't see how the parameters in url change when a new section is loaded (if you just scroll down, there are actually no parameters and the url is the same). I wish it was just as simple as https://www.facebook.com/PageName?post=1 lol

Could you post/pm me the actual link, I might be able to help

PMing you a link now. Although I want my scraper to be generic.

 

[SuF]

This is going to be a lot harder than you think. Facebook's HTML is going to be essentially entirely generated on the client side with Javascript. You will need to execute the Javascript in order to get the DOM that you need to parse. That means the only realistic way of doing this is within the browser. The paging works by detecting where the browsers scroll bar is with Javascript and making a request back to the server for that information so you will need to use Javascript to move the scrollbar yourself.

 

[kmjt]

This is going to be a lot harder than you think. Facebook's HTML is going to be essentially entirely generated on the client side with Javascript. You will need to execute the Javascript in order to get the DOM that you need to parse. That means the only realistic way of doing this is within the browser. The paging works by detecting where the browsers scroll bar is with Javascript and making a request back to the server for that information so you will need to use Javascript to move the scrollbar yourself.

Hmm. Can't I just do the scrolling with something like Robot class from java? It has a scroll method, although i'm not sure how reliable it is. Haven't really played around with it much but I can also just add colour detection on the Facebook page to make sure it scrolled to correct position? Thanks for input.

 

[SuF]

Hmm. Can't I just do the scrolling with something like Robot class from java? It has a scroll method, although i'm not sure how reliable it is. Haven't really played around with it much but I can also just add colour detection on the Facebook page to make sure it scrolled to correct position? Thanks for input.

So you are going to have Facebook open and use the mouse to move stuff? That seems like a lot of effort.

 

[kmjt]

So you are going to have Facebook open and use the mouse to move stuff? That seems like a lot of effort.

I think it would be pretty straight forward. The Robot class has a pretty decent scrolling method so I can just scroll until I reach the bottom of the page (where it needs to reload posts), and then in turn the loading process happens, then I parse? I will put a video together tonight to try to demonstrate what I mean.

To be honest it would probably just be better if I learned javascript eh lol.

 

[Virtual]

After analyzing the page you sent me I found out few things.


https://www.facebook.com/ajax/pagelet/generic.php/PagePostsSectionPagelet?data={"segment_index":**,"page_index":0,"page":***,"column":"main","post_section":{"profile_id":****,"start":1420099200,"end":1451635199,"query_type":8,"filter":1,"filter_after_timestamp":1427470882},"section_index":2,"hidden":false,"posts_loaded":0,"show_all_posts":false}&__user=0&__a=1&__dyn=*****&__req=5&__rev=1673637

** = the segment index (0 - When you load it, the first time you scroll down 1, 2 is the 2nd time and so on)
*** = The page id (you should be able to find that)
**** = The user id (if you're viewing it as a guest, the user id = the page id)
***** = The page hashed id or so (I found out that it doesn't change per page, so that's my assumptions)

P.s I'll PM you an example for your page

 

[70i]

Where on the page is that? I don't see it

Could you post/pm me the actual link, I might be able to help

Save the page so it doesn't load before you can select it.

For this link
https://www.facebook.com/browse/likes?id=10152826439076314&actorid=51931216313

This is the "See More"
https://www.facebook.com/ajax/brows...1931216313&beforetime=0&aftertime=0&start=200