Data scraping

Discussion in 'Technology' started by Superfluous, Dec 19, 2014.

Data scraping
  1. Unread #1 - Dec 19, 2014 at 9:50 AM
  2. Superfluous
    Joined:
    Jul 5, 2012
    Posts:
    18,939
    Referrals:
    5
    Sythe Gold:
    9,135
    Vouch Thread:
    Click Here
    Discord Unique ID:
    247909953925414913
    Discord Username:
    .superfluous.
    Two Factor Authentication User Pool Shark Air Fryer DIAF m`lady Le Kingdoms Player STEVE Creamy

    Superfluous Rainbet.com Casino & Sportsbook
    Crabby Retired Global Moderator

    Data scraping

    So I've got a month off from school and wanted to dabble in web scraping. I don't know much about it (thus far I've only used kimonolabs.com for basic scraping needs), but I've taken like a year's worth of programming courses, so I'm not a total noob.

    For the sake of example, let's say I wanted to scrape the price of google stock every time it's traded (or, if that's too hard, every second). How would one go about doing this, both from a programming and setup point of view? I've heard that python is the way to go, but that's really all I know. Any advice is appreciated.
     
  3. Unread #2 - Dec 19, 2014 at 11:09 PM
  4. kmjt
    Joined:
    Aug 21, 2009
    Posts:
    14,450
    Referrals:
    8
    Sythe Gold:
    449

    kmjt -.- The nocturnal life chose me -.-
    Banned

    Data scraping

    Do you know of a site that updates the stock every time it is traded? If so you can always just write a program that gets the html every second and write a scanner to detect any change?
     
  5. Unread #3 - Dec 20, 2014 at 3:43 PM
  6. Superfluous
    Joined:
    Jul 5, 2012
    Posts:
    18,939
    Referrals:
    5
    Sythe Gold:
    9,135
    Vouch Thread:
    Click Here
    Discord Unique ID:
    247909953925414913
    Discord Username:
    .superfluous.
    Two Factor Authentication User Pool Shark Air Fryer DIAF m`lady Le Kingdoms Player STEVE Creamy

    Superfluous Rainbet.com Casino & Sportsbook
    Crabby Retired Global Moderator

    Data scraping

    I don't know of sites that do it, though some must exist. I think there are ones for BTC, so I could always start there at first. I know nasdaq records (all?) trades in this format (http://www.nasdaq.com/symbol/goog/time-sales?time=1), but getting real-time data would be best.

    But I do know of trading platforms that update every tick and show every trade (thinkOrSwim, e.g.). Is it possible to scrape from something like that? (might look something like this: http://fromzerotooptions.com/wp-con...e-Grid-with-Level-2-Quotes-in-ThinkorSwim.jpg)
     
  7. Unread #4 - Dec 20, 2014 at 4:00 PM
  8. Shin
    Joined:
    Mar 10, 2007
    Posts:
    14,171
    Referrals:
    23
    Sythe Gold:
    196
    Discord Unique ID:
    777373911821713408
    Pool Shark (4) Village Drunk <3 n4n0 (29) Battleship Champion

    Shin Join the Sythe.org Discord
    Retired Administrator Legendary Mudkips $100 USD Donor

    Data scraping

    Do you only want to use Python, or have you looked into other languages as well?
     
  9. Unread #5 - Dec 20, 2014 at 6:14 PM
  10. Superfluous
    Joined:
    Jul 5, 2012
    Posts:
    18,939
    Referrals:
    5
    Sythe Gold:
    9,135
    Vouch Thread:
    Click Here
    Discord Unique ID:
    247909953925414913
    Discord Username:
    .superfluous.
    Two Factor Authentication User Pool Shark Air Fryer DIAF m`lady Le Kingdoms Player STEVE Creamy

    Superfluous Rainbet.com Casino & Sportsbook
    Crabby Retired Global Moderator

    Data scraping

    My background is in java, but I'm happy to use whatever. What did you have in mind?
     
  11. Unread #6 - Dec 21, 2014 at 3:30 AM
  12. kmjt
    Joined:
    Aug 21, 2009
    Posts:
    14,450
    Referrals:
    8
    Sythe Gold:
    449

    kmjt -.- The nocturnal life chose me -.-
    Banned

    Data scraping


    I guess it really depends on the magnitude of data you are trying to scrape. If there isn't much you can probably use any language.
     
  13. Unread #7 - Dec 21, 2014 at 5:31 PM
  14. Superfluous
    Joined:
    Jul 5, 2012
    Posts:
    18,939
    Referrals:
    5
    Sythe Gold:
    9,135
    Vouch Thread:
    Click Here
    Discord Unique ID:
    247909953925414913
    Discord Username:
    .superfluous.
    Two Factor Authentication User Pool Shark Air Fryer DIAF m`lady Le Kingdoms Player STEVE Creamy

    Superfluous Rainbet.com Casino & Sportsbook
    Crabby Retired Global Moderator

    Data scraping

    I'm not really sure what I want to do with things yet, so I'll update this when I have an idea. Thanks guys!
     
< I dont have structured thinking... | Need Help With Constant Bsod Will Pay >

Users viewing this thread
1 guest


 
 
Adblock breaks this site