Web Scraping

General discussions related to php

Moderators: egami, macek, gesf

Post Reply
User avatar
php-forum Fan User
php-forum Fan User
Posts: 979
Joined: Thu Feb 17, 2011 6:52 am
Location: Racine, WI

Wed Jul 11, 2012 7:35 am

Your research is correct. You will need to use either something like cURL or wget to pull the results of the search and then utilizing PHP or a shell script you will parse through the data, typically with regular expressions, to sift through the retrieved html for the data that you wish to capture.

One major thing to consider is the rate of polling to the external sites. If you poll to fast you will come across as a hacker and will potentially be liable for damages if your script brings down the site. As a result of this I have a rule to never explain the code necessary to accomplish this goal past the description I have supplied above.

Post Reply