Web Scraping

General discussions related to php

Moderators: macek, egami, gesf

mico86
New php-forum User
New php-forum User
Posts: 1
Joined: Mon Jul 09, 2012 4:27 am

Web Scraping

Postby mico86 » Mon Jul 09, 2012 4:35 am

Hello everyone!
I would like some advice from you ... I have a website with a search form and I would like in writing, give me back the results of various listings of various sites. I read some things, I saw that the best method is the scraping of information with cURL. In your opinion, how should I proceed? The two main points are:
1) Find the value of research in various ad sites
2) Return the results of these various sites into one site, that is mine.
Thank you all :help:

User avatar
Nullsig
php-forum Fan User
php-forum Fan User
Posts: 981
Joined: Thu Feb 17, 2011 6:52 am
Location: Racine, WI

Re: Web Scraping

Postby Nullsig » Wed Jul 11, 2012 7:35 am

Your research is correct. You will need to use either something like cURL or wget to pull the results of the search and then utilizing PHP or a shell script you will parse through the data, typically with regular expressions, to sift through the retrieved html for the data that you wish to capture.

One major thing to consider is the rate of polling to the external sites. If you poll to fast you will come across as a hacker and will potentially be liable for damages if your script brings down the site. As a result of this I have a rule to never explain the code necessary to accomplish this goal past the description I have supplied above.


Return to “PHP General”

Who is online

Users browsing this forum: No registered users and 2 guests