Scraping webpages in Laravel 5.6

General discussions related to php

Moderators: egami, macek, gesf

Post Reply
devquora
New php-forum User
New php-forum User
Posts: 1
Joined: Mon Jul 09, 2018 4:02 am
Contact:

Mon Jul 09, 2018 4:27 am

In this Tutorial we are going to see how to scrap a webpage in Laravel Framework. Laravel is free open source PHP framework based on MVC design pattern.
Follow the below steps to configure Goutte package in Laravel. In Laravel Framework goutte package is used for scraping webpages.

Here are the steps
Step1 : Locate to your project directory and run below command to install goutte package.

Code: Select all

composer require weidner/goutte
Step2: If you are Laravel below 5.5 then you have to register the service provide and facade class in you
congig/app.php

Simply add following lines of code in provider and aliases array of your app.php file.

Code: Select all

Weidner\Goutte\GoutteServiceProvider::class,  // in providers array

 'Goutte' => Weidner\Goutte\GoutteFacade::class, // aliases array
Step3: Once you register the aliases and provider you can now use Goutte facade in your application.

Below is sample code to scrap a web page in Laravel using Goutte.

Code: Select all

Route::get('/', function() {
    $crawler = Goutte::request('GET', 'https://duckduckgo.com/html/?q=Laravel');
    $crawler->filter('.result__title .result__a')->each(function ($node) {
      dump($node->text());
    });
    return view('welcome');
});

Getting an input by name in Goutte

Code: Select all

$user_agent='Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)';

[b]// Creating a client in  Goutte.[/b]
 $client = new \Goutte\Client();

$url=”https://www.abc.com”;

[b]//setting custom header [/b]

 $client->setHeader('User-Agent',$user_agent);

 $crawler = $client->request('GET',$url);

 $status= $client->getResponse()->getStatus();

if($status==200){
   $crawler->filterXpath('//input[@name="gluser_id"]')->count();
}
 

[b]// Find all images[/b]
    foreach ($crawler->find('img') as $element) {
        echo $element->src . '<br>';
    }
Thats it.
Thanks for reading!!
Web developer at
https://www.onlineinterviewquestions.com/

Post Reply