CURL DETECTED AS ROBOT (Need HELP)

  • Thread starter Thread starter Anonymous
  • Start date Start date
A

Anonymous

Guest
Hi Guys,

I am trying to crawl a website using CURL but it seems the website using imperva (bot management) and detect that my CURL as a bot and I couldn't penetrate the bot management firewall. Do you know how to make our CURL act similar to browser client ?

the website is : https://ib.bankmandiri.co.id

NB: You will get imperva (bot management) landing page when you try to crawl using CURL.
 
I have setup the useragent similar to the browser. The useragent based on the browser but nothing happened.

Could you please crawl the website https://ib.bankmandiri.co.id ? You will get the condition as I mentioned.
 
If the site doesn't want to be crawled, why are you so insistent on crawling it?
 
You must log in or register to reply here.
Back
Top