Monday, December 14, 2015

A day in the life

In the constant pursuit of dependable clients I built a simple scraper for craigslist, I borrowed a PHP Curl wrapper from a github account. At this stage the PHP "script" searches all major cities at craigslist.org checks the "jobs" section for an array of keywords and saves the Title, City and URL to the Mysql database, eliminating duplicates. I have also written a PHP page that loads the results  from the database to a table and displays a clickable link to each page. basically at this point the scripts Ive used search and filter craigslist.org results tailored to my preferences. I did have to change the useragent in the Curl wrapper apparently that was a unique signature that got my attempts blocked.

TODO: 

  1. I have plans to rotate through the VNP servers to alternate the IP address to avoid detection.
  2. Extract the reply address and automate an email reply.
  3. Beautify the results page possibly with bootstrap. 

No comments:

Post a Comment