Semalt Review: Web Scraping For Fun & Profit
You can site scrape without the need for an API. While site owners are aggressive about stopping scraping, they care less about APIs and instead put more emphasis on websites. The facts that many sites do not adequately guard against automatic access creates a leeway for scrapers. Some simple workarounds will help you harvest the data you need.
Getting Started with Scraping
Scraping requires understanding the structure of the data you need and its accessibility. This starts by fetching your data. Find the URL that returns the information you need. Browse through the website and check how the URLs change as you navigate through different sections.
Alternatively, search several terms on the site and check how the URLs change based on your search term. You should see a GET parameter like q= that change whenever you search a new term. Retain the GET parameters necessary for loading your data and remove the others.
How To Deal With Pagination
Pagination keeps you from accessing all the data you need at once. When you click page 2, an offset= parameter is added to the URL. This is either the number of elements on a page or the page number. Increment this number on every page of your data.
For sites that use AJAX, pull up the network tab in Firebug or Inspector. Check the XHR requests, identify and focus on those that pull in your data.
Get Data from Page Markup
This is achieved using CSS hooks. Right-click a particular section of your data. Pull the Firebug or Inspector and zoom through the DOM tree to get the outmost that wraps a single item. Once you have the correct node from DOM tree, view the page source to ensure your elements are accessible in raw HTML.
To site scrape successfully, you need an HTML parsing library that reads in HTML and turns it to an object that you can iterate until you get what you need. If your HTTP library requires that you set some cookies or headers, browse the site on your web browser and get the headers being sent by your browser. Put them in a dictionary and forward with your request.
When you Need a Login to Scrape
If you must create an account and login to get the data you want, you need to have a good HTTP library to handle logins. Scraper login exposes you to third-party sites.
If the rate limit of your web service depends on IP address, set a code that hits the web service to a client-side Javascript. Then forward the results back to your server from each client. The results will appear to originate from so many places, and none will exceed their rate limit.
Poorly Formed Markup
Some markups can be difficult to validate. In such cases, dig into your HTML parser for error tolerance settings. Alternatively, treat the whole HTML document as a long string and do string splitting.
While you can site scrape all kinds of data on the net some sites employ software to stop scraping, and other prohibit web scraping. Such sites can sue you and even have you jailed for harvesting their data. So be smart in all your web scraping and do it safely.
Post a comment