Semalt: Extracting URLs From Web Pages With Beautiful Soup

Beautiful Soup is a high-level Python package used for parsing XML and HTML documents. Beautiful Soup Python library creates a parse tree that is used to extract useful information from HyperText Markup Language (HTML). This library is available for both Python 2 and Python 3 versions.

In most instances, you find that your target data can only be accessed and used as a part of a web page. In such a case, you need to use such web scraping technique that can extract data in the formats that can be analyzed. This is where Beautiful Soup library comes in.

Requirements

You need the right modules to use Beautiful Soup library. To get started, you need to install Python 2.7 programming language on your machine. In this post, you'll learn how to scrape a website and extract all URLs using Requests and Beautiful Soup 4. HTML parsing is a do-it-yourself task, especially with the technical help of Beautiful Soup.

Why Use Beautiful Soup?

Beautiful Soup is a top-ranked Python package that has been used to scrape websites and parse HTML tags since 2004. Recently, Beautiful Soup 4 replaced Beautiful Soup 3 in the industry. Note that BS4 works on both Python versions whereas BS3 only works on Python 2.7. The library comprises of the following inbuilt features:

• Encodings capability – You don't have to panic about encodings once you install the necessary beautiful Soup modules on your machine. The library is automated to convert inputs to Unicode and outputs to UTF-8.
• Navigation capability – Beautiful Soup offers easy to use methods for searching, navigating, and modifying a parse tree.

How to use Beautiful Soup library?

After installing Beautiful Soup on your machine, you can start using the library. To get started, import bs4 library at the beginning of your Python code. Pass content or URL to Beautiful Soup to create a Soup object. However, the library does not fetch the target web page on itself. Here, you have to complete that task manually. You can also easily fetch the preferred web pages using a combination of Python and Beautiful Soup.

Roles of request library

To scrape a page, you need to download it first. You can download web pages using request library. Requests library works by making a "GET" request to the web servers, which will, in turn, download HTML contents of the preferred web page.

Extracting URLs from web pages

Now you have detailed information regarding Beautiful Soup library. A combination of BS4 library and Python will help you fetch a web page very quickly. To extract all the URLs from your target web page, use the "find all" method. This method will give you a compilation of elements with the tag. From bs4, import both Beautiful Soup and requests. Run your code and enter a website or web page to extract the URLs from.