Skip to content
OSULZER.AT
Python12 min read· updated March 2, 2022

Tutorial: Building a Simple Python Web Crawler

Build a simple web crawler in Python with Requests and BeautifulSoup 4. The tutorial shows how to fetch pages and select the information you need.

This post is from 2019 - some of its content may be outdated by now.

In this article, we take a quick look at building a simple web crawler. Crawlers are small automated scripts that visit websites and extract the information you want from them. If needed, crawlers can also interact with websites, which means filling in form fields, logging in, or simulating clicks. In this tutorial, we limit ourselves to visiting a page and selecting the information we want.

One thing up front: when you build crawlers, the guiding idea should always be to cause as little friction as possible. That means you should not bombard the crawled website with requests, but let some time pass between requests. The goal is that the web server does not deliver pages to other users any more slowly while your crawler does its job. If you do not build in wait times or request limits, a crawler can quickly slide into the territory of denial of service attacks. A good rule of thumb is 5 pages per second per domain, and fewer for slow web servers or if the robots.txt file says so explicitly.

Choosing a Website

For this tutorial, we will use the blog of osulzer.at. The overview page lists all the posts published so far. Our goal is to extract cover images, headings, and links to the individual posts. Since the blog is still quite new and only 4 posts had been published when this article was written, we will also crawl the post pages themselves to show how you can jump to further subpages.

Tools You Need

To build our crawler, we use the following tools in this tutorial.

  • Python 3
  • PyCharm - Python editor with handy features (free in the Community Edition)
  • Python packages (can be installed via PyCharm or pip):
    • Requests - Sends requests to web servers and returns HTML code
    • BeautifulSoup 4 - Parser that helps us select elements in the HTML code
    • cssutils - Parser for CSS code
    • urllib - Helps us later in the tutorial to extract the domain from a link

First install Python 3 and then PyCharm. In the PyCharm settings, you can then install “Project Interpreters” such as Requests or BeautifulSoup 4. Of course, it is a good idea to create a virtual environment for the project. If that term means nothing to you and you are still quite new to Python, you can simply install the required packages the normal way.

Website Analysis

Before we start building our crawler, we take a look at the page we want to crawl. We visit the page in the browser of our choice and open the developer console. Firefox or Chrome work best for this.

We look for clues that let us tell the individual elements of the website apart. In our case, all blog posts are located in a section with the CSS class “posts”. Each block with information about a post then has the class “blog-post”. These blocks contain a link, the title, and the cover image, which is defined as the background image of the link. So our crawler has to find all HTML elements with the class “blog-post” and read out the information they contain.

The Crawler

Reading the HTML Code of the Website

In the first step, we need the HTML code of the blog home page. For this we use the “requests” package we installed earlier. Requests sends HTTP requests to the desired URL and then returns the HTML code it receives. In our example, we put the request in its own function. This makes it easier to switch to other methods later.

In practice, our script now looks like this:

import requests

url = 'https://www.osulzer.at/blog'

# returns the html code of the requested url
def get_url_content(url):
    return requests.get(url).text

Pretty simple so far. Now we import BeautifulSoup 4 so we can work with the HTML code more easily. Then we define a new function in which we request the desired URL and pass the HTML code we receive to BeautifulSoup 4.

def get_blog_content(url):
        content = get_url_content(url)
        # pass html to beautifulsoup parser
        soup = BeautifulSoup(content, "html.parser")

“soup” is now a BeautifulSoup 4 object whose content can be queried. As described above, all blog posts on the blog home page have the CSS class “blog-post”. Using this class, we can now narrow down the content we need. For this we use BeautifulSoup 4 and search for all div elements that have the desired class.

def get_blog_content(url):
        content = get_url_content(url)
        # pass html to beautifulsoup parser
        soup = BeautifulSoup(content, "html.parser")
        for post in soup.findAll('div', {'class': 'blog-post'}):
            print(post.find('h2'))

This code should now give us the following output:

<h2>Ubuntu 18.04 Server mit Nginx, MySQL, PHP und SSL aufsetzen</h2>
<h2>Linux Root-Server / vServer Monitoring</h2>
<h2>WordPress für PageSpeed Insights optimieren</h2>
<h2>Spambot Bekämpfung ohne Captcha Abfrage</h2>

BeautifulSoup 4 also outputs the h2 tag. To get only the text, we have to append “.text” to our query.

def get_blog_content(url):
        content = get_url_content(url)
        # pass html to beautifulsoup parser
        soup = BeautifulSoup(content, "html.parser")
        for post in soup.findAll('div', {'class': 'blog-post'}):
            print(post.find('h2').text)

The output now looks like this:

Ubuntu 18.04 Server mit Nginx, MySQL, PHP und SSL aufsetzen
Linux Root-Server / vServer Monitoring
WordPress für PageSpeed Insights optimieren
Spambot Bekämpfung ohne Captcha Abfrage

Now it is time to read out the other information. We get the link to each post by searching our post object for the “a” element and then extracting the “href” attribute with .get().

The background image is defined via CSS. To get the image link without using regex, we use the “cssutils” package. We feed it the inline CSS we found via “post.find(‘a’).get(‘style’)”. After that, we can access the value of “background-image”. To get only the image address, we first remove the unwanted pieces of text at the beginning and end.

import requests
from bs4 import BeautifulSoup
import time
import cssutils

url = 'https://www.osulzer.at/blog'

def get_blog_content(url):

    content = get_url_content(url)
    # pass html to beautifulsoup parser
    soup = BeautifulSoup(content, "html.parser")

    for post in soup.findAll('div', {'class': 'blog-post'}):

        # pass inline css to cssutils for easy reading of the attributes
        bg = cssutils.parseStyle(post.find('a').get('style'))
        print('Title: %s' % post.find('h2').text)
        
        # .get can be used to read attributes of the "a" element
        print('Link: %s' % post.find('a').get('href'))
        
        # the image link we read still contains unwanted styling attributes, which we simply remove
        print('Image: %s' % bg['background-image'].replace('url(', '').replace(')', ''))

Our new output now looks like this:

Title: Ubuntu 18.04 Server mit Nginx, MySQL, PHP und SSL aufsetzen
Link: /blog/ubuntu-1804-server-mit-nginx-mysql-php-und-ssl-aufsetzen
Image: /storage/covers/ubuntu-cover-30.jpg
Title: Linux Root-Server / vServer Monitoring
Link: /blog/linux-root-server-vserver-monitoring
Image: /storage/covers/server-monitoring-cover-29.jpg
Title: WordPress für PageSpeed Insights optimieren
Link: /blog/wordpress-fur-pagespeed-insights-optimieren
Image: /storage/covers/pagespeed-opener-cover-28.jpg
Title: Spambot Bekämpfung ohne Captcha Abfrage
Link: /blog/spambot-bekampfung-ohne-captcha-abfrage
Image: /storage/covers/mail-spam-cover-25.png

Crawling Blog Post Detail Pages

Our crawler can now read the blog home page. Following the same principle, we can also crawl the subpages we found. For example, it would be interesting to find out how many words each post contains. Our approach is the same again. First we take a look at our post page.

In our example, the content of the blog post is displayed in a div with the CSS class “textcontent”. Once again we use BeautifulSoup 4 to get the text, filter special characters out of the text, and then split the string at every space. That looks like this. We then get the (not 100% accurate) word count with the len() function.

def get_blog_post(url):

        content = get_url_content(url)
        soup = BeautifulSoup(content, "html.parser")
    
        # extract the content from the div with the class textcontent
        text = soup.find('div', {'class': 'textcontent'}).text
    
        # replace special characters with spaces
        for char in '=+-.,\n':
            text = text.replace(char, ' ')
        text = text.lower().split()
    
        # prints the length of the list resulting from split
        print('Wörter: %s' % len(text))

Now we only have to integrate the new function into our get_blog_content function. While we are at it, we will also rework a few parts of the function to make everything a bit clearer.

def get_blog_content(url):

        content = get_url_content(url)
        # pass html to beautifulsoup parser
        soup = BeautifulSoup(content, "html.parser")
    
        for post in soup.findAll('div', {'class': 'blog-post'}):
    
            # pass inline css to cssutils for easy reading of the attributes
            bg = cssutils.parseStyle(post.find('a').get('style'))
    
            title = post.find('h2').text
            link = post.find('a').get('href')
            image = bg['background-image'].replace('url(', '').replace(')', '')
            
            print('Title: %s' % title)
    
            # .get can be used to read attributes of the "a" element
            print('Link: %s' % link)
    
            # the image link we read still contains unwanted styling attributes, which we simply remove
            print('Image: %s' % image)
            
            get_blog_post(link)

We are not quite done yet, though. The link we read from our blog is relative, so we first have to assemble the complete URL. The URL we stored in the variable “url” already contains the domain, and we extract it elegantly with the urllib package. The new function for this looks like this:

from urllib.parse import urlparse

def get_domain(url):
    parsed_uri = urlparse(url)
    return '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)

The finished get_blog_content function then looks like this:

def get_blog_content(url):
    
        content = get_url_content(url)
        # pass html to beautifulsoup parser
        soup = BeautifulSoup(content, "html.parser")
    
        for post in soup.findAll('div', {'class': 'blog-post'}):
    
            # pass inline css to cssutils for easy reading of the attributes
            bg = cssutils.parseStyle(post.find('a').get('style'))
    
            title = post.find('h2').text
            link = post.find('a').get('href')
            image = bg['background-image'].replace('url(', '').replace(')', '')
    
            print('Title: %s' % title)
    
            # .get can be used to read attributes of the "a" element
            print('Link: %s' % link)
    
            # the image link we read still contains unwanted styling attributes, which we simply remove
            print('Image: %s' % image)
    
            get_blog_post(get_domain(url) + link)

As soon as we start the crawler, we get the following output:

Title: Ubuntu 18.04 Server mit Nginx, MySQL, PHP und SSL aufsetzen
Link: /blog/ubuntu-1804-server-mit-nginx-mysql-php-und-ssl-aufsetzen
Image: /storage/covers/ubuntu-cover-30.jpg
Wörter: 4619
Title: Linux Root-Server / vServer Monitoring
Link: /blog/linux-root-server-vserver-monitoring
Image: /storage/covers/server-monitoring-cover-29.jpg
Wörter: 704
Title: WordPress für PageSpeed Insights optimieren
Link: /blog/wordpress-fur-pagespeed-insights-optimieren
Image: /storage/covers/pagespeed-opener-cover-28.jpg
Wörter: 3779
Title: Spambot Bekämpfung ohne Captcha Abfrage
Link: /blog/spambot-bekampfung-ohne-captcha-abfrage
Image: /storage/covers/mail-spam-cover-25.png
Wörter: 2301

If you run the code yourself, you will notice that the crawler produces results very quickly. In this tutorial, we work with requests (we could actually also use the already imported urllib directly), and information such as images is not transferred, which makes the crawler noticeably faster than a normal browser. As mentioned above, the crawler’s speed can cause problems on the server of the site being crawled. We are hammering the server with requests via this script. To tone the script down a bit, you can build in pauses. A good place for this is, for example, the function “get_url_content()”. A simple “time.sleep(1)” delays the loading of each URL passed in by one second. This makes the script noticeably slower, but it follows the rule of not putting a disproportionate load on the target server.

A quick tip on the side: Some websites state in their robots.txt file how much time must pass between requests from crawlers. An entry such as “Crawl-delay: 5” means that 5 seconds must pass between requests. Respect these wishes.

Finished Crawler Code

Here is the complete script for you to try out yourself.

import requests
from bs4 import BeautifulSoup
import time
import cssutils
from urllib.parse import urlparse

url = 'https://www.osulzer.at/blog'

def get_blog_content(url):
    content = get_url_content(url)
    # pass html to beautifulsoup parser
    soup = BeautifulSoup(content, "html.parser")

    for post in soup.findAll('div', {'class': 'blog-post'}):

        # pass inline css to cssutils for easy reading of the attributes
        bg = cssutils.parseStyle(post.find('a').get('style'))
        title = post.find('h2').text
        link = post.find('a').get('href')
        image = bg['background-image'].replace('url(', '').replace(')', '')

        print('Title: %s' % title)

        # .get can be used to read attributes of the "a" element
        print('Link: %s' % link)

        # the image link we read still contains unwanted styling attributes, which we simply remove
        print('Image: %s' % image)

        get_blog_post(get_domain(url) + link)


def get_blog_post(url):
    content = get_url_content(url)
    soup = BeautifulSoup(content, "html.parser")

    # extract the content from the div with the class textcontent
    text = soup.find('div', {'class': 'textcontent'}).text

    # replace special characters with spaces
    for char in '=+-.,\n':
        text = text.replace(char, ' ')
    text = text.lower().split()

    # prints the length of the list resulting from split
    print('Wörter: %s' % len(text))



# returns the html code of the requested url
def get_url_content(url):
    # waits 1 second between each request
    time.sleep(1)
    return requests.get(url).text

def get_domain(url):
    parsed_uri = urlparse(url)
    return '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)


get_blog_content(url)

Conclusion

Done, you have built your first web crawler. It is quite rudimentary, but it still does its job. The next stage of development would be, for example, threading the individual crawler instances. That lets several crawlers download information at the same time, which becomes necessary especially for larger websites if you want to read out information in a reasonable amount of time. In its current form, our crawler also does not store any information and does not work on sites that load their content dynamically. For dynamic websites such as Facebook, we have to switch to a so-called headless browser. It creates more overhead but offers better compatibility.

Would you like a crawler tailored to your requirements? I offer web crawler development in Vienna and Lower Austria. You can find more information about my services at crawler and bot development. If the collected data should go straight into your own portal or dashboard, I build that as a web app.

Finally, here are a few more links about web crawlers.

  • Scrapy - A web crawler framework that already comes with many crawler features on board.
  • Headless Firefox - Information about Firefox’s headless mode
  • Threading - Detailed information about Python and threading

Web Crawlers & Bots

Collect data automatically?

Custom crawlers that collect and prepare data reliably.

See the web crawlers

New project?

Contact.

Tell me briefly about your project - I'll get back to you personally.

Your data is used solely to handle your inquiry. More in the privacy policy.