Consuming APIs & Web Scraping

The web gets useful when your Python code can be a careful client: ask for one thing, check the answer, parse the data, and leave the server in peace. An API consumer is a program that calls an API instead of serving one, like a library patron using the desk instead of running the library.

This lesson still does not contact the internet. Real API calls and scraping examples are static shapes; the live Python code works with canned URLs, JSON strings, HTTP-ish responses, and HTML text.

Make API requests deliberately

The requests library is the common third-party Python package for sending HTTP requests from Python, but this browser lesson cannot import it or call a live URL. You can still learn the request shape: method, URL, query parameters, headers, timeout, and response checks.

A query parameter is one key-value filter placed in the URL's query string, such as reader=Mina. A timeout is the maximum time your client waits before giving up, like deciding you will not stand at the counter forever.

from urllib.parse import urlencode
 
base_url = "https://api.bookclub.test/reading-log"
params = {
    "reader": "Mina",
    "page": 1,
    "per_page": 2,
}
 
query_string = urlencode(params)
full_url = f"{base_url}?{query_string}"
 
headers = {
    "Accept": "application/json",
    "User-Agent": "better-python-course/1.0",
}
 
print("GET", full_url)
for name, value in headers.items():
    print(f"{name}: {value}")
print("Timeout: 10 seconds")

GET https://api.bookclub.test/reading-log?reader=Mina&page=1&per_page=2
Accept: application/json
User-Agent: better-python-course/1.0
Timeout: 10 seconds

In a normal local project, that shape often becomes this kind of requests code. This is static because it would need network access:

Static shape only - not run in this browser lesson:
 
response = requests.get(
    "https://api.bookclub.test/reading-log",
    params={"reader": "Mina", "page": 1, "per_page": 2},
    headers={"Accept": "application/json"},
    timeout=10,
)
response.raise_for_status()
data = response.json()

A request method is the HTTP action word your client sends. As a consumer, you mostly use GET to read data, POST to create data, PATCH or PUT to update data, and DELETE only when the API explicitly says that is allowed.

Headers, authentication, and checks

An authentication token is a secret text value that proves your request is allowed to access something. Many APIs expect an Authorization header, a header that carries credentials, often in the Bearer <token> shape.

Do not print real tokens, commit them to Git, paste them into screenshots, or hard-code them into lessons. Treat them like keys to a building. The shape is useful to learn; the actual value stays in environment variables or a secret manager in real projects.

api_token = "<token from environment>"
 
headers = {
    "Accept": "application/json",
    "Authorization": f"Bearer {api_token}",
}
 
for name, value in headers.items():
    print(f"{name}: {value}")

Accept: application/json
Authorization: Bearer <token from environment>

A content type is a header value that tells you the body's format, such as application/json. A response check is the habit of inspecting the status code and content type before trusting the body. It is like checking the stamp and label on a package before you unpack it.

import json
 
responses = [
    {
        "status": 200,
        "headers": {"content-type": "application/json"},
        "body": '{"items": [{"title": "Clean APIs"}]}',
    },
    {
        "status": 200,
        "headers": {"content-type": "text/html"},
        "body": "<h1>Reading log</h1>",
    },
    {
        "status": 429,
        "headers": {"retry-after": "30"},
        "body": '{"error": "too many requests"}',
    },
    {
        "status": 500,
        "headers": {"content-type": "application/json"},
        "body": '{"error": "server failed"}',
    },
]
 
for response in responses:
    status = response["status"]
    content_type = response["headers"].get("content-type", "")
    if 200 <= status < 300:
        if "application/json" not in content_type:
            print(f"{status}: unexpected content type -> {content_type}")
            continue
        payload = json.loads(response["body"])
        title = payload["items"][0]["title"]
        print(f"{status}: parsed first title -> {title}")
    elif status == 429:
        wait = response["headers"].get("retry-after", "a little")
        print(f"{status}: rate limited -> wait {wait} seconds")
    elif 500 <= status < 600:
        print(f"{status}: server problem -> retry later with a limit")
    else:
        print(f"{status}: fix the request before retrying")

200: parsed first title -> Clean APIs
200: unexpected content type -> text/html
429: rate limited -> wait 30 seconds
500: server problem -> retry later with a limit

Pagination, rate limits, and retries

Pagination is splitting a long API result into smaller pages so the server and client do not move everything at once. Think of a librarian handing you one box of records at a time, with a note for the next box.

A rate limit is a server rule that caps how many requests you can send in a time window. A retry is a later attempt after a temporary failure, like stepping out of line, waiting, and returning instead of hammering the bell.

Try it - walk canned API pages

This playground follows next_page links inside canned JSON. It also stops when a page says 429, because polite clients obey rate limits.

Real retry code also needs a maximum attempt count. Infinite retries turn one temporary failure into a noisy loop.

Scraping as a fallback

Web scraping is extracting data from web pages when there is no stable API, feed, export, or database access. It is the fallback move, not the first move, because pages are designed for people and can change without warning.

Before scraping, check the site's documentation, footer links, data portal, feeds, CSV exports, and terms. A feed is a published update list, often RSS or Atom, that programs can read more reliably than page layout. Terms of service are the site's usage rules, robots.txt is a public crawl-policy file that tells automated clients which paths are disallowed, and crawl delay is a requested pause between automated requests.

HTML parsing and selectors

HTML is the markup language web pages use to describe headings, links, lists, and other page structure. An HTML tag is a named page marker such as <a>, and an HTML attribute is extra tag data such as href="/articles/1".

A selector is a pattern for choosing page elements, like sticky tabs on a filing cabinet. Many HTML tools use selector-style ideas, but the live example below uses Python's standard html.parser so it can run here.

from html.parser import HTMLParser
from urllib.parse import urljoin
 
html_text = """
<section class="articles">
  <a class="article-link" href="/articles/python-web-client">Python as a Web Client</a>
  <a class="article-link" href="/articles/responsible-scraping">Responsible Scraping</a>
</section>
"""
 
class ArticleLinkParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_article_link = False
        self.current_href = None
        self.links = []
 
    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        if tag == "a" and attrs.get("class") == "article-link":
            self.in_article_link = True
            self.current_href = urljoin("https://bookclub.test", attrs["href"])
 
    def handle_data(self, data):
        title = data.strip()
        if self.in_article_link and title:
            self.links.append((title, self.current_href))
 
    def handle_endtag(self, tag):
        if tag == "a":
            self.in_article_link = False
            self.current_href = None
 
parser = ArticleLinkParser()
parser.feed(html_text)
 
for title, url in parser.links:
    print(f"{title} -> {url}")

Python as a Web Client -> https://bookclub.test/articles/python-web-client
Responsible Scraping -> https://bookclub.test/articles/responsible-scraping

BeautifulSoup is a third-party Python library that makes HTML parsing friendlier than writing your own parser class. Its shape is static here because the package is not loaded in this lesson:

Static shape only - not run in this browser lesson:
 
soup = BeautifulSoup(html_text, "html.parser")
for link in soup.select("a.article-link"):
    print(link.get_text(strip=True), link["href"])

Scrapy is a scraping framework, a larger tool that organizes crawling projects with queues, rules, and pipelines. Selenium is browser automation, meaning code that controls a real browser for pages that need clicks or browser-rendered content. Playwright is a modern browser automation library often used for testing and pages whose content appears after browser code runs.

Use the lightest tool that fits. If an API exists, consume the API. If a static page has a few public links, parse HTML. If the page needs browser behavior, reach for browser automation only after checking that automation is allowed.

You can now be a careful Python web client: shape requests, send headers, respect limits, parse JSON, and scrape only when it is the right fallback. Next, you will flip sides and see how Python frameworks build web apps and APIs that respond to clients like yours.